Robots.txt monitoring

Fetches /robots.txt, parses it the way a crawler does, and reports whether the URLs this monitor requires are still crawlable.

Part of content, links and seo: is the page, the api and what crawlers read still right? In the grid, its column is headed “Robots”.