Your AI agent knocks. Does the website let it in?

The website's robots.txt allows the vendor's reading agent (ChatGPT-User, Claude-User, Google-Agent, meta-externalfetcher) on the homepage, and our real fetch with that user agent was not turned away. Then the chatbot can usually read the page when you ask it to.

Reading in chat is the fetch you trigger yourself, like “summarise this page”. Search is the index the chatbot cites sources from (OAI-SearchBot, Claude-SearchBot, Googlebot, meta-webindexer). Training is about whether content may be used to train models (GPTBot, ClaudeBot, Google-Extended, meta-externalagent).

Google and Meta say their user-triggered fetchers (Google-Agent, meta-externalfetcher) generally ignore robots.txt. We still show what the website sets there, and whether its server rejects the user agent. That is the part that really works.

Agents that drive your browser (ChatGPT agent, Claude in Chrome, Gemini in Chrome) see the website like you do. robots.txt does not apply there, bot protection does. If we detect Cloudflare, Akamai, DataDome, HUMAN, Imperva or AWS WAF, or the site already blocks a plain Chrome fetch, we show “captcha likely”. It is a hint, not proof: the website decides on every visit.

The 500 websites every morning at 4:30 Berlin time. A change only counts once a second check in the same run confirms it. Live checks run on demand and are valid for 6 hours.

We compiled it by reach in Germany, based on public rankings (Similarweb, Semrush, agof, IVW). The first 250 ranks are ordered by reach; after that the order within categories is estimated. If a website is missing, just check it individually.

Yes. Under every result you can set up a free alert for that website. You get one email when access changes for one of the four agents, nothing else. Unsubscribe with one click.