BUSINESSDevelopments · · 2 min read · Source: Google Search Central Blog
robots.txt: One line could opt you out of AI training at four vendors
Google, Microsoft, OpenAI and Anthropic announced a shared signal for robots.txt files on October 30, 2026. It is meant to let website owners state in a single step that their content should not be used to train generative AI.
Key facts
- Announced on October 30, 2026 by Google, Microsoft, OpenAI and Anthropic; the token is __usage_ai-training in robots.txt.
- Microsoft honors it from February 1, 2027, Google aims for 2027, and OpenAI for the coming months, according to SiliconANGLE.
- So far it's an individual IETF draft, not a standard, and a catch-all rule (User-Agent: *) does not include the token.
What the signal looks like
Today, every vendor needs its own entry, such as GPTBot or Google-Extended. The plan is one shared token that sits in robots.txt where a crawler name would go:
User-Agent: __usage_ai-training
Disallow: /You can also apply it to specific paths. One catch: under the draft, a catch-all rule (User-Agent: *) does not cover this token, so you have to add it by name.
Who joins, and when
- Microsoft: honors the new preferences from February 1, 2027.
- Google: aims to support it in 2027.
- OpenAI: "in the coming months," according to SiliconANGLE.
- Anthropic: an Anthropic employee wrote the draft; the draft itself gives no date.
- Meta: supports the idea, according to SiliconANGLE, but has not committed to implementing it.
What the signal doesn't do
None of this is final. The text is an individual IETF draft, not an adopted standard, and details may change. The token also only states a preference about use. It isn't a crawling rule, so bots can still visit your pages. Each vendor's existing controls stay in place.
The side effect on your visibility
Vendors won't all treat the signal the same way. Google plans to read it as disallowing Google-Extended unless you name Google-Extended explicitly. According to Google, only sites that specifically allow Google-Extended can appear in Gemini Apps responses and receive clicks via grounding. Microsoft keeps training and grounding separate: you remain eligible for Bing search after opting out, though Microsoft warns that opting out can reduce your visibility in AI experiences.
What this means for you
- You don't have to change anything today: wide support only starts in 2027, and existing per-vendor entries keep working.
- If you object to AI training, you get a simpler route, but it only covers future use and nothing technically enforces it.
- The real question is strategic: do you want to show up in AI answers or protect your content? Doing both takes clearly separated rules.
How to use this
Open yourdomain.com/robots.txt and check which AI crawlers are listed. Then decide, section by section, whether your site should appear in AI answers, and write that down before you add the new signal in 2027.
Luigi's take
For marketers this is good news, because a fiddly chore becomes a clear decision. Few companies have ever deliberately managed their robots.txt with AI in mind, even though it has become a positioning question: if you want to be named in AI answers, you need to be findable. That's why i4u.business deliberately allows AI crawlers. I want the tool pages to be cited in AI answers.
That isn't right for everyone. If you make money from exclusive content such as courses, studies or specialist writing, you may rightly decide otherwise. My advice: split your website in two. Keep open whatever is meant to make you known, and protect what you sell. And don't add the new signal until you understand what it does on Google and Bing. A quickly pasted one-liner can cost you more visibility than it protects.
Source
Google Search Central Blog: Supporting a shared AI training control for websites
More sources
- Bing Blog (Microsoft) – Termine, Syntax, Trennung von Training und Grounding
- IETF Datatracker – Entwurf vom 30.10.2026, noch kein Standard
- SiliconANGLE – Zeitplan OpenAI, Haltung von Meta
This article was researched and written with AI assistance. Responsible: Luigi Lugmayr.
