Every robots.txt on the Web Says Nothing Yet

On 30 October, Kevin Kelley, a technical lead for standards at Anthropic, submitted an Internet-Draft to the IETF’s AI Preferences working group called “Machine-Readable Training Preference Signal”. The same day Google, Microsoft and OpenAI published posts saying they would support it, and The Verge reported that Anthropic had already adopted it. A publisher who wants to keep their pages out of AI training will no longer have to list every company’s crawler by name. One group in robots.txt is meant to cover all of them.

The most useful sentence I found wasn’t in the coverage. In his message to the working group’s mailing list, Kelley wrote that wildcard rules never match the new token, and “as a result, a robots.txt file that doesn’t use this mechanism, as is true of every one published today, expresses no preference at all.”

So on the day of the announcement, the number of sites that had opted out through the shared signal was zero. That is how the draft is designed, and I think the design is right. But it means the announcement is the start of a long adoption curve, and the four companies behind it have not agreed on when they start reading the signal or on what an opt-out covers.

What the draft actually adds

The mechanism is small. A site operator writes an ordinary robots.txt group whose User-Agent line names a “policy token” rather than a crawler:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

According to the draft, this expresses “a preference against AI training for all paths except those under /press-kit/, for which it expresses a preference in favor.” Lookup uses the path matching and longest-match rule that RFC 9309 already defines. The draft says it “introduces no new directive names and no new grammar”, so parsers that don’t know the token will skip the group as they would any group for an unknown user agent.

Four rules in the draft decide what a file means:

How a consumer reads the training preference in a robots.txt file A decision flow from draft-kelley-aipref-training-preference-signal-00. First question: does any group name the policy token __usage_ai-training? Wildcard groups do not count. If no, no preference has been expressed, which the author says is true of every robots.txt published as of 30 October 2026. If yes, look up the path with RFC 9309 longest-match rules. A disallowed path means a preference against AI training; an allowed path means a preference in favor. A separate note says the crawler's own product-token rules still govern fetching, unchanged. READING THE SIGNAL · DRAFT-KELLEY-AIPREF-TRAINING-PREFERENCE-SIGNAL-00 yes no Is there a group for __usage_ai-training? Look up the path with RFC 9309 longest match Disallowed: against Allowed: in favor No preference expressed Every file published as of 30 Oct 2026 Separately: rules for the crawler's own token still govern fetching, unchanged User-Agent: * groups never match the policy token. Vendor tokens such as Google-Extended keep their old meaning but do not count as a preference under this draft.
The left branch is the default for the whole web today. The draft treats silence as silence, not as consent or refusal.

The wildcard rule is the one I’d defend most strongly. If User-Agent: * with Disallow: / counted as a training opt-out, a great many files written for other reasons, such as staging servers, old blanket blocks and misconfigurations, would suddenly carry a legal-sounding preference their owners never chose. Reading only an explicit token means every preference under the scheme is one somebody actually wrote down.

Four companies, four start dates

The coverage made it sound like a joint launch. The primary documents describe four separate timetables.

Meta’s head of communications, Andy Stone, voiced support for a common signal, but according to The Verge he did not say whether Meta will honor it.

When each company says it will read the shared training token A timeline from October 2026 to October 2027. On 30 October 2026 the draft was submitted and Anthropic said it had started implementing it. OpenAI said coming months, with no date. Google said early 2027. Microsoft will honor the preferences from 1 February 2027 and stops reading NoArchive and NoCache as training or grounding preferences after 1 March 2027. The draft itself expires on 3 May 2027 unless it is revised. Microsoft's temporary transition grounding preference lasts until recrawl or 30 September 2027. ADOPTION DATES · AS STATED BY EACH COMPANY, 30 OCT 2026 Oct 26 Jan 27 Apr 27 Jul 27 Oct 27 Google: "early 2027" — Google crawler documentation, 30 Oct 2026 Google: "early 2027" 30 Oct 2026: draft submitted; Anthropic implementing — IETF ai-control list 1 Feb 2027: Microsoft honors new preferences — Bing blog 1 Mar 2027: NoArchive/NoCache no longer read for training or grounding — Bing blog 3 May 2027: draft-00 expires — IETF 30 Sep 2027: Bing temporary transition grounding preference ends at the latest — Bing blog 30 Oct 2026 Draft filed; Anthropic implementing 1 Feb 2027 Microsoft reads it 1 Mar Bing stops reading NoArchive/NoCache as AI preferences 3 May Draft -00 expires 30 Sep Bing transition ends OpenAI: "coming months", per The Verge; no date. Meta: support for the idea, no commitment to honor it.
Only Microsoft has put a date on it. A site that adds the token today is, for most of these companies, writing a note to be read later.

None of this is a scandal. A draft numbered -00 is, in the IETF’s own boilerplate, “inappropriate to use … as reference material or to cite them other than as ‘work in progress.’” Microsoft says as much: the working group’s efforts “continue toward a final standard,” but “we believe the current work represents significant progress that merits adoption.” Still, if you run a site and read a headline saying the biggest names in AI back a new opt-out, the honest version is that one company has started, one has a date, and two have a rough season.

The same Disallow means different things at Google and Microsoft

The draft takes its definition of training from the working group’s vocabulary document: “Using an asset to modify the learned parameters of a generative AI model.” That is narrow on purpose. It doesn’t cover retrieval, where a model reads a page at answer time and cites it. The industry calls that grounding.

Google’s post explains how it will translate the shared token: “When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended.” Google’s crawler documentation, though, says Google-Extended governs two things: training future Gemini models, “and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI.” The same post adds that “websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.”

Put those together and a publisher who writes the shared training opt-out, and nothing for Google-Extended, gets treated as having disallowed a token that also controls Gemini grounding. Google doesn’t spell out the grounding consequence in its post, and it says it will keep refining the approach. But going by its own documentation, an opt-out that the IETF text defines as being about training covers more at Google than training.

Microsoft went the other way. Its post says that “training and grounding are independent preferences” and that “your training setting does not change your grounding setting, and vice versa.” Training can be set in robots.txt or per page. Grounding is a separate, page-level Microsoft preference with three values: Allow, Disallow, and a new DisallowExtended that keeps content in Microsoft’s own AI products but out of the grounding services it sells to third parties. Both default to Allow.

What a shared training opt-out reaches at Google and at Microsoft A two-row comparison of a robots.txt group that disallows __usage_ai-training, with no vendor-specific token set. At Google the opt-out is read as disallowing Google-Extended, which by Google's documentation covers training Gemini models and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It does not affect AI Overviews and AI Mode, which have a separate Search Console control, or Google Search results. At Microsoft the opt-out covers training only. Grounding is a separate page-level preference that defaults to Allow, and Bing search eligibility is unaffected. ONE LINE, TWO SCOPES · "DISALLOW: /" UNDER __USAGE_AI-TRAINING, NO VENDOR TOKEN SET Model training Grounding in AI answers Search results Google Microsoft Covered: read as disallowing Google-Extended — Google Search Central blog, 30 Oct 2026 Covered, via Google-Extended Gemini Apps and Vertex AI grounding governed by Google-Extended — Google crawler documentation Gemini Apps, Vertex AI: under Google-Extended Google-Extended does not affect Search inclusion or ranking; AI Overviews and AI Mode use the Search Console control Not affected (AI Overviews: separate) Covered: AI Training preference in robots.txt — Bing blog, Oct 2026 Covered from 1 Feb 2027 Grounding is a separate page-level preference, default Allow — Bing blog Not affected: separate page-level setting Preferences do not change eligibility for Bing search results — Bing blog Not affected (Bing results) Dark cells: reached by the opt-out. The draft itself defines training as changing a generative model's learned parameters, and says nothing about grounding.
The token is shared. What it switches off is not. At Google the opt-out lands on a control that bundles training with Gemini grounding.

The draft allows this. Section 4.5 says consumers “MAY document that they interpret rules that match proprietary policy tokens as being more specific,” and that “private agreements MAY override preferences expressed via this mechanism.” A publisher who wants training off and Gemini grounding on can still get there by explicitly allowing Google-Extended and disallowing the shared token, and Google’s post says vendor controls “will continue to function alongside this new joint signal.” That works, but it brings back the per-vendor bookkeeping the shared token was supposed to remove.

The problem a shared name does solve

The draft does fix something real, and old. In 2007, Yang Sun and colleagues looked at 7,593 websites across education, government, news and business, and collected 2,925 distinct robots.txt files. They found that “the robots of popular search engines and information portals, such as Google, Yahoo, and MSN, are generally favored by most of the websites,” and reported “a strong correlation between the search engine market share and the bias toward particular search engine robots.” Their concern was a “rich get richer” web, in which access goes to whichever crawlers site owners happen to know by name.

Per-vendor AI tokens recreate that. If you have to know every company’s crawler token to state one preference, the preference reaches the companies you’ve heard of and misses the rest. A single, vendor-neutral token any new entrant can read is the right fix. Kelley’s message also says he spent a lot of time checking how deployed parsers behave, and that the draft “avoids relying on the areas where deployed parsers behave differently (and let me tell you, there are plenty!)”. A format this old needs that kind of care.

What the token can’t fix is enforcement. robots.txt has always depended on crawlers choosing to comply. In 2010, C. Lee Giles and colleagues measured crawler behavior against robots.txt rules and found that “most commercial web crawlers’ behaviors are ethical,” but that “many commercial crawlers still consistently violate or misinterpret certain robots.txt rules.” A preference signal that isn’t even a crawl directive asks for more trust than that. Lawyers are already arguing about what follows when a crawler ignores the file. Chien-Yi Chang and Xin He make the case in a 2025 paper that it can support contract or tort claims “under certain conditions.” Google’s post notes that the EU’s code of practice for general-purpose AI providers recognizes robots.txt as a primary way to express machine-readable preferences. The legal weight of this one line of text is going up, and it’s worth noticing who decides what the line means.

Katie MacKinnon and Emily Maemura argue that robots.txt “flattens consent into machine-readable parameters.” The Google-Extended mapping shows that the flattening happens on the reading side too. The publisher writes one word, Disallow. Each company decides how much of its own product line that word covers.

What I’d do with a site today

I would add the group. It costs two lines, deployed parsers that don’t know the token will ignore it, and the draft says it doesn’t change how any crawler’s own rules are read. Anthropic is reading it now, Microsoft will from 1 February, and the others say they’ll follow.

Then I’d check the vendor tokens I already have, because they keep their old meaning and some of them carry more than training. If I cared about showing up in Gemini answers, I’d add an explicit Google-Extended allow and accept that “one signal for everyone” now means one signal plus the exceptions. And I’d read the next revision of the draft before the current one expires on 3 May 2027, because the companies have committed to supporting a document that the IETF still calls work in progress.

References

  1. Kelley, K. (2026). Machine-Readable Training Preference Signal. Internet-Draft draft-kelley-aipref-training-preference-signal-00, IETF, 30 October 2026.
  2. Kelley, K. (2026). New draft: draft-kelley-aipref-training-preference-signal-00. IETF ai-control mailing list, 30 October 2026.
  3. Mueller, J. (2026). Supporting a shared AI training control for websites. Google Search Central Blog, 30 October 2026.
  4. Google (2026). Google’s common crawlers: Google-Extended. Google for Developers, accessed 31 October 2026.
  5. Google (2026). Search generative AI control. Search Console Help, accessed 31 October 2026.
  6. Madhavan, K. (2026). Your content, your choice: clearer preferences for training and grounding. Microsoft Bing Blogs, October 2026.
  7. OpenAI (2026). Making it easier for websites to express AI training preferences. OpenAI, 30 October 2026.
  8. The Verge (2026). The biggest names in AI are backing a new way to opt out of scraping. The Verge, 30 October 2026.
  9. Koster, M., Illyes, G., Zeller, H. and Sassman, L. (2022). Robots Exclusion Protocol. RFC 9309, IETF, September 2022.
  10. Sun, Y., Zhuang, Z., Councill, I. G. and Giles, C. L. (2007). Determining Bias to Search Engines from Robots.txt. IEEE/WIC/ACM International Conference on Web Intelligence (WI’07), 149–155, November 2007.
  11. Giles, C. L., Sun, Y. and Councill, I. G. (2010). Measuring the web crawler ethics. Proceedings of the 19th International Conference on World Wide Web, 1101–1102, April 2010.
  12. Chang, C.-Y. and He, X. (2025). The liabilities of robots.txt. Computer Law & Security Review, 58, 106176, September 2025.
  13. MacKinnon, K. and Maemura, E. (2025). Infrastructural consent: robots.txt as a protocol for automated data extraction. Information, Communication & Society, 29, 3326–3343, December 2025.