BlogFindings
AI training opt out: Google and Microsoft named one robots.txt token, and 0 of 1,024 files carry it
Google and Microsoft each published plans on 30 October 2026 to honour a single shared robots.txt signal for AI training, written as the user-agent token __usage_ai-training. Google's crawler documentation puts its support in early 2027 and Microsoft's post names 1 February 2027. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 2 November 2026 and parsed the 1,024 files that resolved into at least one user-agent group. None named the new token, and none carried the Content-Usage line the IETF working group's own draft defines instead. 1,016 of the 1,024 already hold a wildcard group that RFC 9309 says a crawler reading the new token must obey.
That is a real change to the file this product reads on every scan, so it is worth being exact about what was announced and what is live. Nothing is live. Both companies published dates in 2027, the syntax is not in the working group draft that the announcements cite, and the token's shape has a consequence under the existing robots.txt standard that neither post discusses. Lantad has not measured anything either company will do, and this post does not pretend otherwise. What it can measure is the file the new token will land in, so it asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 2 November 2026 and read the result with the parser the scanner ships. This blog has reported the AIPREF work before, including that the standard was not deployable, and the useful question now is narrower: what does the arriving token mean for a file that was written years before it existed.
In short
- The AI training opt out that Google and Microsoft announced on 30 October 2026 is a robots.txt user-agent token written __usage_ai-training, and neither company honours it yet: Google's common crawlers documentation says early 2027 and Microsoft's post says it will begin honouring the new preferences as it crawls on 1 February 2027.
- Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 2 November 2026, as LantadBot with redirects followed and no JavaScript executed. 1,024 of the 1,101 files that answered HTTP 200 parsed into at least one user-agent group, carrying 63,799 Allow and Disallow rules, and not one named __usage_ai-training.
- Not one of those 1,024 files carried a Content-Usage line either, which is the robots.txt syntax draft-ietf-aipref-attach-05 of 19 August 2026 actually defines. The token the two vendors are shipping does not appear in that working group draft at all.
- Because the shared signal is a user-agent product token rather than a new directive, RFC 9309 section 2.2.1 says a crawler that finds no group naming it must obey the wildcard group instead. Lantad measured 1,016 of the 1,024 files carrying such a group on 2 November 2026, and 12 of those close the site root with Disallow: / .
- Google's announcement states the policy token acts as a Google-Extended disallow only when Google-Extended is not explicitly specified. Lantad found 159 of the 1,024 files naming Google-Extended on 2 November 2026, 91 of which allow it at the root, so those 91 have already written the line that governs instead of the shared token.
| Vendor | Published | Stated start | Legacy control it replaces |
|---|---|---|---|
| 30 October 2026 blog post | Early 2027 | Google-Extended, offered since 2023, continues to work alongside | |
| Microsoft | 30 October 2026 blog post | 1 February 2027 | NoArchive and NoCache, unread for training after 1 March 2027 |
| OpenAI | Post returned HTTP 403 to LantadBot | None in its crawler docs | GPTBot, documented with no reference to a shared token |
What did Google and Microsoft actually announce?
Google's post is titled Supporting a shared AI training control for websites and carries the date Friday, October 30, 2026. It states that the control is based on work from the AIPREF working group at the Internet Engineering Task Force, that existing company-specific controls will continue to function alongside the new joint signal, and that the proposal is consistent with the EU Code of Practice for General Purpose AI Model Providers under the EU AI Act, which it says recognises robots.txt as a primary method for web publishers to express machine-readable preferences. It calls the proposal a draft proposal in its own words. The operative sentence for anyone with an existing file is this one: when a website uses the policy token to opt out of training, and Google-Extended is not explicitly specified, Google plans to interpret that as if the publisher had disallowed Google-Extended.
Microsoft's post, written by Krishna Madhavan and dated the same day, goes further and publishes the syntax. It gives the robots.txt form as a user-agent group headed __usage_ai-training with ordinary Disallow and Allow rules beneath it, so a whole-site refusal is two lines and a path refusal names the path. It says training defaults to Allow when no applicable preference is published. It then adds a second control that is not shared and not in robots.txt: AI Grounding, set only at page level through a meta tag or an HTTP header, with Allow, Disallow and a new DisallowExtended that limits grounding to Microsoft-operated experiences and excludes grounding services Microsoft provides to third parties. Microsoft is explicit that these settings govern use of content, not crawl access, and that an Allow setting does not override a robots.txt rule blocking Bingbot from crawling. That distinction is the same one this blog made when Microsoft's earlier opt out turned out to be a meta tag rather than a robots.txt token.
Both posts credit the IETF working group, and that credit is where the detail stops matching. draft-ietf-aipref-attach-05, dated 19 August 2026 and intended as a Proposed Standard, defines a robots.txt rule named Content-Usage, with the example line Content-Usage: train-ai=n and an optional path prefix. It does not define or mention a user-agent token called __usage_ai-training anywhere. The vocabulary side of that work, which this blog covered when a third usage category arrived and search began overriding the other two, is still in discussion, and Google's post says as much: it describes the mechanism as bringing the benefit of the IETF training definition while multistakeholder discussions continue on the rest of the vocabulary. So the definition is borrowed from the working group and the carriage is not.
What the vendors are shipping
- User-Agent: __usage_ai-training
- Disallow: /
- A user-agent group, so RFC 9309 group
- selection applies to it
- Named by Google and Microsoft, 30 October 2026
What the working group draft defines
- Content-Usage: train-ai=n
- A rule inside a group, not a group heading
- Optional path prefix: Content-Usage: /ai-ok/ train-ai=y
- draft-ietf-aipref-attach-05, 19 August 2026
- Does not mention __usage_ai-training
Does the new AI training opt out work today?
No, and both vendors say so on their own pages. Google's common crawlers documentation carries a note stating that Google intends to support the AIPREF shared machine-readable training preference signal, with user-agent __usage_ai-training, in early 2027. Microsoft's post says it will begin honouring the new preferences as it crawls on 1 February 2027, and sets a second date of 1 March 2027, after which NoArchive and NoCache will no longer be read as a preference for training or grounding. A site that writes the new token today is writing a line that nothing reads until February 2027 at the earliest.
The adoption figure matches that. Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 2 November 2026, every request sent as LantadBot/1.0 with redirects followed and no JavaScript executed, which is the same single-request method described in the method behind every score and in the scanner's own crawler page. 1,101 hosts answered HTTP 200, 160 answered 403, 50 answered 404, 44 answered 503, and 46 did not answer at all, 21 with a connection or TLS failure and 25 timing out at fifteen seconds. The remaining 18 were split across 202, 429, 401, 406, 418 and 498, which accounts for all 1,419. Of the 1,101 that answered HTTP 200, 1,024 parsed into at least one user-agent group, and those files carried 63,799 Allow and Disallow rules between them.
Across those 1,024 files, the count naming __usage_ai-training is zero. The count carrying a Content-Usage line is also zero. Neither number is a surprise three days after an announcement, and neither is a criticism of the sites: there is nothing to gain yet from writing a token no crawler reads. They are worth recording because they establish the baseline against which the 2027 dates can be checked, and because a zero measured across a thousand real files is a different statement from an assumption that adoption is low. It is the same reason this blog measured the files when an EU draft made robots.txt evidence and found 4 of 1,419 sites serving the other file that draft names. A standard's deployment is a fact about servers, not about intentions, and it can be read.
Why does writing the opt out as a user-agent token change what existing files say?
A new directive would have been inert in an old file: a parser that does not know Content-Usage ignores the line, and a file that never carried one says nothing about training. A user-agent token behaves differently, because every robots.txt file already has an answer for every possible product token. RFC 9309, the robots exclusion protocol, says in section 2.2.1 that crawlers must use case-insensitive matching to find the group that matches the product token, and that if no matching group exists, crawlers must obey the group with a user-agent line with the asterisk value, if present. The token __usage_ai-training is valid under the standard's own syntax rule, which permits letters, underscores and hyphens, so it slots into that machinery rather than sitting outside it.
The consequence is measurable, so Lantad measured it. Evaluating the token __usage_ai-training against the site root of each of the 1,024 parsed files with the evaluator in core/src/robots.ts, the group that applies is the wildcard group on 1,016 files, and no group applies at all on the remaining 8, which RFC 9309 section 2.2.2 leaves allowed by default. Of the 1,016, 1,004 allow the root and 12 close it with Disallow: / . The 12 are spread across the government, education, news, SaaS and ecommerce strata, and 10 of them are sciencedirect.com, theregister.com, sap.com, instacart.com, wa.gov, amsterdam.nl, helsinki.fi, scielo.org, gmarket.co.kr and eluniversal.com.mx.
For those 12 files, two published statements point in opposite directions. Microsoft's post says training defaults to Allow when no applicable training preference is published, and none of the 12 has published one. RFC 9309 says a crawler that finds no group for its token must obey the wildcard group, and on these files that group forbids the root. Neither announcement says which reading wins, and the honest position is that the documents do not resolve it rather than that one of them is wrong. It is a narrow population, 12 of 1,024, and it is the sort of precedence question that turns out to matter: this blog found 343 of 716 files whose Allow lines changed no verdict at all once the rules were actually evaluated rather than read. Anyone who wants to see what their own file resolves to for a given token can run it through the robots.txt tester, and the industry split in who closes the root at all is in the measurement of which industries block AI crawlers.
Flow: robots.txt file, 1,024 parsed to Group naming the token? 0 files; Group naming the token? 0 files (none found) to Wildcard group? 1,016 files; Wildcard group? 1,016 files (allows /) to Root allowed, 1,004 files; Wildcard group? 1,016 files (disallows /) to Root disallowed, 12 files; Group naming the token? 0 files (no wildcard) to No group applies, 8 files.
What should you check on your own site before 2027?
The one item with a deadline attached is Microsoft's. Its post says that if a site currently uses NoArchive or NoCache it should publish the new preferences before 1 March 2027, and that after that date those two tags will no longer be read as a preference for training or grounding. The population that affects is small and this blog has already counted it: on 5 October 2026 Lantad found 7 of 1,077 home pages sending noarchive, the tag two AI vendors read as a training opt out. Seven sites is not a crisis, but for each of those seven the protection they wrote lapses on a published date and the replacement has to be written in a different place.
The grounding half deserves attention from anyone who cares about appearing in answers rather than about training. Microsoft's grounding preference is page-level only, with no site-wide robots.txt form, so expressing it across a large site means a meta tag or an HTTP header on every page rather than two lines in one file. Microsoft also says plainly that selecting Disallow or DisallowExtended for grounding will reduce brand visibility in AI experiences, and that publishing a setting is not the same as activating it because it must crawl the page to discover the setting. Those are the two facts to weigh before a site-wide decision gets made on a training question and silently applied to a grounding one.
Nothing above changes what is worth measuring today, which is whether a crawler can read the page at all. A training preference is a statement about use, and it operates on content a crawler has already fetched. OpenAI's crawler documentation, read on 2 November 2026, names GPTBot for training, OAI-SearchBot for search, ChatGPT-User for user-initiated fetches and OAI-AdsBot for checking submitted ad landing pages, and mentions no shared token at all; its only timing note is that search results can take about 24 hours to reflect a robots.txt change. OpenAI also has a post at openai.com/index/helping-websites-express-ai-training-preferences/, written here as plain text because nothing from it is quoted: the host answered LantadBot with HTTP 403 on 2 November 2026, so this post reports only what OpenAI's crawler documentation says. Against that, the gate that decides whether any of this matters is still the ordinary one: when Lantad asked the same 1,419 hostnames on 26 October 2026, 980 cleared every gate OpenAI's own documentation implies, 25 were closed by a robots.txt rule, and 217 permitted the crawler in the file and then answered the home page with HTTP 403. A shared opt-out token is a better way to say no. It does not help a site that is accidentally saying nothing.
- Writing __usage_ai-training today changes nothing yet Google's crawler documentation says early 2027 and Microsoft's post says 1 February 2027. Lantad found 0 of 1,024 corpus files naming the token on 2 November 2026.
- Check whether you rely on NoArchive or NoCache Microsoft says these stop being read as a training or grounding preference after 1 March 2027. Lantad found 7 of 1,077 home pages sending noarchive on 5 October 2026.
- Check whether your file names Google-Extended explicitly Google says the shared token acts as a Google-Extended disallow only when Google-Extended is not explicitly specified. 159 of 1,024 corpus files name it, 91 allowing it at the root, on 2 November 2026.
- Read what your wildcard group already says RFC 9309 section 2.2.1 sends an unmatched token to the wildcard group. 1,016 of 1,024 corpus files carry one and 12 of those disallow the root, on 2 November 2026.
- Grounding cannot be set in robots.txt at all Microsoft's grounding preference is page-level only, through a meta tag or an HTTP header, and it says a setting is not active until the page is recrawled.
- Confirm a crawler can read the page before tuning use preferences Of the same 1,419 hostnames on 26 October 2026, 980 cleared every gate OpenAI's documentation implies, 25 were closed by a robots.txt rule and 217 permitted the crawler then answered HTTP 403.
Lantad
Published .
An AI training opt out has until now meant one rule per company. Google's own AI crawler token for it, Google-Extended, has existed since 2023, and eight more training tokens sit alongside it in the registry this scanner ships. On 30 October 2026 Google and Microsoft each published a plan to replace that list with one line. Google's Search Central post, posted by John Mueller, says website owners will be able to use a shared policy token within robots.txt to set broad preferences across participating AI providers, and that it aims to support the control along with other AI providers in 2027.
Common questions
What is the new AI training opt out in robots.txt?
It is a single shared user-agent token, written __usage_ai-training, that a site heads a robots.txt group with and then writes ordinary Disallow or Allow rules beneath. Google and Microsoft each published plans to honour it on 30 October 2026, so one group can express a training preference to several AI providers instead of one rule per company. Microsoft's post states that training defaults to Allow when no applicable preference is published.
Does writing __usage_ai-training in robots.txt do anything yet?
Not yet. Google's common crawlers documentation says it intends to support the signal in early 2027, and Microsoft's post says it will begin honouring the new preferences as it crawls on 1 February 2027. OpenAI's crawler documentation, read on 2 November 2026, does not mention the token at all. Lantad found 0 of 1,024 corpus robots.txt files naming it on 2 November 2026.
Will the shared token override my existing Google-Extended rule?
No. Google's post states that the policy token is interpreted as a Google-Extended disallow only when Google-Extended is not explicitly specified, and that existing company-specific controls continue to function alongside the new signal. Lantad measured 159 of 1,024 corpus files naming Google-Extended on 2 November 2026, 68 disallowing it at the root and 91 allowing it, and for all 159 the explicit line is the one that governs.
Is the shared token the same as the IETF AIPREF syntax?
No, and the difference is in the carriage rather than the definition. draft-ietf-aipref-attach-05, dated 19 August 2026, defines a robots.txt rule named Content-Usage, with the example Content-Usage: train-ai=n, and it does not mention __usage_ai-training. Both vendor posts credit the IETF working group for the AI training definition while shipping a user-agent token the draft does not define. Lantad found 0 of 1,024 corpus files carrying either form on 2 November 2026.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
Answer Engine Optimization (AEO)
Answer Engine Optimization (AEO) is the practice of making a page readable, allowed, and structured enough that AI answer engines can use it as a source when they compose an answer.
Block AI training but allow AI search: the IETF vocabulary added a third category, and search now overrides the other two
draft-ietf-aipref-vocab-08, published 14 September 2026, replaces the two usage categories of revision 07 with three: train-ai, ai-use and search. Section 4.3 states that the search category overrides any usage falling into the other categories, which speaks directly to the request site owners make most often. None of it is deployable. The aipref charter set 31 August 2026 for both working group documents to reach the IESG, neither has been sent, and the vocabulary's own Note to Readers says its contents do not reflect consensus of the working group either in whole or part.
The AI preferences standard you cannot deploy yet
The IETF vocabulary for expressing AI usage preferences reached its sixth draft on 28 April 2026. The companion draft that defines where to write one down expired on 1 May 2026, so there is still nothing on your site for a crawler to read.
Does blocking AI training affect search rankings: 52 of 54 sites refusing Google-Extended still allow Googlebot
Cloudflare announced a Disallow AI Training setting and an Accountable designation on 15 September 2026, built on the promise that refusing training costs a site nothing in search. Lantad requested /robots.txt once from each of 1,027 hostnames on 16 September 2026. 700 files parsed into at least one group, 54 of them disallow Google-Extended at the site root, and on 52 of those 54 Googlebot is still allowed.
