# Block AI training but allow AI search: the IETF vocabulary added a third category, and search now overrides the other two > draft-ietf-aipref-vocab-08, published 14 September 2026, replaces the two usage categories of revision 07 with three: train-ai, ai-use and search. Section 4.3 states that the search category overrides any usage falling into the other categories, which speaks directly to the request site owners make most often. None of it is deployable. The aipref charter set 31 August 2026 for both working group documents to reach the IESG, neither has been sent, and the vocabulary's own Note to Readers says its contents do not reflect consensus of the working group either in whole or part. - Canonical page: https://lantad.co/blog/block-ai-training-allow-ai-search-vocab-08-third-category - This file: https://lantad.co/blog/block-ai-training-allow-ai-search-vocab-08-third-category.md - Last substantive update: 2026-10-20 ## Key facts - **Published:** 2026-10-20 - **Category:** Findings - **Author:** Lantad - **Length:** 3715 words - **Takeaway 1:** Block AI training but allow AI search is now a sentence the IETF vocabulary can express in three parts rather than two: draft-ietf-aipref-vocab-08, published 14 September 2026, defines train-ai, ai-use and search, where revision 07 of 19 August 2026 defined only train-ai and search. - **Takeaway 2:** Section 4.3 of revision 08 states that the search category of use overrides any usage that falls into other categories, naming AI Training and AI Use specifically, so a search application's use of an asset is not also governed by whatever the site said about training. - **Takeaway 3:** The new middle category, ai-use, is defined as using an asset as input to a generative AI model where the asset is not directly provided by the user, which is the retrieval step between training and search that neither earlier revision named. - **Takeaway 4:** Neither draft is deployable on 20 October 2026. The aipref working group charter set 31 August 2026 for both documents to reach the IESG, the vocabulary's IESG state on the IETF datatracker is still I-D Exists, and its Note to Readers states that its contents DO NOT REFLECT CONSENSUS of the Working Group either in whole or part, with sections 3 and 4 carrying their own notes to the same effect. - **Takeaway 5:** What a site can deploy today is one vendor token at a time. OpenAI's bots documentation states that each setting is independent of the others, and Google's common crawlers page, carrying Last updated 2026-10-15 UTC, states that Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. ## Summary The request a site owner makes most often about AI is not block everything and it is not allow everything. It is this: block AI training but allow AI search, keep the referral traffic and the citations, refuse the model weights. Until last month the standard being written to express that request could not express it cleanly, because it recognised only two kinds of use. On 14 September 2026 that changed. [draft-ietf-aipref-vocab-08](https://datatracker.ietf.org/doc/html/draft-ietf-aipref-vocab-08) now defines three usage categories instead of two, and the third one sits exactly where site owners have been arguing. This post reports a development in somebody else's document and makes no new measurement of its own. The IETF AI preferences working group is drafting two specifications: a vocabulary for what you are allowed to say, and a companion document for where you write it down. Lantad has read each revision since the sixth as it appeared, including [the revision where preferences stopped inheriting](https://lantad.co/blog/ai-usage-preferences-stopped-inheriting) and [the one that still could not name a party](https://lantad.co/blog/ai-usage-preferences-cannot-name-a-party), and the honest summary each time has been that the thing is not deployable yet. That is still the summary. What is new is that the vocabulary has grown a category, the categories now have a precedence rule between them, and the deadline the working group set itself for finishing has passed without either document moving. All three facts are read from the drafts and the datatracker rather than from anybody's summary of them. ## Can I block AI training but allow AI search? Today, yes, imperfectly, and not through any standard. You do it one vendor at a time, by naming that vendor's product tokens in robots.txt and giving them different verdicts. OpenAI publishes four tokens and says of them, in its own [crawler documentation](https://developers.openai.com/api/docs/bots), that each setting is independent of the others, and it gives allowing OAI-SearchBot while disallowing GPTBot as its worked example. Google publishes one token for the purpose, Google-Extended, and states on its [common crawlers page](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) that Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. Two vendors, two mechanisms, two vocabularies, and nothing that generalises to the third vendor. That is what the IETF work is for. A single machine readable sentence, read by every crawler, that says which kinds of use you permit, so that the answer does not have to be rewritten each time a company ships a new bot. Both halves of it are still Internet-Drafts. The vocabulary defines the categories and values; the companion document defines the robots.txt rule and the HTTP header that carry them. Neither is an RFC, no crawler is obliged to read either, and the earlier post here called it [the standard you cannot deploy yet](https://lantad.co/blog/the-ai-preferences-standard-you-cannot-deploy-yet) for that reason. Revision 08 does not change that conclusion. It changes what the sentence would mean once you could write it. The interesting part for anyone who has actually tried to express this preference is that revision 08 does not simply add a category and leave the rest alone. It adds a precedence rule, and the rule runs in the direction that favours search. Section 4.3 states that the search category of use overrides any usage that falls into other categories, and it names AI Training and AI Use as the categories it overrides. So the three-way split is not three independent switches. It is two switches and an exemption, and the exemption is the one that most sites want granted. Whether that is the right design is a question for the working group, and the draft is explicit that the question is not settled. What it is not is a detail: it decides what a site has agreed to when a crawler's use of a page falls into more than one category at once, which is the normal case rather than the edge case for a product that both indexes and summarises. For the practical question underneath all of this, the useful instrument remains the file you already have. A robots.txt rule is the only thing in this area that changes what a crawler actually fetches today, which is why it is worth reading yours per crawler rather than per file, and the [robots.txt tester](https://lantad.co/tools/robots-txt-tester) resolves it one product token at a time the way [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) says a crawler must. Nothing in a draft changes that, and nothing in this post is a recommendation to wait for one. ## What revision 08 added that revision 07 did not have The count is the simplest way to state it. Revision 07, dated 19 August 2026, listed two categories in its table: AI Model Training carrying the label train-ai, and Search carrying the label search. Revision 08, dated 14 September 2026, lists three in section 4: AI Training with the label train-ai, AI Use with the label ai-use, and Search with the label search. The new one is in the middle, and it is defined as using an asset as input to a generative AI model, where the asset is not directly provided by the user. Read that definition carefully, because it names a thing that has been missing from this argument for two years. Training is a one-off act on a corpus. Search is a pointer back to your page. Between them sits the case where a model fetches your page at the moment somebody asks a question, puts your words into its context window, and answers from them. That is not training, because no parameter is modified. It is also not search in the draft's sense, because the primary purpose is not to select assets and send the user to their original location. It is the mechanism behind most of what an [AI crawler](https://lantad.co/glossary/ai-crawler) does when a person is waiting for an answer, and until revision 08 the vocabulary had no label for it at all. The qualifier in the definition is doing real work too. The asset is not directly provided by the user excludes the case where somebody pastes your page into a chat window, which is the same line OpenAI draws in its own documentation when it says of ChatGPT-User that because these actions are initiated by a user, robots.txt rules may not apply. A standard that tried to govern what a person does with a page they are reading would govern nothing, because there is no request for a server to see. Confining ai-use to fetches the user did not make keeps the category inside the set of things a site can actually express a preference about. What revision 08 does not change is the value set or the resolution rule, and both matter to anyone reading the draft for the first time. The serialised form permits exactly two values, y and n, encoded as Structured Fields tokens; any other value, and an absent key, resolve to unknown. And since revision 07 deleted the inheritance paragraph, unknown stays unknown rather than falling back to a broader category. So a site that writes train-ai=n and says nothing about ai-use has not thereby refused ai-use. It has said nothing about it, and the [earlier post on that deletion](https://lantad.co/blog/ai-usage-preferences-stopped-inheriting) is the detail. Three categories with no inheritance means three statements to make, not one. ## Where you would write the preference down, and the gap left open A vocabulary with nowhere to put it is a word list, so the working group's second document defines two carriers. [draft-ietf-aipref-attach-05](https://datatracker.ietf.org/doc/html/draft-ietf-aipref-attach-05), dated 19 August 2026, adds a Content-Usage rule to robots.txt groups and defines a Content-Usage HTTP response header field. The header is a Structured Fields dictionary and applies to the content of the HTTP message rather than to the resource as a whole, with Content-Usage: train-ai=n as the draft's own example. The robots.txt rule takes an optional path pattern followed by a usage preference, so Content-Usage: train-ai=n states a group-wide preference and Content-Usage: /ai-ok/ train-ai=y states one for a subtree. Path matching uses the same longest-prefix rule as Allow and Disallow, and the draft says a preference applies only to paths a crawler is permitted to fetch in the first place. That last clause is the one worth holding onto, because it settles the relationship between the two layers rather than leaving it to interpretation. A Content-Usage line is not an access control and never overrides a Disallow. It describes what a crawler may do with bytes it was already allowed to have. A site that disallows a crawler and then grants it search=y has granted nothing, and a site that wants the crawler to read the page and limit what it does afterwards has to leave the path open. Two layers, in that order, which is the same split an [AI visibility](https://lantad.co/glossary/ai-visibility) measurement has to respect: access first, then what happens to what was accessed. What the attachment draft does not do is set a precedence rule between its own two carriers. It offers them as alternatives or as additions, in its own words, and it does not say which wins when the robots.txt rule and the HTTP header disagree. For a static site that is a non-problem, since one person writes both. For anything behind a CDN it is a real gap: the header is the layer an edge rewrites, the file is the layer the origin serves, and the two are routinely maintained by different teams with different release cadences. A reader deploying this on the day it becomes deployable would have to pick a convention and hope the crawler picked the same one. The earlier reading of [the Content-Signal approach](https://lantad.co/blog/content-signal-asks-disallow-blocks) ran into the same shape of problem from the other direction, where a declaration and an access rule sat in one file and said different things. None of this is yet something to implement, and the point of reading it now is narrower: it tells you what shape the eventual answer will have, which is a per-category statement attached to a path, carried in a file or a header, and subordinate to the access decision. If that shape holds, then the work a site does today on its robots.txt is not wasted when the standard lands, because the access layer underneath is the same layer. The [crawler directory](https://lantad.co/tools/ai-crawlers) and the [method notes](https://lantad.co/research) here are both built on that assumption. ## The deadline the working group set itself has passed This is the part that a reader deciding whether to plan around the standard needs, and it is not in either draft's text. It is on the working group's own page. The [aipref charter](https://datatracker.ietf.org/wg/aipref/about/) lists two milestones, both dated 31 August 2026: a standards track specification describing vocabulary for expressing AI-related preferences to the IESG for publication, and standards track protocol specifications to associate the AI-related preferences to the IESG for publication. On 20 October 2026 neither milestone is marked done. The vocabulary's IESG state on the [datatracker document page](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/) reads I-D Exists, which is the state a draft is in before it has been sent anywhere, and its expiry is 18 March 2027. The attachment draft expires 20 February 2027. Both are active Internet-Drafts with an intended status of Proposed Standard, and both are seven weeks past the date the charter set for handing them over. The draft is candid about why. Immediately after the About This Document block, the vocabulary carries a Note to Readers stating that its contents DO NOT REFLECT CONSENSUS of the Working Group either in whole or part, and sections 3 and 4 carry their own notes saying those sections lack consensus. Section 4 is the one that defines the three categories this entire post is about. So the headline development here, a third usage category and a precedence rule that resolves the most common request a site owner makes, is also text that the working group has not agreed on, published in a document that is late. Both things are true and a reader should hold them together rather than picking one. It would be convenient for a company selling measurement if the standard were nearly finished, and it is not. The group is chaired by Mark Nottingham and Suresh Krishnan in the Web and Internet Transport area, the drafts are being revised steadily rather than abandoned, and revision 08 arriving three and a half weeks after revision 07 is the behaviour of a live document. But a category added without consensus can be removed without consensus, and anybody writing a product roadmap around train-ai, ai-use and search as a stable triple is reading more into revision 08 than it claims for itself. The two individual submissions covered here earlier, on [grant binding and the inability to name a party](https://lantad.co/blog/ai-usage-preferences-cannot-name-a-party), are still individual submissions and have not been adopted. The practical reading is to treat this as a forecast rather than a plan. If the shape survives, a site will eventually have one sentence to write instead of a token list per vendor. Until then the token list per vendor is the whole of the mechanism, and the sites getting it wrong are getting it wrong at the access layer rather than the preference layer, which the next section is about. ## What you can deploy today, and what Lantad has already counted The deployable mechanism is per vendor and it is unglamorous. OpenAI documents four product tokens with distinct jobs: GPTBot for content that may train foundation models, OAI-SearchBot for showing sites in ChatGPT search answers, OAI-AdsBot for checking landing pages submitted as ads, and ChatGPT-User for fetches a person triggered. Its documentation states that each setting is independent of the others, which is the vendor-specific version of what the vocabulary is trying to make general, and it adds that where a site allows both bots it may use the results from just one crawl for both use cases. Google's single token works differently again: Google-Extended has no separate HTTP request user agent string at all, so it is a declaration attached to crawls Google was already making rather than a crawler you can see in a log. The split between those two designs is the reason a generic answer has been hard to write, and it is covered per platform on the [how a crawler reaches ChatGPT](https://lantad.co/how-to-get-cited/chatgpt) page. How well does the per vendor mechanism work in the field? Lantad has measured that three times and the answer each time is that most sites are not using it. On 28 September 2026 we parsed 1,025 robots.txt files from this repository's committed corpus and found 172 naming GPTBot, 103 naming ChatGPT-User, 80 naming OAI-SearchBot and 4 naming OAI-AdsBot, with [38 files refusing GPTBot at the root while letting ChatGPT-User in](https://lantad.co/blog/block-chatgpt-38-files-refuse-gptbot-allow-chatgpt-user). On 16 September 2026 we parsed 700 files from the industry half of the corpus and found 54 disallowing Google-Extended at the site root, with [52 of those 54 leaving Googlebot allowed](https://lantad.co/blog/does-blocking-ai-training-affect-search-rankings-52-of-54), which is the train-and-search split being made deliberately and correctly by a few dozen sites. And on 17 October 2026, across 1,008 parsed files, [81 allowed Googlebot at the root and refused GPTBot](https://lantad.co/blog/googlebot-vs-gptbot-81-of-1008-files-differed) while 925 allowed both and not one refused Googlebot while allowing GPTBot. Those three counts describe the same situation from three angles. A small, consistent minority of sites, in the tens rather than the hundreds, are expressing the preference this post is about. The large majority express nothing, and because a crawler whose token matches no group falls through to the wildcard group, expressing nothing resolves to whatever the wildcard says, which on most sites is allow. That is the gap the standard would close, and it is also why closing it is worth doing: the current mechanism asks every site to track every vendor's token list and update it when a vendor renames one, which is a maintenance task nobody has signed up for. So the answer to the question in the title, as of 20 October 2026, is that you can block AI training and allow AI search, you have to do it per vendor rather than per preference, and the general mechanism that would replace the per vendor work exists only as two late Internet-Drafts whose own text disclaims consensus. If you want to know what your file currently says to each of these crawlers rather than what you intended it to say, that is a reading of your own robots.txt one product token at a time, as [LantadBot](https://lantad.co/bot) does it, and it is the part of this that does not need a standard to be useful. ## Questions and answers **Can I block AI training but allow AI search?** Yes, per vendor, using robots.txt product tokens rather than any standard. OpenAI's crawler documentation says each setting is independent of the others and gives allowing OAI-SearchBot while disallowing GPTBot as its example. Google's common crawlers page, carrying Last updated 2026-10-15 UTC, states that Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. There is no single cross-vendor sentence that expresses the preference today, which is what the IETF aipref work is for. **What is the ai-use category in draft-ietf-aipref-vocab-08?** It is the third usage category, new in revision 08 of 14 September 2026, defined as using an asset as input to a generative AI model where the asset is not directly provided by the user. It names the retrieval step that happens when a model fetches a page to answer a question, which is neither training nor search. Revision 07 of 19 August 2026 defined only train-ai and search and had no label for it. **Does search override a train-ai preference under the new draft?** Under revision 08, yes, within the draft's own terms. Section 4.3 states that the search category of use overrides any usage that falls into other categories, naming AI Training and AI Use specifically. The draft's Note to Readers also states that its contents do not reflect consensus of the working group either in whole or part, and section 4 carries its own note to the same effect, so the rule is proposed text rather than settled text. **When will the IETF AI preferences standard be finished?** No date can be given from the published record. The aipref charter set 31 August 2026 for both working group documents to reach the IESG for publication and neither milestone is marked done. On 20 October 2026 the vocabulary's IESG state on the datatracker reads I-D Exists and the draft expires 18 March 2027, while the attachment draft expires 20 February 2027. Both are active and being revised, but neither has been sent to the IESG. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot