Six of the nine AI vendors we track publish one crawler token

What blocking training and allowing search actually requires

The recipe is a robots.txt edit, so its limits are the limits of the file format. RFC 9309 defines a robots.txt as a set of groups, each headed by one or more user-agent product tokens and holding allow and disallow rules that apply to crawlers matching those tokens. A crawler picks the group whose token matches it and ignores the rest. Nothing in the format lets you address a purpose, a product, or an intention. You address a name.

That means the recipe has a precondition that is almost never stated alongside it: the vendor must have published at least two names, and must have told you which of its activities each name governs. Given two names you can write two groups and the separation is real. Given one name you can write one group, and whatever that group says applies to everything the vendor does with the content it fetches under that name.

This is a different failure from the ones a scan usually surfaces. A file that returns the wrong status code is broken in a way you can see, and we have written before about how a 404 and a 503 on robots.txt mean opposite things under the RFC. A rule that your CDN overrides is a conflict between layers, which is the subject of why robots.txt is the weaker of the two layers that decide whether a crawler reaches you. Neither of those describes what happens here. Here the file is syntactically perfect, the rule is honoured, and the outcome is still not the one the recipe promised, because the distinction you were trying to express was never expressible.

The honest way to check it is not to look at your file at all. It is to look at the vendor's documentation and count the names.

Fifteen tokens, nine vendors, three purpose classes

The registry lives in core/src/bots.ts and is the same list that drives our AI crawler reference. Each entry carries a robots.txt product token, the vendor, one purpose class, the vendor's published request user agent where there is one, and a link to the documentation the entry was taken from. Counted on 1 August 2026 there are 15 entries.

The purpose field takes one of three values: training, search, or user_agent. Nine of the 15 tokens are classed training, three are classed search, and three are classed user_agent. Three of the entries carry no user agent string at all, because the vendor documents the token for robots.txt purposes but sends no matching header, which is why three of these tokens will never appear in your logs no matter how long you grep for them. Those three are anthropic-ai, Google-Extended and Applebot-Extended.

Two things about that purpose field are worth stating plainly, because it would be easy to read more into it than it holds. First, it is a decision we made, not something we measured. Assigning ClaudeBot to training is a transcription of what Anthropic writes, and assigning Amazonbot to training is a judgment call recorded in a comment in the file. Second, the class governs scoring: a separate constant lists which purposes cost Access score when disallowed, and it names search and user_agent only. Disallowing a training crawler is treated as a deliberate owner choice and costs nothing. That means six of the 15 tokens can cost score when blocked, and all six belong to OpenAI, Anthropic and Perplexity. None of the six single-token vendors has a token whose block affects the number we print.

The registry also assumes the request telling you which token it is can be believed, and that assumption deserves the caveat we gave it in full when we wrote that a user agent is a claim rather than an identity. Counting published names is not affected by that, since the count comes from documentation. Anything you infer from your own logs is.

What OpenAI, Anthropic and Perplexity actually document

Three vendors give you the two names the recipe needs, and all three say in their own words what each name is for. That wording matters more than the token count, because a name with no stated scope is not a control.

OpenAI's crawler documentation lists four crawlers. GPTBot is described as crawling content that may be used in training its generative AI foundation models. OAI-SearchBot is described as surfacing websites in search results in ChatGPT's search features. ChatGPT-User covers certain user actions in ChatGPT and Custom GPTs, and the same page states that because these actions are initiated by a user, robots.txt rules may not apply, adding that ChatGPT-User is not used for crawling the web in an automatic fashion and directing site owners to use OAI-SearchBot for managing search opt-outs and automatic crawl. That is the cleanest statement of the split any vendor makes, and it is the reason our page on getting cited in ChatGPT can give a specific instruction rather than a general one. We have separately counted the four IP range files OpenAI publishes for these crawlers.

One detail worth recording: the URL our registry carries for OpenAI, platform.openai.com/docs/bots, now returns a 301 to developers.openai.com/api/docs/bots. We observed that redirect on 1 August 2026. The content is the same and the old link still resolves, so nothing is broken, but a registry that stores documentation URLs is storing something that moves.

Anthropic's crawler article documents three: ClaudeBot, which it describes as collecting web content that could potentially contribute to training, Claude-User, which accesses websites when individuals ask Claude questions, and Claude-SearchBot, which it says navigates the web to improve search result quality. The article does not mention anthropic-ai anywhere. We carry that token because a great many robots.txt files in the wild still name it, not because Anthropic currently documents it.

Perplexity's bot documentation is the shortest and makes the strongest claim. PerplexityBot is described as designed to surface and link websites in search results on Perplexity, followed by an explicit sentence that it is not used to crawl content for AI foundation models. Perplexity-User is described as supporting user actions, visiting a page when a user asks a question and linking it in the response. A vendor stating in its own documentation that a named crawler is not used for training is exactly what makes an allow rule on that name a decision rather than a hope, and it is why our Perplexity guidance treats the two tokens differently.

For the other six vendors the choice is all or nothing

The remaining six vendors publish one token each, and the shape of the gap is different in each case.

Google is the one where the single token is least like a limit and most like a boundary. Google's crawler documentation, last updated 14 July 2026, describes Google-Extended as a standalone product token publishers use to manage whether crawled content may be used for training future generations of Gemini models and for grounding in Gemini Apps and Vertex AI, and states plainly that Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. So the training control exists and is clean. What has no separate control is the fetch itself, because the same page states that crawling preferences addressed to the Googlebot user agent affect Google Search including Discover and all Google Search features, as well as other products such as Google Images, Google Video and Google News. One name, many products. Googlebot is not in our registry at all, which we audited in full when we wrote about the one crawler our scanner does not model, and it is the reason our AI Overviews guidance has to talk about Googlebot rather than about an AI-specific token.

Apple's Applebot-Extended is a training opt-out that sends no user agent of its own, so it can be written into a file and never observed in a log. Meta-ExternalAgent and Bytespider are single tokens from Meta and ByteDance respectively; both vendors document them on hosts this site does not link to, so the pages are developers.facebook.com/docs/sharing/webmasters/crawler and zhanzhang.toutiao.com, given here as plain text.

Common Crawl's CCBot page is a genuinely different case, and the honest reading is that a single token is appropriate there. Common Crawl publishes an archive rather than a product. What it does with your page is put bytes in a public dataset, and who then trains on that dataset is not something Common Crawl decides or can offer you a token for. A one-token vendor is not automatically a vendor withholding a control.

Amazon is the case where our own file admits the model does not fit. Amazonbot is carried as purpose training, and the comment above the entry in core/src/bots.ts records why: Amazon documents Amazonbot as improving Alexa answers, which is a mixed search and training role, and it is classed as training so that blocking it alone does not take the search-class Access penalty. That is a scoring decision written down at the point it was made, and it is the single clearest example in the registry of a real crawler that the three-value purpose field cannot describe. One entry needed a judgment call and got one. The next section is about how common that shape is in aggregate.

Cloudflare reports mixed-use crawlers at over 36 percent of activity

Everything above is a count of documentation. It says the recipe is unwritable for six vendors, but it says nothing about how much crawler traffic those vendors represent, and Lantad has not measured that. We do not run a network, we scan individual sites on request, and a per-site scan cannot produce a share of global crawler activity. Somebody who does run a network published one.

Cloudflare's report of 1 July 2026, Content Independence Day, one year on, states that when looking at the crawlers Cloudflare identifies by purpose, 52 percent of crawler requests are now for AI training as of June 2026, up from 22 percent in Spring 2025. In the same paragraph it states that mixed-use crawlers, which it defines as those blending search, agent use and training, represent over 36 percent of activity, and that pure search crawling now represents a small and declining share of overall crawler activity despite remaining critical for publisher visibility. It gives no figure for that last share, so neither do we.

Those are Cloudflare's measurements on Cloudflare's network, and the qualifier is load bearing twice over. It is a sample of the web that sits behind one CDN, and it is a classification Cloudflare performs using its own bot identification rather than a neutral definition anybody else can reproduce. What it is not is an estimate. It is a network operator counting requests it terminates, which is a stronger position than either of us has for measuring anything else in this post.

Read against the count of published names, the two line up in a way that is uncomfortable for the recipe. The advice assumes crawler traffic separates into training and search, and the operator best placed to check reports that over a third of it is neither cleanly. Cloudflare's own product response has been to move the decision off robots.txt: we looked at what happens when its managed robots.txt writes a Content-signal line above ordinary Disallow rules, and the answer was that the signal is an unknown field to a standard parser while the Disallow rules do the work. Expressing a purpose in a file that only understands names remains the open problem, and a vendor that publishes one name has already resolved it in its own favour.

None of this makes blocking training pointless. For OpenAI, Anthropic and Perplexity the split is real, documented, and worth using. It means the sentence should carry its scope: block training and keep search where the vendor lets you, and know which vendors those are.

What to check on your own site this week

The useful version of this is not a template to paste. It is four questions, and three of them are answered by reading rather than by scanning.

Count the names before you write the rule. For every vendor whose crawler you intend to treat differently by purpose, open that vendor's own documentation and count how many product tokens it publishes and what it says each one governs. If the count is one, the rule you were about to write is an on or off switch, and you should decide knowing that. If the count is more than one, check that the vendor states which activity each token covers, because a second name with no stated scope is not a second control.

Then check what your file currently says about the six. A robots.txt written a year ago against a wildcard group is applying one decision to Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot together. That may well be what you want. It is worth being the thing you chose rather than the thing the file happened to do.

Separately, confirm that the rules you did write are the ones taking effect, which is a different question from whether they are correct. Our methodology page sets out how the access part of the score is computed and which purpose classes cost points when disallowed, so you can see what a blocked token does to the number before you change anything.

Finally, be careful what you conclude from a log. Three of the fifteen tokens send no user agent, so their absence proves nothing, and a present user agent is an unverified claim. The count that is solid is the one from the vendor's documentation page, and that count says the same thing today that it says in our registry: three vendors split, six do not.