BlogFindings
Crawler user agent: 1,004 robots.txt files named 2,209 tokens, and 120 name one Anthropic no longer documents
A user-agent line in robots.txt is an exact product token, and a token that matches nothing is a rule that does nothing. Lantad requested /robots.txt once from each of 1,419 hostnames on 17 September 2026. 1,004 files parsed into at least one group and named 2,209 distinct tokens between them, 1,098 of which appear in a single file, 813 of which contain a character RFC 9309 does not let a crawler put in its own name, and 120 of which name anthropic-ai or Claude-Web, neither of which appears on Anthropic's crawler page.
That makes the contents of these lines worth counting rather than assuming. On 17 September 2026 we requested /robots.txt over HTTPS, once, from each of the 1,419 hostnames in this repository's two committed corpus frames, identifying ourselves as LantadBot/1.0, following redirects, with a twenty second timeout, from one network location. 29 requests failed at the transport layer, 301 answered a status other than 200 and 38 answered 200 with a content type that was not text/plain, leaving 1,051 usable responses of which 1,004 named at least one user agent. Every figure below is out of those 1,004. No crawler was observed doing anything: this is a census of what files say, not a measurement of what any bot does.
In short
- A crawler user agent token is matched exactly and case-insensitively, not searched for: RFC 9309 section 2.2.1 states that the product token MUST contain only uppercase and lowercase letters, underscores and hyphens, and that crawlers MUST use case-insensitive matching to find the group that matches the product token.
- Lantad requested /robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus frames on 17 September 2026. 1,051 answered HTTP 200 with a text/plain content type and 1,004 of those parsed into at least one group, naming 2,209 distinct tokens between them.
- 1,098 of the 2,209 tokens appear in exactly one file, 996 of the 1,004 files name the wildcard, and 566 name the wildcard and nothing else, so the median file names one token.
- Anthropic's crawler page, dated 7 April 2026, names ClaudeBot, Claude-User and Claude-SearchBot and names neither anthropic-ai nor Claude-Web. 114 of the 1,004 files name anthropic-ai and 58 name Claude-Web, against 36 that name Claude-SearchBot.
- 813 of the 2,209 tokens contain a character the specification does not permit in a crawler's own name, among them 425 containing a space and 187 containing a slash, and 189 of the 1,004 files name at least one of them.
What is a crawler user agent token?
It is the name a crawler gives itself, and the specification is unusually strict about it. RFC 9309, the Standards Track document that defines the Robots Exclusion Protocol, opens section 2.2.1 with the sentence that decides everything else: crawlers set their own name, which is called a product token, to find relevant groups. The product token MUST contain only uppercase and lowercase letters, underscores and hyphens. Its ABNF is narrower still, admitting the hyphen, the underscore and the two letter ranges and nothing besides.
Two consequences follow and neither is obvious from reading a file. The first is that the token is the crawler's property rather than the site's. A site owner writing a name into robots.txt is guessing at a string a vendor published somewhere else, and nothing in the file can tell them whether the guess landed. The second is that matching is exact. The specification requires crawlers to use case-insensitive matching to find the group that matches the product token, and if no matching group exists, to obey the group with a user-agent line with the asterisk value, if present. Case is forgiven. Spelling is not.
Google's robots.txt documentation, carrying Last updated 2026-08-31 UTC, describes one refinement on top of that, which is that its crawlers find the group with the most specific user agent that matches, and that all non-matching text is ignored, so googlebot/1.2 and googlebot star are both equivalent to googlebot. That forgiveness is Google's implementation rather than a property of the protocol, and it applies to Google's crawlers. It is not a promise from anybody else.
The practical shape of the failure is quiet, which is why it needs counting rather than reasoning about. A group whose token matches no crawler is not an error. It costs nothing at parse time, it produces no warning anywhere, and the only way to see it is to compare the string against what the vendor publishes. If you want the per token verdicts for your own file before reading further, the robots.txt tester runs the same parser these figures came from, and the crawler registry lists the tokens each vendor documents.
Flow: user-agent: SomeName to Read as a product token; Read as a product token to Compared with the crawler's own name; Compared with the crawler's own name to Exact match, case aside; Compared with the crawler's own name to No match; Exact match, case aside to Obey that group; No match to Fall back to the wildcard group; Fall back to the wildcard group to No wildcard: no rules apply.
How many names does a robots.txt file address?
Usually one, and occasionally hundreds. The median file in this sample names a single token, and that token is the wildcard: 996 of the 1,004 files name the asterisk and 566 name it and nothing else. For more than half of these sites, therefore, robots.txt addresses no crawler by name at all, which is a defensible position and worth recognising as one. An earlier count found that 89 of 145 files ruled only on the whole site rather than on any narrower path, and the same bluntness shows up here on the other axis.
The mean tells a different story from the median because the tail is very long. Across the 1,004 files there are 2,209 distinct non-wildcard tokens, and 1,098 of them, very nearly half, appear in exactly one file. 297 appear in two. Only 60 appear in more than twenty. A list of names that is half unique is not a vocabulary that site owners share; it is a pile of strings that arrived by different routes and mostly stopped being maintained.
At the top of the tail sit files that have become registers. museodelprado.es names 639 tokens in 25,344 bytes, bhf.org.uk names 553, who.int names 527 and n26.com names 481. Those four files are among the longest in the sample by token count and, between them, they name exactly three of the 15 AI crawler tokens this scanner evaluates, all three of them the same one, CCBot. n26.com names none. Length here is not diligence: the biggest files are the oldest ones, and what they are long with is a list nobody has revisited.
The contrast is instructive rather than embarrassing. nzherald.co.nz names 388 tokens and all 15 of the registry tokens. france24.com names 323 and eleven of the 15. Both are news sites, which is consistent with an earlier finding that the sites that block AI crawlers are the ones with editors. A long file maintained by somebody who reads crawler documentation looks entirely different from a long file that has merely accumulated, and the token count alone cannot tell them apart. That is the whole argument for reading the names rather than counting the lines, and it is what our methodology does with every file it parses.
| Host | Tokens named | File bytes | Of the 15 AI crawler tokens |
|---|---|---|---|
| museodelprado.es | 639 | 25,344 | 1 |
| bhf.org.uk | 553 | 20,481 | 1 |
| who.int | 527 | 18,983 | 1 |
| n26.com | 481 | 14,556 | 0 |
| nzherald.co.nz | 388 | 44,665 | 15 |
| france24.com | 323 | 12,929 | 11 |
| healthline.com | 162 | 8,864 | 8 |
Do the names match a crawler the vendor still documents?
For the largest AI vendors this is checkable in a minute, because each publishes a page listing the tokens it uses, and we read all three on 17 September 2026. Anthropic's crawler page, which carries the date 7 April 2026, names three: ClaudeBot for collecting web content to improve its models, Claude-User for pages fetched because somebody asked Claude a question, and Claude-SearchBot for search result quality. It names neither anthropic-ai nor Claude-Web.
The files have not caught up. 114 of the 1,004 name anthropic-ai and 58 name Claude-Web, with 52 naming both and 120 naming at least one. Against that, 153 name ClaudeBot, 37 name Claude-User and 36 name Claude-SearchBot. So the two strings that do not appear on Anthropic's page are named in more files than the two that do, and 95 of the 120 files carrying a dropped token do not name Claude-SearchBot at all. This is the mechanism described in an earlier post, that a renamed crawler token leaves your robots.txt group matching nothing, showing up as a count rather than as an example.
The same gap runs the other way at OpenAI. OpenAI's crawler documentation names four tokens, GPTBot for training, OAI-SearchBot for surfacing sites in ChatGPT's search features, ChatGPT-User for user actions, and OAI-AdsBot for ads, and it carries no date on the page. 167 files name GPTBot, 98 name ChatGPT-User, 77 name OAI-SearchBot and four name OAI-AdsBot. 97 of the files that name GPTBot do not name OAI-SearchBot, so a site that wrote one rule about OpenAI has in most cases written a rule about training and said nothing about the crawler that decides whether it appears in an answer. Perplexity's documentation names two, PerplexityBot and Perplexity-User, which are named in 98 and 46 files here.
Being honest about our own position: this scanner's registry at core/src/bots.ts holds 15 tokens and anthropic-ai is one of them, kept deliberately with a comment marking it a legacy robots token with no published request user agent. Keeping it means a site that still writes that rule gets told what its rule does; it does not mean the token is current. The wider point is the one an earlier count made, that six of the nine AI vendors we track publish one crawler token and the separation between training and answering is newer than most of these files. A rule written before a vendor split its crawlers is not wrong so much as unaware, and what GPTBot sees on your own pages is the cheapest way to find out which side of that split you are on.
| Token | Vendor | On the vendor page | Files naming it |
|---|---|---|---|
| ClaudeBot | Anthropic | Yes | 153 |
| anthropic-ai | Anthropic | No | 114 |
| Claude-Web | Anthropic | No | 58 |
| Claude-User | Anthropic | Yes | 37 |
| Claude-SearchBot | Anthropic | Yes | 36 |
| GPTBot | OpenAI | Yes | 167 |
| ChatGPT-User | OpenAI | Yes | 98 |
| OAI-SearchBot | OpenAI | Yes | 77 |
| OAI-AdsBot | OpenAI | Yes | 4 |
| PerplexityBot | Perplexity | Yes | 98 |
| Perplexity-User | Perplexity | Yes | 46 |
Tokens the specification does not let a crawler have
813 of the 2,209 tokens contain at least one character outside the letters, underscore and hyphen that RFC 9309 permits in a product token, and 189 of the 1,004 files name at least one of them. Counting each token once, by the first disqualifying character class it carries, 425 contain whitespace, 187 contain a slash, 81 contain a dot, 77 contain a digit and no whitespace, slash or dot, and 43 carry something else.
The whitespace group is the largest and the clearest. A token such as sogou web spider, named in 16 files, offline explorer in 11, or chatgpt agent in five, cannot be a conforming product token, because a crawler is not permitted to have a space in its name. The slash group is the same mistake in a different direction: linkscan/8.1a unix, moget/2.1, lwp-trivial/1.34 and chatgpt-user/2.0 are user agent header fragments rather than product tokens, copied out of a log line where the version number belonged. Google's parser would read chatgpt-user/2.0 as chatgpt-user, as its page says, but that is one vendor's leniency and not a guarantee, and the token was still written wrong.
The digit group is where the finding cuts against the specification rather than against the files. Digits are not in the ABNF, and yet AI2Bot, named in 63 files here, is the token the Allen Institute publishes on its own crawler page at allenai.org/crawler, with a user agent string reading Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler). Ai2Bot-Dolma, img2dataset and MJ12bot are in the same position. Real crawlers carry names the standard says they may not have, so a file naming AI2Bot is doing the right thing by a rule the specification does not sanction, and treating the grammar as a validity test would flag the correct entries along with the broken ones. That is worth saying plainly rather than reporting 813 as a defect count.
The remaining 43 are neither. go!zilla appears in five files, lwp::simple and www::mechanize in three each, and then there is a group that misreads the field entirely: dotbot star, star aihitbot star, star mlbot star and, in one file, the string dollar-sign llm. Those are glob patterns written into a field that takes no patterns. As Google's page notes in passing, trailing wildcard text is simply ignored by its parser, which means a site writing star aihitbot star has written a rule that Google reads as one thing, that a strict parser reads as a name no crawler has, and that the author almost certainly believed was a pattern. The same misreading is why a wildcard in the path has no effect at the end of a rule, and it is a good reason to check the file against a parser rather than against intuition.
Lantad
Published .
A robots.txt rule is addressed to a name. The crawler user agent token on the left of the group decides which crawler reads the Disallow lines under it, and the protocol gives that name no tolerance at all: it is compared, whole and case-insensitively, against the token the crawler set for itself. Get it right and the rule applies. Get it wrong by one character and the file still parses, the page still validates, and the group is simply never selected by anything.
Common questions
What is a crawler user agent token in robots.txt?
It is the name a crawler sets for itself, and it is the value on a user-agent line. RFC 9309 section 2.2.1 states that the product token MUST contain only uppercase and lowercase letters, underscores and hyphens, and that crawlers MUST use case-insensitive matching to find the group that matches it. Case does not matter; spelling does.
Does a misspelled user agent token cause an error?
No, and that is the problem. A group whose token matches no crawler parses normally, reports nothing, and is never selected. Across the 1,004 files measured on 17 September 2026, 813 of the 2,209 distinct tokens carried a character the specification does not permit in a crawler's name, and 189 files named at least one of them.
Is anthropic-ai still a valid token to block?
Anthropic's crawler page, dated 7 April 2026, names ClaudeBot, Claude-User and Claude-SearchBot, and names neither anthropic-ai nor Claude-Web. A rule naming anthropic-ai costs nothing and does nothing on current documentation. 114 of the 1,004 files measured still name it, against 36 that name Claude-SearchBot.
Should I copy an AI crawler blocklist into robots.txt?
A pasted list is what produces these figures: 81 of the 339 files naming three or more tokens carry one of nine identical token sets, and one set of 40 files came from a platform default. Read each vendor's own crawler page instead, since that is the only place the current token is published, and check the result with a parser.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
A renamed crawler token leaves your robots.txt group matching nothing
Google replaced Google-NotebookLM with Google-GeminiNotebook on 16 July 2026 and supports the old name only until August 2026. A group headed by a name no crawler answers to falls through to the wildcard, or to no rules at all, and nothing on your site reports it.
Most blocked AI crawler: GPTBot in 82 of 718 robots.txt files, and its search twin in 24
Lantad requested /robots.txt once from each of 1,027 hostnames across eight industry sectors on 12 September 2026, and 718 of the responses parsed as robots.txt. GPTBot was disallowed at the site root by 82 of them, seven more than the fourth placed token. The wider gap is inside each vendor: 59 of the 718 refused GPTBot while leaving OAI-SearchBot allowed.
GPTBot detection: 6 of 15 crawler tokens publish a User-Agent to match
A robots.txt product token and an HTTP User-Agent header are two different strings, and only the second one is what your access log holds. Read at nine vendors' own documentation on 27 August 2026, 6 of the 15 tokens in Lantad's crawler registry carry a complete User-Agent string that can be matched literally.
