AI crawler list: 201 tokens on one blocklist, and 7 of the 9 files naming 50 or more refused every citation crawler

What is on the AI crawler list, and what is not

The file is one group. Read at source on 6 November 2026, it carries 204 lines beginning User-agent, then a single Disallow rule whose pattern is a slash, and nothing else. Under RFC 9309, the robots exclusion protocol, consecutive user-agent lines extend the same group, so the structure is correct: one group addressed to 204 named crawlers, refusing the whole site to all of them.

Those 204 lines resolve to 201 distinct tokens. Three are written twice in two letter cases, meta-externalagent alongside Meta-ExternalAgent, meta-externalfetcher alongside Meta-ExternalFetcher, and Webzio-Extended alongside webzio-extended. RFC 9309 section 2.2.1 requires case-insensitive matching on the product token, so each pair is one token stated twice and the second statement changes nothing. That is a harmless redundancy rather than a defect, and it is only visible if you apply the standard's own matching rule instead of counting lines, which is the habit this blog is built on.

The repository describes itself as a list of AI agents and robots to block, and its README adds that the list contains AI related crawlers of all types, regardless of purpose. That phrase is the whole finding of this post, stated by the authors before anyone measures anything. It is not a complaint about the project, which is explicit about what it is. It is a warning that the list was never built to preserve the distinction a site owner needs, and the project is reasonable to leave that to the person pasting it in. The file is hosted at github.com/ai-robots-txt/ai.robots.txt, named here in plain text because that host is not registered for outbound links on this site.

Set the list against the registry this scanner ships. Lantad evaluates 15 tokens, defined with a vendor and a purpose in core/src/bots.ts, and every one of the 15 is on the list. Running the comparison the other way is the more useful number: 186 of the 201 tokens are names this scanner does not evaluate at all. Some of those are crawlers with real documentation that this registry has simply not added. Others are agent strings with no published specification behind them. Either way the list is a superset of what any measurement on this site can speak to, including this one, and that limit belongs in the open rather than in a footnote. The vocabulary itself is covered in our note on what an AI crawler is.

Does anyone actually deploy a 201 token blocklist?

Mostly not, and the shape of the answer matters more than the headline. Of the 1,419 hostnames asked on 6 November 2026, 31 failed at the network level and 284 answered with a status other than 200. 1,104 answered HTTP 200, and 1,031 of those parsed into at least one user-agent group. The remaining 73 returned a body with no group in it, a population this blog has measured directly before when it found that 34 of 1,104 sites answered a crawler with an HTML page. That 1,031 is the denominator for everything below.

278 of the 1,031 files named at least one of the 201 tokens. 753 named none of them, so the median file in this corpus names nothing on the list, and the median token on the list appears in 6 files. Wholesale adoption is rarer still. Three files named 100 or more of the 201, six named between 50 and 99, and 111 named between 15 and 49. The long tail is the ordinary case: 93 files named between one and four tokens, which is a site that has heard of GPTBot and perhaps ClaudeBot and written two rules.

So the list is not sweeping the web. What it is doing is setting the vocabulary. The most named token in this corpus is the wildcard, which 1,023 of the 1,031 files carry. After it, the twelve most named tokens are led by GPTBot at 186 files and ClaudeBot at 177, and eleven of those twelve are on the list; the exception is adsbot-google at 168, an advertising crawler rather than an AI one. That ordering is the same one this blog found when it reported that GPTBot was the most blocked AI crawler in 82 of 718 files on a smaller frame in September. A list that supplies the names people write is influential whether or not anyone pastes it whole, and that influence is harder to see than an adoption rate.

Twenty three of the 201 tokens appeared in no corpus file at all. Among them is Claude-Agent, the token Anthropic documents for the browsers it hosts for Claude users, which this blog measured four days ago and found that 0 of 1,102 robots.txt files named it. The two measurements agree, which is the point of keeping the corpus stable. The others in that group of 23 are mostly coding agents and assistant strings, Codex, Kimi-Agent, Trae, qodercli, Lightpanda and ElevenlabsBot among them, names that exist on the list and nowhere in this corpus. A list that runs ahead of deployment is doing its job; it also means a token count is not a measure of risk.

What does blocking a citation crawler actually cost?

Three of the 15 tokens this scanner evaluates exist to put a site into an AI answer rather than into a training set, and all three are on the list. The vendors say so on their own documentation pages, each read at source on 6 November 2026.

OpenAI's crawler documentation states that OAI-SearchBot is for search, that it surfaces websites in ChatGPT's search features, and, in the sentence that settles the question, that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. The same page states that each setting is independent, so refusing GPTBot does not affect search visibility. Anthropic's help centre article separates ClaudeBot, which collects web content that may contribute to training, from Claude-SearchBot, which it describes as navigating the web to improve search result quality for users. Perplexity's bot documentation states that PerplexityBot is designed to surface and link websites in search results on Perplexity, that it is not used to crawl content for AI foundation models, and recommends allowing it in robots.txt so that a site appears in search results.

Read together, those three pages say the same thing: the training decision and the citation decision are separate, the vendors built separate tokens so a site can take them separately, and refusing the citation token removes the site from the answers. A list that groups all of them under one Disallow collapses that distinction back into a single choice.

The corpus wide picture is better than the list implies, and it deserves saying plainly before the bad news. 128 of the 1,031 files refused at least one training crawler at the site root on 6 November 2026. 78 of those 128 kept every citation crawler allowed, which is the split a site owner would pick deliberately. 50 refused at least one citation crawler, and 21 refused all three. So among sites taking any position on AI crawlers, roughly three in five hold the line that matters, which is consistent with the finding this blog published on a narrower frame that 52 of 54 sites refusing Google-Extended still allow Googlebot.

The per token counts show where the cost lands unevenly. PerplexityBot is refused at the root by 48 files, nearly twice the 27 refusing OAI-SearchBot and roughly double the 25 refusing Claude-SearchBot, despite Perplexity documenting it as the crawler that links a site rather than trains on it. Part of that is name shape: PerplexityBot reads like a company's main crawler, which is exactly the confusion a renamed or ambiguous crawler token produces in a file written from a list.

The nine files that named 50 or more of the tokens

This is the group where the list's purpose blindness stops being theoretical. Nine of the 1,031 files named 50 or more of the 201 tokens on 6 November 2026. Seven of the nine refused OAI-SearchBot, Claude-SearchBot and PerplexityBot at the site root: congress.gov, theglobeandmail.com, qcitymetro.com, nzherald.co.nz, amazon.com, semafor.com and cnn.com. Six of those seven are publishers or a legislature, which is the population this blog has repeatedly found doing most of the blocking, reporting that 42 of 77 files blocking a citation crawler were news sites on a stratified frame in September.

Two of the nine did not. slate.com named 75 of the tokens and refused none of the three citation crawlers, while refusing eight of the nine training crawlers. postman.com named 66 and refused one training crawler and no citation crawler. Those two files are the demonstration that a long token list and a preserved citation path are compatible. Whoever wrote them went through the names and made a decision per crawler rather than per vendor, and the result is a file that blocks training broadly and leaves every answer engine able to read and link the site.

One qualification matters and it limits what this section can claim. Lantad measured which tokens these files name and what each file's root verdict is. It did not measure where the names came from, and it cannot: a robots.txt file carries no provenance. A file naming 124 of the 201 tokens is strong circumstantial evidence of a list being used, and congress.gov is the file in question, but the inference is ours and not a measurement. Several of these files could have been assembled from vendor documentation, from a content delivery network's generated output, or from a different list entirely. This blog has published the managed variant of the same mechanism, noting that Cloudflare will generate your robots.txt and its token list changes without you, which produces similar files by a route nobody pastes.

What is measured, and is the thing worth taking away, is the correlation. Among the nine files carrying the most AI crawler names, seven have closed the site to every crawler whose vendor documents it as the one that cites. The more names a file carries, the more likely it is to have taken the citation decision without separating it out. You can check any single file a token at a time with the robots.txt tester, which evaluates a path per crawler rather than reporting whether the syntax parses.

How a rule you never wrote decides the verdict

A blocklist is not the only way to refuse a crawler you did not mean to refuse, and in this corpus it is not even the main one. Of the 50 files refusing at least one citation crawler at the root, 43 did it through a group that names the token. The other 7 never named the crawler anywhere: their refusal came from the wildcard group, which was written for something else and applied by default.

The mechanism is in the standard rather than in any vendor's page. RFC 9309 section 2.2.1 requires a crawler to find the group matching its product token using case-insensitive comparison, to combine the rules of matching groups into one, and, if no group matches, to obey the group whose user-agent line is a star, if one is present. Section 2.2.2 then resolves conflicts within the selected group by the most specific match, defined as the rule with the most characters. Nothing in that chain gives an unnamed AI crawler special treatment.

The consequence is that silence is not neutral. 1,023 of the 1,031 files in this corpus carry a wildcard group, so for all but 8 of them, any crawler the file does not name inherits whatever that group says. If the wildcard group closes the site, the crawler is closed out without appearing in the file. Lantad has measured this from the other direction and found that 559 of the 581 pages GPTBot lost were closed by a rule that never named it, and the same arithmetic governs the crawlers a site currently allows: a wildcard edit made for an unrelated reason moves them all at once.

That is the argument against treating a pasted list as the end of the job, and it is also the argument against treating its absence as safety. A file that names 50 AI crawler tokens has at least recorded a decision that can be read and revised. A file that names none, which is 753 of the 1,031 here, has handed every one of those crawlers to a single wildcard rule and will not show you that it did. Both failure modes end in a citation crawler being refused; only one of them leaves evidence. Our methodology describes how the scanner resolves this per token, and the AI crawler index lists the tokens it checks.

What this measurement does not cover

Four limits, and they are the reason this post reports a correlation rather than a cause.

The first is provenance, covered above and worth repeating because it is the easiest thing to misread here. No file in this corpus states where its token names came from. The relationship measured is between how many blocklist tokens a file names and what its root verdicts are, not between copying a list and losing citations.

The second is coverage. This scanner evaluates 15 tokens and the list carries 201, so 186 of them are names Lantad did not test a verdict for. The three citation crawlers in the registry are the three it can speak to with vendor documentation behind them, and a complete answer would need the same documentation check for every token on the list. That work has not been done here, and until it is, the citation counts in this post are a floor rather than a total. Our research index carries the published measurements that do exist.

The third is scope. Every verdict above is for the site root, the path a slash on its own. A file that allows a citation crawler at the root and disallows the paths where the content actually lives would read as allowed in this measurement and refuse the crawler in practice. Root verdicts are the right comparison across a thousand files and the wrong one for any single site, which is why a per path check exists and why this blog separately measured that 79 of 115 sites refused a crawler their robots.txt allows once the request was actually made.

The fourth is time. The list changes, and a file copied from it does not. Its first line is currently __usage_ai-training, the shared training preference token Google and Microsoft each published plans for on 30 October 2026, and which this blog measured on 2 November 2026 as present in 0 of 1,024 corpus files. Four days later exactly one file in this corpus carries it, svt.se, which names it with a comment citing the IETF draft and disallows it. The list was ahead of the web on that token and the web is beginning to follow. The reverse case is equally real and this blog has measured it: 134 of the 1,031 files still name anthropic-ai, a token absent from Anthropic's current documentation, which is the same rot found when 1,004 files named 2,209 tokens and 120 named one Anthropic no longer documents. A pasted list is correct on the day it is pasted. The file keeps the date and the web does not.

If you want the decision rather than the measurement: name the crawlers you mean to refuse, keep the three citation tokens out of that group unless you have decided you do not want to be cited, and write the citation tokens into groups of their own so the next wildcard edit cannot take them with it. The platform guides at how to get cited by ChatGPT and how to get cited by Perplexity set out what each engine needs after the robots.txt question is settled, and generative engine optimization covers where that sits in the wider problem.