ClaudeBot robots.txt: 159 of 1,014 files named it, and on 43 the name sits in the wildcard's own group

What does a ClaudeBot robots.txt rule control, and what does it not?

Anthropic documents its crawling on one help centre page, and the page is built around a three column table: the bot, what it is for, and what happens when you disable it. That third column is the part a publisher should read twice, because the three rows do not say the same kind of thing. For ClaudeBot the page states that when a site restricts access, it signals that the site's future materials should be excluded from our AI model training datasets. No consequence for being found, cited or answered is stated anywhere in that row. For Claude-User the page states that disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search. For Claude-SearchBot it states that disabling it prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.

So Anthropic has written the block training and allow search advice into its own documentation, with the two halves named. A ClaudeBot robots.txt rule is the training half. The cost it carries, on the vendor's own account, is a training opt out and nothing else, and a site that wants to be present in Claude's answers while staying out of its training sets has a documented way to say so. The same page also tells owners not to try the shortcut: it states that alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file. This blog read the address list that page links on 7 September 2026 and found one file of 26 prefixes covering all three bots with no per-bot structure, which is the practical reason the file is the only place the distinction can be drawn.

Anthropic is the most generous of the three vendors that run a user-triggered agent, and saying so is the difference between reporting and campaigning. OpenAI's bots documentation states that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. Perplexity's documentation says of its own user fetcher that since a user requested the fetch, it generally ignores robots.txt rules, which this blog counted against the same corpus when 102 of 1,013 files named PerplexityBot. Anthropic's page makes no such reservation for Claude-User: it says the rule works and tells you what it costs. That is the most useful promise of the three, and it is the one the files in this corpus most often fail to act on.

What was measured on 18 October 2026, and what was not

The population is the 1,419 hostnames in the two committed corpus seed files in this repository: worker/seeds/corpus-seeds-platform.json with 392 hostnames across ten platform strata, and worker/seeds/corpus-seeds-industry.json with 1,027 across eight industry strata. Each hostname was asked once for https://hostname/robots.txt, as LantadBot/1.0, from one network location, with redirects followed and a twenty second timeout. That user agent is the one described on the bot page, and it was never altered to impersonate a crawler belonging to anybody else.

The funnel sets every proportion below, so it is worth stating in full. 1,093 hostnames answered HTTP 200 and 7 answered 202, giving 1,100 in the 2xx range. 37 of those served an HTML document at the robots.txt address rather than a robots file, a defect this blog has counted across the same corpus before, and they were excluded. 49 more parsed into no user-agent group at all, usually a file holding only a Sitemap line or a comment, and a file with no group can name no token, so those were excluded too. That leaves 1,014, the denominator for every file count in this post. Of the rest, 175 hostnames answered 403, 51 answered 503, 49 answered 404, 5 answered 429 and 3 answered 401, with four single instances of other statuses, and 32 requests failed before any status arrived, 9 of them by timeout. A 403 on the robots file is a firmer refusal than anything the file could have contained, the same asymmetry behind 92 of 1,056 sites refusing GPTBot the file.

Parsing and evaluation used parseRobotsTxt and evaluateRobots from core/src/robots.ts, the code that ships in the product rather than a script written for this post. A file counts as naming a token when a user-agent line lowercases to that exact string. A root verdict is the answer evaluateRobots returns for the path slash, which is the strongest form of the question, since a file can leave the root open and still close most of what matters. The approach is the one set out on the methodology page and used for the crawlability study, and a reader can run the same evaluation on one file with the robots.txt tester, which answers per token rather than reporting the file as a whole.

Three limits apply to all of it. No Anthropic crawler was observed doing anything, because there are no server logs here, so every statement about what Claude does is a statement about what Anthropic publishes. One URL per hostname was requested, at the apex, so nothing here covers subdomains, where 73 of 277 gave an AI crawler a different answer from their apex. And these 1,419 front doors come from two committed sampling frames built for another purpose, not from the web, so no proportion here generalises past them.

159 files named ClaudeBot, and 75 closed the root to it

159 of the 1,014 files named ClaudeBot, 44 named Claude-User and 42 named Claude-SearchBot. 162 named at least one of the three and 36 named all three. For scale on the same files, 171 named GPTBot, 144 named Google-Extended, 139 named CCBot, 128 named Bytespider, 119 named Meta-ExternalAgent, 113 named Applebot-Extended, 110 named Amazonbot and 101 named PerplexityBot. ClaudeBot is the second most named AI crawler in this corpus and sits where the most recent counts in this series put it, between 140 of 1,013 files naming CCBot and 108 of 1,016 naming Amazonbot.

75 of the 1,014 files disallowed ClaudeBot at the site root. 68 of those 75 name the token themselves and the other 7 are caught by a wildcard group that never mentions Anthropic: coralvilleanimalhospital.com, amsterdam.nl, wa.gov, helsinki.fi, scielo.org, gmarket.co.kr and instacart.com all write Disallow: / under the wildcard and shut out everything, which is the pattern behind 559 of 581 pages lost to a rule that never named the crawler. The refusal is also unusually blunt. Every one of the 75 root blocks was decided by an exact Disallow: / and not one by a narrower pattern, so no site in this corpus closed the root to ClaudeBot and then reopened a section of it. That is a real difference from the citation crawlers: twelve of the forty-five files blocking PerplexityBot reopened specific paths, most of them advertising and shopping sections. Against a training crawler the decision in these files is all or nothing.

Where the 75 sit is more informative than the count. 37 of them are news sites, out of the 65 news hostnames whose file parsed, against 39 blocking GPTBot in the same stratum. Education contributes 9 of 100, ecommerce 6 of 80, healthcare 5 of 84 and government 4 of 78, while the 120 SaaS files produced 3 blocks and the 90 finance files 2. The concentration repeats what this blog found when it counted blocking by category and saw that 42 of the 77 files blocking a citation crawler were news sites. The other half of the distribution is the one worth noticing: 91 of the 159 files that name ClaudeBot leave the root open to it, 70 of those still carry at least one Disallow inside the ClaudeBot group, and 1,879 Disallow rules sit in those groups between them. Naming ClaudeBot is not the same act as blocking it, and in this corpus the more common reason to write the token is to hand it the ordinary housekeeping rules the site gives every crawler.

On 43 files the ClaudeBot line changes nothing the wildcard had not already decided

A robots.txt group is one or more user-agent lines followed by one or more rules, and consecutive user-agent lines extend the same group rather than starting new ones. Google's robots.txt documentation, last updated 2026-08-31, puts it plainly: you can group together rules that apply to multiple user agents by repeating user-agent lines for each crawler. RFC 9309 defines a group the same way and adds the fallback, that a crawler with no matching group must obey the group headed by the wildcard. Put those two sentences together and a third follows that nobody writes down: a file that lists a crawler by name on a line immediately above User-agent: * has not made a decision about that crawler. It has added the crawler to the wildcard's own group.

43 of the 159 files naming ClaudeBot do exactly that. The ClaudeBot line sits in a group whose tokens also include the wildcard, so the rules it selects are the rules every unnamed crawler already received, and all 43 leave the root open to ClaudeBot. 39 of the 43 are the same file: their first line is the comment # Squarespace Robots Txt, and the group runs 29 user-agent lines naming AI2Bot, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, DuckAssistBot, Google-Extended, GPTBot, Meta-ExternalAgent, omgili, Quora-Bot, TikTokSpider, YouBot and the three Google advertising tokens, then one more user-agent line carrying the wildcard, and only then the rules. Those rules are ordinary Squarespace housekeeping: Disallow /config, /search, /account, /api/ and a long run of query-string patterns. The group token counts across the 43 run from 10 to 51 with a median of 30.

The effect is a file that reads like a policy and functions like a default. A reader scanning it sees twenty-nine crawlers named above the wildcard and concludes the site has taken a position; the group selection rule says those twenty-nine names bought nothing the wildcard line underneath them did not already provide. Whether that was anyone's intention is not something a file can tell you, and this post does not claim it was a mistake. What it can tell you is the outcome, and the outcome is that a third of the ClaudeBot mentions in this corpus do no work. The same arithmetic applies to the other tokens in those files: 43 of the GPTBot mentions, 43 of the Google-Extended mentions, 42 of the CCBot mentions and 41 of the anthropic-ai mentions are in the same position. It is the inverse of the finding that GPTBot is named in 4.5 percent of robots.txt files and the wildcard in 77: here the name is present and the wildcard still decides.

One file in the corpus shows the opposite subtlety. lapresse.ca heads a group with the bare token Claude, which is a prefix of all three Anthropic bots, so under the group selection rule it binds Claude-User and Claude-SearchBot while the file's separate ClaudeBot group, an exact match, wins for ClaudeBot. Four files name claude-code, which is a developer tool rather than a web crawler and matches no agent Anthropic documents. Case is not the hazard in any of this: RFC 9309 requires case-insensitive matching, confirmed against this corpus from the other direction when 84 mis-cased crawler tokens all bound anyway. Position is the hazard, and it is invisible to every validator that checks syntax.

113 of the 159 said nothing about Claude-User or Claude-SearchBot

113 of the 159 files naming ClaudeBot named neither Claude-User nor Claude-SearchBot. That is the number that decides what the corpus knows about Anthropic's table, because the two unnamed bots are the two the vendor ties to visibility. 40 of those 113 go further and disallow ClaudeBot at the root while leaving both other agents to whatever the wildcard says. nature.com, jstor.org, healthline.com, jamanetwork.com, unesco.org, britishmuseum.org, broadinstitute.org and cshl.edu all sit in that group, and jstor.org writes the Anthropic support URL as a comment on the line above its ClaudeBot group, so the page describing the other two bots was open while the rule was written.

Across the whole set, 50 of the 75 files that disallow ClaudeBot at the root leave Claude-User allowed there and 53 leave Claude-SearchBot allowed. Only 8 of those 50 reach that result through a group that names Claude-User: 40 fall through to the wildcard and 2 match no group at all. On Anthropic's documentation that outcome is the better one, since it is the training opt out without the visibility cost, but on these files it is mostly not a configuration. It is the wildcard being permissive. The same gap on OpenAI's pair, counted against this corpus on 28 September 2026, ran 38 files refusing GPTBot while allowing ChatGPT-User, and Anthropic's is the widest of the three, which matters more here than elsewhere because Anthropic is the only one of the three vendors whose page says the second rule would have worked.

Of the 36 files that name all three Anthropic bots, 14 allow all three at the root, 13 refuse all three, 8 refuse ClaudeBot while allowing both others, and one, sciencedirect.com, refuses ClaudeBot and Claude-User while leaving Claude-SearchBot allowed. The 13 refusing everything are overwhelmingly news and reference: cnn.com, nbcnews.com, france24.com, dr.dk, nzherald.co.nz, news24.com, scientificamerican.com, theglobeandmail.com, rappler.com, eluniversal.com.mx, qcitymetro.com, congress.gov and amazon.com. The 8 drawing the documented line are bloomberg.com, forbes.com, motherjones.com, theregister.com, figma.com, ebay.com, bancolombia.com and ally.com. No file in the corpus did the reverse, allowing ClaudeBot at the root while refusing Claude-User, which is the one configuration that would be hard to explain.

Two of those 8 draw the line narrower than it looks. bloomberg.com writes a Claude-User group at line 276 containing Disallow: / at line 277 and Allow: /$ at line 283, and the end anchor means the allow covers the home page URL and nothing else: Claude-User is permitted at slash and refused at /about. figma.com writes Allow: /$ for a group heading both Claude-SearchBot and Claude-User with no Disallow under it, so there the anchor adds nothing and the rest of the site stays open. Six files wrote a Crawl-delay inside a ClaudeBot group, the non-standard extension Anthropic's page documents supporting with a ClaudeBot example of its own: github.com asks for 1 second, canonical.com for 2, decathlon.fr and thrivemarket.com for 5, rivm.nl for 60 and dtu.dk for 1,000. None of it changes what a crawler is permitted to read, and none of it can be verified from outside, which is the standing limit on every token in this file: a user agent is a claim rather than an identity, and the published address lists that would check such a claim are thin enough that none of the 2,344 addresses OpenAI publishes answered a reverse DNS lookup. The practical check is narrow enough to do by hand. Open the file, find every line naming Anthropic, confirm the value is exactly ClaudeBot, Claude-User or Claude-SearchBot, and confirm the line immediately below it is a rule rather than User-agent: *. Then decide the second and third rows of Anthropic's table on purpose, because leaving them to the wildcard is a decision too, just not one anybody made. The same question for the other engines is laid out on the AI visibility glossary entry and in the per-platform guides under how to get cited.