Claude-SearchBot: 25 of 1,024 robots.txt files refused it, and 55 of the 67 that kept it never named it

What is Claude-SearchBot, and how is it different from ClaudeBot?

Anthropic's help centre article carries the date 7 April 2026 and documents three tokens. ClaudeBot collects web content that may be used for training Anthropic's generative AI models. Claude-User accesses websites when a Claude user asks a question that requires it. Claude-SearchBot indexes content to improve the relevance and accuracy of search responses, and the page describes it as navigating the web to improve search result quality for users. On robots.txt the page says that Anthropic's bots respect do not crawl signals by honouring industry standard directives in robots.txt, and that to block a bot from an entire website you add the rule to the robots.txt file in your top level directory.

Three tokens on the page, four in the registry. The fourth is anthropic-ai, an older string that the current article does not name, and it is still the second most widely named Anthropic token in the corpus. Lantad has published that gap before, reporting that 1,004 files named 2,209 crawler tokens and 120 named one Anthropic no longer documents. It matters here because a file that names anthropic-ai and nothing else has written a rule for a crawler that is not the one deciding whether Claude's search answers can read the site.

The distinction that carries a consequence is the one between training and search. Refusing the training crawler withholds the content from model training and leaves the search index alone. Refusing the search crawler withholds the content from the index the answers are built from. Anthropic separates them into different tokens precisely so a site can make those two decisions separately, and the same separation exists at OpenAI, which is why this blog measured 980 of 1,419 sites clearing every gate OpenAI's documentation implies rather than counting one token. The question this post asks is how many sites use the separation, and how many land on one side of it by accident.

How many sites refuse Claude's search crawler?

Of the 1,419 hostnames asked on 3 November 2026, 46 failed at the network level, 266 answered with a status other than 200, and 1,107 answered HTTP 200. 1,024 of those parsed into at least one user-agent group, carrying 75,651 Allow and Disallow rules between them. That 1,024 is the denominator for everything below, and it is the same frame this blog used when it reported that 0 of 1,024 files carried the new shared AI training token the day before.

46 of the 1,024 files named Claude-SearchBot in a user-agent group. 18 of those 46 disallowed it at the site root. The other 28 named it and left the root open, which is a deliberate allow: a group exists for the crawler and it permits the path. Seven more files disallowed the crawler at the root without naming it anywhere, which brings the total refusing it to 25.

Set that against the training crawler. 177 files named ClaudeBot, and 84 of those 177 disallowed it at the root. Counting every file rather than only the ones that name it, 91 refuse ClaudeBot and 25 refuse Claude-SearchBot. Either denominator tells the same story: the training crawler is named by nearly four times as many files and refused by between three and five times as many, depending on which way you count. The gap is the finding, and it is the right shape, because a site that wants to be read by an assistant without being used to train one would produce exactly this asymmetry.

The direction of travel is also measurable, because this corpus was read the same way 16 days earlier. On 18 October 2026 Lantad published that 159 of 1,014 files named ClaudeBot and 75 disallowed it at the root. On 3 November the same method over the same seed files returned 177 naming it and 84 of those refusing it, counted the same way. Both the naming and the refusing rose, on a denominator that moved by ten files, and a reader should treat a two week change on a sample of this size as a direction rather than a rate.

The 67 files that refused training and kept search

Crossing the two verdicts is where the file stops being a list of rules and starts being a policy. 67 of the 1,024 files disallowed ClaudeBot at the root and allowed Claude-SearchBot. 24 refused both. One refused the search crawler and allowed the training crawler. The remaining 932 allowed both, which in the large majority of cases means they said nothing about Anthropic at all: 840 of the 1,024 files named no Anthropic token of any kind.

So among the files that take a position on Anthropic, the common position is the one a site owner would choose on purpose. Refuse the training crawler, keep the search crawler, stay eligible to be read and cited. That is a better result than this blog has usually found when it compares a training token against its search twin, and it is worth saying plainly, because the finding here is not that sites are getting this wrong.

The qualification is how they are getting it right. Of the 67, only 11 named Claude-SearchBot in a group and allowed it there. In 55 of the 67, no group named the search crawler, so the verdict was decided by the wildcard group, which permitted the root. One file matched no group at all. In other words the separation between training and search in those 55 files is not a decision about search. It is the absence of a decision, and it currently resolves the way the site owner would want.

That distinction has a cost the day the wildcard group changes. A site that writes Disallow: / under the wildcard for an unrelated reason, or adopts a platform default that does, moves 55 files worth of this pattern from allowed to refused without anyone editing a line about Anthropic. Lantad has measured that mechanism before from the other end, reporting that 559 of the 581 pages GPTBot lost were closed by a rule that never named it. The same arithmetic applies to a crawler you are currently allowing.

How a wildcard rule blocks a crawler you never named

The mechanism is in the standard rather than in any vendor's documentation. RFC 9309, the robots exclusion protocol, says in section 2.2.1 that crawlers must use case insensitive matching to find the group that matches the product token, that the rules of matching groups must be combined into one group, and that if no matching group exists, crawlers must obey the group with a user-agent line with the star value, if present. Section 2.2.2 adds that the most specific match must be used, and defines the most specific match as the one with the most octets.

Read that in order and the consequence for Claude-SearchBot is mechanical. A file that names it gets what it wrote. A file that does not name it gets the wildcard group, whatever that group was written for. A file with neither has no rules to apply and the crawler proceeds. Nothing in the chain treats an unnamed AI crawler as a special case, which is why this blog keeps returning to the point that a robots.txt file naming the wildcard rather than the AI crawler is still a decision about the AI crawler.

Seven files in this corpus refused Claude-SearchBot through that route on 3 November 2026. In every one of the seven the deciding rule was Disallow with the pattern slash, under the wildcard group. Six of the seven named no Anthropic token anywhere in the file, so nothing in them is addressed to Anthropic and the crawler is refused the whole site. Five of the seven refused GPTBot through that same wildcard rule. A sixth refused it through a group written for GPTBot itself, and the seventh allowed GPTBot in a group of its own while leaving Anthropic's search crawler to the wildcard, which is the shape a file takes when one vendor gets attention and another does not.

The single reverse case is the clearest illustration in the set. One file closes the site under its wildcard group with Disallow and the pattern slash on line 5, then opens two groups further down, one naming ClaudeBot and one naming Claude-User, each carrying Allow and the pattern slash. Two of Anthropic's three documented crawlers are named and welcomed. The third is not named, so it takes the wildcard group and is refused the whole site. The file says yes to training, yes to user initiated fetches and, without mentioning it, no to search. You can check this on a single file with the robots.txt tester, which evaluates a path per crawler token rather than reporting the file's syntax.

What an outside citation dataset reports about blocking the search crawler

Lantad measures the site side of this. It does not measure whether a site gets cited, and this section reports somebody else's figures, in their voice rather than ours.

Muck Rack publishes a recurring report called What Is AI Reading, run with Generative Pulse. The October 2026 edition carries a last updated date of 28 October 2026 at muckrack.com/blog/what-is-ai-reading-october-2026 and describes its dataset as Generative Pulse data covering more than 40 million AI responses and 207 million citations in 2026, alongside a September analysis of 1,020 prompts across ChatGPT, Claude, Gemini and Perplexity. Under a heading stating that crawler access still affects whether publishers are cited, it reports a strong relationship between crawler access and AI visibility, that among major publishers blocking Claude's search crawler 81 percent received almost no Claude citations, and that just 12 percent of publishers allowing the crawler saw similarly low citation levels.

Those are the two figures, quoted as published. Now the part a measurement blog owes its reader. The report names no publishers in that analysis, gives no count for either group, and states no measurement period for it, while the dataset figures it cites alongside cover the whole of 2026. A relationship of that size reported without a denominator is a reason to go and look at your own file rather than a number to repeat. Note also that the report says Claude's search crawler and not Claude-SearchBot: the mapping from that phrase to the token Anthropic documents is ours, and it is the obvious reading rather than a stated one.

What the two sources do together is divide the work honestly. Theirs is evidence that the search crawler is the one with a citation consequence. Ours is a count of how many sites have actually written a rule about it, which is 46 of 1,024. The reason this blog measures citations cautiously is that they move: Lantad reported an external study finding that 50 to 83 percent of 37,500 citations were gone by day two. A robots.txt verdict does not move on that timescale, which is why it is the half worth measuring directly.

News sites are where search crawler blocking concentrates

The Muck Rack figures are about publishers, and publishers are the one stratum of this corpus where refusing the search crawler is common rather than rare. Of the 68 news files that parsed, 15 disallowed Claude-SearchBot at the root. Across the other 956 files, 10 did. News supplied 68 of the 1,024 parsed files and 15 of the 25 refusals.

The training picture in news is different again. 41 of the 68 news files disallowed ClaudeBot at the root, and 42 of the 68 named some Anthropic token, against 184 of 1,024 corpus wide. News sites are not ignoring these crawlers: they are the population that reads the documentation and writes rules, and 26 of the 68 wrote the training refusal while leaving search allowed. That is the deliberate version of the pattern in section three, and it is concentrated in exactly the sites with editors, which this blog has reported before when it found that 41 of 69 news files disallowed GPTBot against 2 of 97 finance files.

One honest limit on the news comparison, and it cuts against the figure. 128 news hostnames were asked and only 69 answered HTTP 200, with 68 parsing. Every other stratum returned a higher share. So the news row rests on roughly half its sampling frame, and the half that did not answer is not a random half: a site that refuses an unknown crawler the robots.txt file is the kind of site likely to have written a rule in it. The direction of that bias is unknown and the honest statement is that the news figure is computed on the files that answered, as every figure in this post is.

Outside news the pattern is thin to the point of absence. Zero of the 119 saas files, zero of the 91 finance files, zero of the 84 healthcare files and zero of the 77 travel files refused the search crawler, and not one of the ten platform strata returned more than a single refusal. For most of the corpus the Anthropic question is not answered wrongly. It is not answered at all, which is the same conclusion the crawlability research on this corpus reaches about most per crawler rules.

What this measurement does not show

This is a reading of 1,024 text files on one day, so the list of things it cannot tell you is longer than the list it can. It does not show whether Anthropic honours any of these rules. The help centre page says its bots honour industry standard directives in robots.txt, that is a vendor statement rather than a measurement, and verifying it needs server logs from the sites in question rather than a request to each of them. This blog has published the gap between a robots.txt rule and the behaviour that follows it, including that 79 of 115 sites refused a crawler their own robots.txt allows, and the inverse gap is just as real and is not measured here.

It does not show intent. A file that allows the search crawler through its wildcard group may have been written by somebody who considered the question and saw no need for a rule, or by somebody who has never heard of the token. The 11 files that named Claude-SearchBot and allowed it are the only ones in this corpus where a deliberate allow is on the record, and the measurement cannot promote the other 56 into that category.

It does not show citations. Nothing in a robots.txt file says whether Claude names a site in an answer, and no scan of a site can establish that. The relationship between the two is what the Muck Rack report addresses and what Lantad is reporting rather than measuring. The same caution applies in the other direction: this blog measured that 52 of 54 sites refusing Google-Extended still allow Googlebot, which says what the files do and not what either crawler then did.

And it does not show your site. The corpus is a stratified sampling frame for research, committed in this repository and described on the methodology page, and your file is not in it. The useful move from here is to evaluate your own root path against each Anthropic token separately rather than against a single notion of an AI crawler, check which group actually decides the verdict, and read how Claude's own retrieval works on the how to get cited by Claude page. If the deciding group for your search crawler verdict is the wildcard, you have a rule you did not write, and the only question is whether it currently says what you would have said. Lantad's own crawler identifies itself and publishes what it does at /bot, and the registry of tokens it evaluates is listed on the AI crawlers page.