GPTBot vs ClaudeBot: 22 of 1,026 robots.txt files gave the two crawlers different answers

GPTBot vs ClaudeBot: do site owners treat them as one decision?

They do. 187 of the 1,026 files named GPTBot in a user-agent line and 178 named ClaudeBot, and 168 named both. That overlap is the finding: of the 197 files naming at least one of the two, 168 named both, and only 29 named one without the other. 829 files named neither token, which is the ordinary case and not a permissive one, because silence hands the crawler whatever the wildcard group says.

The stronger test is what the files naming both actually did with them. 167 of those 168 gave GPTBot and ClaudeBot the same root verdict. A site owner who goes to the trouble of writing out both tokens writes the same rule under each of them, almost without exception. Across the whole corpus GPTBot was refused the root by 97 files and ClaudeBot by 91, and those two populations overlap in 83 files, so the difference between the headline refusal counts for the two companies is six files out of a thousand.

Naming a token is not the same as refusing it, and sometimes it is the opposite. 5 of the 19 files naming GPTBot without naming ClaudeBot allowed both crawlers the root, and 3 of the 10 naming ClaudeBot without naming GPTBot did the same. cloudflare.com, redis.io and mejuri.com each wrote an explicit Allow: / under their GPTBot group. openstax.org named GPTBot and gave it one rule, Disallow: /books/, which matches a deeper path and leaves the root permitted, otto.de names GPTBot in a group it shares with ChatGPT-User, OAI-Searchbot, PerplexityBot and Perplexity-User, whose five Disallow rules all name deeper paths, so the root is allowed there too.

The files that name these tokens are not typical files. The 168 naming both carried a median of 8 user-agent groups, against a median of 1 across all 1,026 and 1 among the 829 naming neither. These are long, maintained blocklists rather than a default file with a line added, which is consistent with what this blog found when it counted 2,209 crawler tokens across 1,004 files. The population that distinguishes between AI vendors at all is small and it is concentrated in sites that take the file seriously.

News sites are where that concentration is sharpest. 39 of the 68 news files named both tokens and 38 of the 68 refused both, which matches the pattern this blog reported when it measured 41 of 69 news files disallowing GPTBot in October. The decision in that stratum is made deliberately and it is made about AI as a category, not about OpenAI or Anthropic as companies.

What OpenAI and Anthropic each document their training crawler for

The two tokens are not mirror images, and a file that treats them as interchangeable is making an assumption the documentation does not support. Both vendors run more than one crawler and split them by purpose, which this blog has covered for OAI-SearchBot and for Claude-SearchBot separately. GPTBot and ClaudeBot are each the training member of their family, which is why they are the pair worth setting against each other.

OpenAI's crawler documentation, read at source on 6 November 2026, lists four tokens: GPTBot for content that may be used to train generative AI foundation models, OAI-SearchBot for surfacing sites in ChatGPT's search features, OAI-AdsBot for checking landing pages submitted as ads, and ChatGPT-User for user-initiated visits. On GPTBot the page states that disallowing it indicates a site's content should not be used in training generative AI foundation models. On ChatGPT-User it states that because these actions are initiated by a user, robots.txt rules may not apply. The page also says each setting is independent, and that where a site allows both OAI-SearchBot and GPTBot, OpenAI may use one crawl's results for both purposes.

Anthropic's crawler support article, read the same day, names three tokens: ClaudeBot for content that could contribute to model training, Claude-User for fetches a Claude user asked for, and Claude-SearchBot for search result quality. It states that Anthropic's bots respect do not crawl signals by honouring industry standard directives in robots.txt, that Anthropic supports the non-standard Crawl-delay extension, and that if a robots.txt file names ClaudeBot, Anthropic follows that rule even where a general setting says something different. It points to a source address list for its crawlers, published as plain text at https://claude.com/crawling/bots.json, on a host this site does not link.

Read side by side, the asymmetry is in the verb. Anthropic's page makes a compliance statement about its own crawlers. OpenAI's page describes what a Disallow line indicates. Lantad has not tested either claim here and this post makes no finding about it, though the blog has measured the gap between a rule and a fetch before, reporting that 234 of 592 files banning GPTBot were served a 200 anyway.

Why naming one crawler changes what happens to the other

The 22 files that split the two crawlers are mostly not expressing a preference between OpenAI and Anthropic. 21 of the 22 named exactly one of the two tokens, and the token they did not name fell through to the wildcard group. Whether that is a block or a pass depends entirely on what the wildcard group happens to say, and the wildcard group was written for search engines and scrapers years before either of these crawlers existed.

The mechanism is RFC 9309, the Robots Exclusion Protocol, published in September 2022. Section 2.2.1 requires case-insensitive matching of the product token, requires that more than one matching group be combined into one, and states that where no matching group exists a crawler must obey the group whose user-agent line is the asterisk, if one is present. If no group matches and there is no asterisk group, no rules apply. Section 2.2.2 then decides within the selected group: the most specific match by octet count wins, an allow beats an equivalent disallow, and no match at all means the path is allowed.

The consequence that catches people is that a named group replaces the wildcard rather than adding to it. Write a group for GPTBot and the wildcard group stops applying to GPTBot entirely, including any Disallow lines in it that were doing useful work. This blog has measured the same mechanism from the other direction, finding that 559 of the 581 pages GPTBot lost were closed by a rule that never named it and that a trailing star changed nothing in 168 comparisons.

In this run the asymmetry shows up in both directions. 6 files refused GPTBot through a wildcard group that never named it, and 7 refused ClaudeBot the same way. On 5 files no group applied to GPTBot at all and on 6 none applied to ClaudeBot, which under section 2.2.1 means no rules apply and the root is allowed. The robots.txt tester on this site resolves a path per token for exactly this reason: reading the file top to bottom tells you what it says, and only resolving a token tells you what it does. You can also see which named tokens a file carries with the AI crawler check.

The 22 files that split the two crawlers, read one by one

The 14 files refusing GPTBot alone are led by publishers and research bodies: 404media.co, cbsnews.com, irishtimes.com, rte.ie, nejm.org, science.org, dukehealth.org, uct.ac.za, alibaba.com, mentraglass.com, mxb.dev, ryanmulligan.dev, sia.codes and botcity.dev. 13 of the 14 name GPTBot, disallow the root under it, and never mention ClaudeBot. 404media.co is the plainest case: its wildcard group disallows five specific paths and nothing else, then a GPTBot group disallows the root. ClaudeBot inherits that wildcard group and is allowed everything outside those five paths.

The 8 refusing ClaudeBot alone are washingtonpost.com, sfchronicle.com, aftenposten.no, drugs.com, broadinstitute.org, fornidental.com, kidzandteendental.com and gmarket.co.kr. washingtonpost.com writes a ClaudeBot group disallowing the root with Allow lines for two marketing directories, and never names GPTBot, so GPTBot takes the wildcard group and its path-level rules. Seven of the eight follow that shape.

gmarket.co.kr is the exception and it inverts the whole pattern. Its wildcard group disallows the root outright, and it then writes a single group whose user-agent lines list eight tokens, GoogleBot, BingBot, Yeti, Daumoa, Twitterbot, GPTBot, AdsBot-Google and Ads-Naver, carrying Crawl-delay: 1 and two Allow rules, Allow: /n and Allow: /e. Under section 2.2.2 neither Allow matches the root, so no match means allowed, and GPTBot is admitted to a site that blocks everything by default. ClaudeBot is not on the list, takes the wildcard group, and is refused. Naming GPTBot there was an act of permission, not exclusion.

botcity.dev is the only file in the corpus that split the two crawlers while naming both, and it did so on purpose. Its second line is a comment reading that the strategy is to allow search, both AI and traditional, and block training-only crawlers. It then disallows the root for the wildcard group and writes Allow: / for Googlebot, PerplexityBot, ClaudeBot, Claude-User and GoogleOther, and Disallow: / for GPTBot and CCBot. ClaudeBot is allowed there because Anthropic's training token sits in a file whose author treated Claude as a search product. One file in 1,026 made the distinction this post set out to look for, and the rest of the corpus answers the question by not asking it. If you want the per-crawler picture for your own site rather than this corpus, that is what a Lantad scan reports, and the methodology page states how each verdict is reached.

What this measurement does not show

It reads files. No request from GPTBot or ClaudeBot was observed, so nothing here says whether either crawler honours the rule it was given, and the compliance sentences quoted above are each vendor's own claim rather than a Lantad finding. The blog has reported the same caution when it measured whether a robots.txt block stopped 12 of 18 AI chatbots.

Every token was resolved at the site root only. A file that permits the root and refuses a deeper path was counted as permitting, so the figures here are a floor on refusal and not a rate. 393 of the 1,419 hostnames contributed nothing: 32 requests failed at the transport layer, 221 answered with a 4xx status, which RFC 9309 treats as imposing no restrictions, 61 answered with a 5xx, which crawlers commonly read as a site-wide disallow, and 79 answered 200 with a file holding no user-agent group at all. None of those populations allows or refuses either crawler.

The corpus is a stratified sampling frame for crawlability rather than a census of the web, so a stratum count such as 7 of 68 news files indicates direction and not a national rate. Single-day figures also move: this blog published naming counts of 186 for GPTBot and 177 for ClaudeBot from the PetalBot run on 5 November, against 187 and 178 the next day on the same 1,026 denominator, which is the ordinary churn of files being edited.

Finally, this is a measurement about reading, not about answers. Whether a crawler can fetch a page is the first gate and not the last, which is why the blog measured 980 of 1,419 sites clearing every gate for ChatGPT and keeps the two questions apart. If what you need is a continuous report on what AI assistants say about your brand across engines, rather than whether their crawlers can read your pages, Lantad is the wrong layer and a monitoring platform is the right one: this site's own comparison with Profound states that Profound is well established at exactly that job and that Lantad works one layer upstream of it. Readers who want the downstream reading can start from the how to get cited guides and the AI visibility definition.