Most blocked AI crawler: GPTBot in 82 of 718 robots.txt files, and its search twin in 24

Which is the most blocked AI crawler?

The frame is the industry corpus committed at worker/seeds/corpus-seeds-industry.json: 1,027 hostnames sorted into eight sectors, being government, education, healthcare, news, SaaS, ecommerce, travel and finance. It is a fixed list somebody chose, not a random sample of anything, and this is the first post here that uses it. Every earlier robots.txt count on this blog ran on the smaller platform frame, which is why a figure such as 1 of 323 files naming a Mistral token is not directly comparable with anything below.

745 of the 1,027 answered with a 2xx. Twenty seven of those returned an HTML document rather than text, which is a content management system rendering a page for a path it does not have, so they carry no rules and are set aside. That leaves 718 files that parse, and every figure in this post is out of 718 unless it says otherwise.

Each of the 718 was evaluated with the scanner's own parser against the site root, once per token, for the 15 AI crawler tokens the product tracks. That list is a configuration choice rather than a measurement: it is what core/src/bots.ts holds today, and a crawler absent from it is absent from these counts. Matching follows RFC 9309, the Robots Exclusion Protocol of September 2022, which requires the most specific match to win, measured in octets, with an allow rule winning a tie. A site that names no token at all is counted as allowing every one of the 15, because that is what a wildcard group permitting the root actually does.

Two numbers describe each token: how many files name it in a user-agent line, and how many files leave it refused at the root once the rules are resolved. They are not the same number and the difference is the subject of a later section. GPTBot leads both, at 113 files naming it and 82 leaving it blocked. CCBot, Common Crawl's archive crawler and the oldest token in the set, is second at 80 despite feeding no consumer product directly. Bytespider is third at 76 and ClaudeBot fourth at 75.

Seven files separate first from fourth. On a population of 718 that is not a ranking anybody should act on, and it is worth saying so on a page that a search for the most blocked AI crawler will reach. What the table supports is a coarser and more durable statement: a little over one file in nine refuses the largest training crawlers, and roughly one in thirty refuses the search crawlers that decide whether a site can be cited in ChatGPT.

The training crawler is refused and its search twin is let through

Sort the table by operator instead of by rank and the shape changes. OpenAI runs a training crawler and a search crawler, and OpenAI's crawler documentation is explicit about which is which: GPTBot "is used to make our generative AI foundation models more useful and safe", while OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features". On this corpus the first is blocked at 82 sites and the second at 24. Anthropic's split reads the same way. Its help centre article, last updated on 7 April 2026, says ClaudeBot "helps enhance the utility and safety of our generative AI models by collecting web content", while Claude-SearchBot "navigates the web to improve search result quality for users". ClaudeBot is blocked at 75 sites, Claude-SearchBot at 20.

The per site counts make it clearer than the totals do. 59 of the 718 files block GPTBot while leaving OAI-SearchBot allowed. 55 block ClaudeBot while leaving Claude-SearchBot allowed. Only 23 block both OpenAI tokens. Whatever the popular framing, the median decision recorded in these files is not a refusal of an AI company. It is a refusal of one use, made by people who read the documentation and found the seam.

One file states the reasoning in its own comments. ally.com sorts its groups into three labelled blocks. The training group carries the comment "Training-only crawlers - disallowed. Each operator below serves citations through a different token that is allowed above, so this costs no presence in AI answers", and then pairs GPTBot with OAI-SearchBot, ClaudeBot with Claude-SearchBot and Applebot-Extended with Applebot inside the comment itself, before nine user-agent lines and one Disallow. A second group is headed "Bundled tokens - allowed" and holds Google-Extended, meta-externalagent and Amazonbot, on the stated grounds that each is one directive governing training and citation together, so refusing it would remove the site from those answers while withholding nothing already public.

That second judgement is supportable on the documentation. Google's common crawlers page, last updated on 14 July 2026, describes Google-Extended as a control that sends no request of its own: "Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." It governs training and grounding in Gemini, and the page states that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". A token that covers training and grounding in one directive gives a site owner no seam to split, which is exactly why 66 of the 718 refuse it and 95 name it at all: it is a live decision rather than a default. The mechanics of that one token were the subject of an earlier post on the two Gemini systems it covers.

Which industries block AI crawlers, and which never started

The eight sectors do not behave alike and the spread is wide enough that a single corpus wide percentage would mislead in both directions. News is the outlier in every column. 46 of its 65 parseable files name at least one of the 15 tokens and 44 refuse at least one, so on this frame blocking is the normal state of a news robots.txt rather than an exception in it. Ten of the 718 refuse all 15 tokens, and the six that do it by naming every one are all news sites: dr.dk, news24.com, lapresse.ca, nbcnews.com, nzherald.co.nz and rappler.com. The remaining four, being amsterdam.nl, wa.gov, helsinki.fi and scielo.org, refuse all 15 without naming any, because a wildcard group already disallows the root for every crawler.

Finance is the opposite end. 6 of 94 files name a token, 2 refuse one. Government is close behind at 6 of 85 naming and 7 refusing, the extra two being wildcard blocks rather than anything aimed at AI. Travel names 9 of 78 and refuses 4. On these three sectors the AI crawler question has, for the most part, not been answered in the file at all, which under RFC 9309 answers it by default in the permissive direction.

SaaS is the interesting middle case and it runs the other way from news. 22 of its 120 files name a token, and only 7 refuse one. Fifteen SaaS sites wrote AI crawler tokens into robots.txt in order to let them in. postman.com heads a group containing GPTBot, ClaudeBot and anthropic-ai with the comment "Google-Extended and Applebot-Extended gate AI training only, not search", followed by an instruction to move those lines to the blocked group at the bottom if the owner wants to opt out of training, which is a file written as documentation for its next editor. mediamarkt.de, an ecommerce site, writes ChatGPT-User, GPTBot and OAI-SearchBot each with an explicit Allow line. github.com, cloudflare.com, netlify.com, supabase.com and zapier.com all name GPTBot in a group that permits the root.

This is the sector pattern anybody selling AI visibility should keep in view, because the two halves of the market want opposite things. A publisher is trying to stop a crawler and measure whether the stop worked. A SaaS company is trying to be read and needs to know whether anything in its stack is quietly preventing that, which is usually rendering rather than robots.txt. The same scan answers both questions and the verdicts point in opposite directions.

A study on a different population reached a compatible conclusion from another angle: the sites that block AI crawlers are the ones with editors. What this measurement adds is that the effect survives outside the news and misinformation comparison, and that the sectors nobody asks about, being finance, travel and government, have mostly not written a rule at all.

Naming a crawler is not the same as blocking it

144 of the 718 files name at least one of the 15 tokens. 108 leave at least one refused at the root. Those two figures get treated as interchangeable in most write ups of robots.txt data, including some that count a token's appearance and report it as a block, and on this corpus they differ by 36 files in one direction and 4 in the other.

Of the 144 that name something, 104 refuse something and 40 refuse nothing at all. A file in that second group has written AI crawler tokens deliberately and then allowed them, or has scoped a Disallow to a path that is not the root. 36 files name GPTBot inside a group that permits the site root, and 42 do the same for OAI-SearchBot. Meanwhile 4 files refuse tokens they never name, because a wildcard group disallows the root and every crawler falls into it, which is a general closure rather than an AI policy.

The carve outs are where a count of names goes furthest wrong. nbcnews.com writes a GPTBot group whose first line is Disallow: / and whose next lines are Allow: /select and Allow: /select/, so the token is both named and blocked and the site still offers one section. Resolving that correctly is the whole job of a parser: under RFC 9309 the longest match wins, so a request for /select is permitted and a request for the home page is not. A rule written as a bare token count would record this site identically to one that refuses everything.

Two other distributions are worth recording because they cut against the idea of a settled convention. 574 of the 718 files name no AI crawler token whatsoever, so for most sites in this frame the question has never been put in writing. At the other end, 9 files name all 15, being sciencedirect.com, dr.dk, news24.com, nbcnews.com, nzherald.co.nz, rappler.com, postman.com, mediamarkt.de and ally.com, and those nine do not agree with each other about what to do: six refuse the lot, and three name the lot in order to sort it. Anybody checking their own file can resolve it the same way this scan did with the robots.txt tester, which evaluates each token separately rather than searching for its name.

What a 403 on robots.txt hides

282 of the 1,027 hostnames did not give us a file to read, and the reasons are not evenly spread. 165 answered HTTP 403 to LantadBot, and 57 of those 165 are news sites, which is 44 percent of the news frame against 4 of 130 in SaaS and 5 of 130 in education. A further 52 answered 503, 24 answered 404, 19 failed to resolve in DNS, 9 timed out and 4 returned 429.

Two of those outcomes have defined meanings and they point in opposite directions. RFC 9309 states that if a status code "indicates that the robots.txt file is unavailable to the crawler, then the crawler MAY access any resources on the server", which covers the 403 and the 404 alike. By the letter of the specification, 189 of these hostnames told every crawler that nothing is restricted. That is plainly not what the 403 sites intend, and the contradiction is worth sitting with: an edge that refuses to serve robots.txt is refusing to serve the one file whose whole purpose is to carry the refusal. The same specification says a file that is unreachable "is undefined and the crawler MUST assume complete disallow", so the 52 sites returning 503 were, at the moment we asked, disallowing everything by accident.

This is also the limit on the sector finding above and the reason to state it rather than bury it. The news sector lost 57 of its 128 hostnames to a 403 before any rule could be read, and a site whose edge blocks an unknown scanner outright is more likely than average to be one that also refuses AI crawlers in the file we could not see. The 44 of 65 figure is therefore a floor on news blocking, not an estimate of it. Every other sector's figure moves the same way for the same reason, just less.

A 403 to an identified scanner is not a robots.txt finding at all, it is a bot management finding, and it is the most common way an external measurement of this kind goes quietly wrong. We have hit the same wall from the other direction before, when 5 of 391 home pages sent an X-Robots-Tag noindex and all five turned out to be captcha pages, and again when 122 of 391 robots.txt requests answered with a cross host redirect. The response in front of the file keeps turning out to be as consequential as the file.

What to check on your own site after reading a ranking like this

The first thing this data argues against is copying anybody's file. The nine sites that named all 15 tokens disagreed with each other about what to do with them, the six that refused everything were all in one industry, and 574 sites had written nothing at all. There is no consensus to copy, and a robots.txt pasted from a vendor blog post inherits a decision somebody else made about their business model.

The second is that the question worth answering is not which crawler is most blocked but which of them you meant to block. The split that 59 sites found between GPTBot and OAI-SearchBot is available to anybody who reads two documentation pages, and it is the difference between declining to be training data and declining to appear in an answer. Perplexity publishes no separate training token at all, so getting cited by Perplexity is governed by the same two tokens that govern everything else it does, and Anthropic's fleet needs the same care, which its three named bots and one shared IP list make harder than it looks.

The third is that the file is only the first of several gates, and on most sites it is not the one that fails. A permissive robots.txt in front of a page that ships no text to a non-rendering fetcher grants access to nothing worth reading, which is what prose parity measures and why what GPTBot sees is a separate check from what it is allowed to fetch. Structured data sits behind the same gate: markup that never reaches a parser is structured data in name only.

Fourth, whatever you decide, check that the file is being served. Fifty two sites in this frame were returning 503 to a plain GET on the day we asked, which under the specification is the strictest possible instruction, and none of them chose it. A rule you cannot fetch is a rule nobody is following. The method behind each of these checks is written up on the methodology page, and the wider set of measurements this blog has published sits on the research index.

Last, remember what a count of files is not. Nothing here observed a crawler. A robots.txt line is a request, and the evidence that requests are sometimes ignored is not thin: a controlled study found that 6 of 10 assistants never asked for the file before fetching. Writing the rule is the part a site owner controls. Whether it is honoured is a separate measurement, and it needs a server log rather than a scan.

What this measurement does not say

This counts text in files. It does not observe behaviour, and the distance between those two things is the whole reason to write this section rather than leave it implied.

No crawler was watched. No access log was read. Nothing here reports whether GPTBot, ClaudeBot, PerplexityBot or any other agent obeyed, ignored or ever requested any of these files, and no figure in this post should be quoted as evidence that a block worked. The related question of what a block costs has been measured by others rather than by us: a difference in differences study of news publishers put the figure at roughly 7 percent of traffic within six weeks, which is a finding about consequences, not about compliance.

Each hostname was asked once, on one day, from one network location, with no retry. A site that was briefly unwell reads here as permanently unreachable, and a site that varies its response by geography or by request volume was seen from one vantage point. Robots.txt is also frequently served differently to different user agents, and we asked as one identified scanner.

The 15 tokens are a configuration in this repository rather than a census of the field. Tokens outside it, including every crawler token this product does not yet track, are invisible to these counts even when the file names them, which is how a file can name a token that matches no crawler anybody runs. The corpus is full of them: 53 of the 718 files name cohere-ai, 47 name Diffbot, 45 name omgili and 38 name YouBot, and not one of those appearances is in any figure above.

Finally, the frame is 1,027 hostnames that somebody put in a file, stratified by industry and weighted towards large and well known organisations in each. It is not the web, it is not a probability sample, and the eight sector sizes are close enough to each other that they say nothing about how many sites of each kind exist. It supports statements about these 1,027 hostnames on this day, and those statements are the only ones made here.