Which industries block AI crawlers? 41 of 69 news files disallowed GPTBot, and 2 of 97 finance files did

Which industries block AI crawlers, sector by sector?

News is the answer, and it is not close. 41 of the 69 news robots.txt files that parsed disallowed GPTBot at the root, which is a clear majority of a sector, and the same files behaved the same way towards the other training crawlers: 40 disallowed ClaudeBot at the root and 40 disallowed CCBot. No other sector in the corpus reaches double figures as a share. Education comes second in absolute terms with 10 of 108, healthcare has 9 of 96, and both of those sit near one in ten rather than three in five.

At the other end are the two sectors that would be most people's guess for the defensive ones. Finance disallowed GPTBot at the root on 2 of 97 parsed files, and the two were ally.com and bancolombia.com. Travel disallowed it on 2 of 87, airbnb.com and tripadvisor.com. SaaS disallowed it on 3 of 120. These are not sectors that have weighed the question and come down on the open side, because mostly they have not written anything down: 6 of the 97 finance files named any of the seven tokens tested, and 6 of the 87 travel files did. The absence of a rule is the whole of the policy.

This points the same way as an earlier count on a different frame, where news sites supplied 42 of the 77 robots.txt files blocking a citation crawler, and it is worth saying plainly that the direction is unsurprising. News organisations are in litigation and licensing negotiations with model vendors, and a robots.txt line is the cheapest instrument they have. What the sector split adds is the size of the gap. A sector whose business is publishing text has decided the text is the asset; sectors whose websites are brochures for a product sold elsewhere have mostly not treated the question as one that concerns them.

Three cautions on reading the table. First, these are verdicts at the root path only, so a site that leaves the root open and disallows a hundred directories beneath it counts here as open, which is accurate about the root and says nothing about the rest. Second, the counts are of files that parsed, and the parse rate differs by sector: 69 of 128 news hostnames served a readable file against 120 of 130 in SaaS, so each sector's denominator is its own. Third, naming and blocking are different acts, and in two sectors they come apart in opposite directions, which the next sections are about. Anyone who wants the same verdict for their own file against a named token can get it from the robots.txt tester.

How 1,027 hostnames were read, and what the counts leave out

The frame is the industry half of this repository's corpus seed files, which is eight strata of roughly 130 hostnames each: government, education, healthcare, news, SaaS, ecommerce, travel and finance. The platform strata that make up the rest of the corpus are excluded here because they are sorted by the technology a site is built on rather than by what it does, so they cannot answer a question about industries. The frame and its construction are set out on the methodology page and in the crawlability study.

Of the 1,027 hostnames, 750 answered robots.txt in the 2xx range, 190 answered a 4xx, 56 answered a 5xx and 31 failed in the client before any status arrived. Those outcomes are not interchangeable and the scanner does not treat them as such. Under RFC 9309 an unavailable file imposes no restrictions, so a 4xx reads as open; a 5xx is treated here as a site-wide disallow until it recovers, which is deliberately the pessimistic reading rather than a guess. 39 of the 750 served an HTML document at the robots.txt address, which parses into no usable rules, and 45 parsed into no user-agent group at all. Both of those count as files that answered, which is why the parsed figure is a ceiling on how many sites have a working policy and not a count of them. An earlier sweep found a defect in 232 of 1,059 files, and nothing here contradicts that.

The second pass requested each permitted host's home page, which is where the 232 refusals in the hero table come from, and it ran only against hosts whose own file left the root open to this crawler. 930 hosts qualified, 697 answered 200 with an HTML content type, and the gap between those two numbers is the subject of the last section.

Four limits bound every figure above, and they all cut the same way. These are single reads from one address, so a site serving different files to different networks is recorded as whatever it served here. A verdict at the root is not a verdict about a site. The seven tokens tested are not the whole population: the register of names in the wild is far longer, and our own census found 2,209 tokens across 1,004 files, including names no vendor documents. And a robots.txt file read today is not necessarily the file a crawler is acting on today, because caching sits in between, which is the subject of when a robots.txt edit reaches a crawler. A second read of all 1,027 hostnames the same day parsed 753 files rather than 750 and returned the same verdicts everywhere except theregister.com, which parsed into no named group on that read while three later reads reproduced the first result. Six hostnames in total answered differently within the hour, among them un.org moving from 404 to 200 and mcgill.ca from 200 to 429. That is the measurement's own noise floor, and it is roughly half a percent of the frame.

Why 40 of 55 sites let the search crawler in without deciding to

The headline split in this corpus looks like sophistication. 55 of the 750 files disallowed GPTBot at the root while leaving OAI-SearchBot allowed there, and 24 of those 55 were news sites. Read as policy that is exactly the distinction the vendor documentation invites: OpenAI's crawler documentation describes GPTBot as the crawler whose content may be used to train its foundation models and OAI-SearchBot as the one used to show websites in ChatGPT search answers, and it states that a site opting out of search may still appear as a navigational link. Refuse the training crawler, keep the search crawler, keep the referral traffic. It is the obvious trade.

Only 15 of the 55 actually made it. The rule that allowed OAI-SearchBot at the root was an explicit Allow line on 13 files, and on 2 more it was an Allow pattern ending in a dollar sign, which under RFC 9309 anchors the match to the end of the path and therefore grants the home page and nothing below it. bloomberg.com is one of those two: its OAI-SearchBot group opens with Disallow slash and then carries 45 rules, among them an Allow of slash dollar, and because RFC 9309 requires that the most specific match win and defines the most specific match as the one with the most octets, the two-character pattern beats the one-character Disallow and the home page alone stays open.

On the other 40 files OAI-SearchBot was never named. It is allowed at the root because section 2.2.1 of the specification says that where no group matches a crawler's product token, the crawler must obey the group whose user-agent line carries the asterisk value, and on those sites the wildcard group leaves the root open. So the apparent training against search distinction is, four times in five, the arithmetic of having named one token and not the other. Those sites wrote GPTBot into a file because GPTBot is the name that appeared in the coverage they read, and OAI-SearchBot inherited the permissive default that every unnamed fetcher gets. Our earlier count found OAI-SearchBot named in 16 of 140 files against GPTBot in 25, and this is the same asymmetry showing up as an unintended permission.

The mechanism runs in both directions, which is the part worth checking on your own file. 73 of the 78 files that disallowed GPTBot at the root did it in a group that names GPTBot. The other 5 never mention it: amsterdam.nl, wa.gov, helsinki.fi and scielo.org disallow the root for every crawler through the wildcard group, so GPTBot is refused as a side effect of a site that is closed to everything, and instacart.com reaches the same result the same way. Those are not AI policies and should not be counted as such. The general case of a wildcard rule doing work nobody attributed to it is one this blog keeps meeting: the wildcard group closed 559 of the 581 pages GPTBot lost in an earlier measurement, by a rule that never named the crawler.

35 files named GPTBot in order to let it in

Naming a crawler is not the same as blocking it, and in this corpus a third of the files that name GPTBot do the opposite. 35 of the 750 named GPTBot and left the root open to it. 13 of those sit in the SaaS stratum, and the pattern there is a single explicit line: cloudflare.com, netlify.com and redis.io each head a group with GPTBot and give it Allow slash and nothing else. github.com is a variation worth separating, because GPTBot heads a group there alongside OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot that carries 165 rules, none of which matches the root, so the root stays open without any line saying so.

Finance supplies 4 of the 35 and they are the clearest statement of intent in that sector, since 6 finance files name a token at all. commbank.com.au names GPTBot with Allow slash. geico.com writes Allow slash separately for GPTBot, CCBot, ChatGPT-User, OAI-SearchBot, Claude-Web and ClaudeBot, and carries a Content-Signal line in its wildcard group reading search equals yes, ai-input equals yes, ai-train equals no, which asks for something the Disallow lines in the same file do not enforce. Whether an AI vendor reads that line is a separate question and nothing here measures it.

Two of the seven tokens need their documentation read before their counts mean anything. Google-Extended, named in 86 files and disallowed at the root in 60, is not a crawler: Google's common crawlers documentation, last updated 2026-10-15 UTC, describes it as a standalone product token with no separate HTTP user agent, used to control whether crawled content trains future Gemini generations and whether it is supplied to models at prompt time through grounding, and states that it does not affect a site's inclusion in Google Search and is not a ranking signal there. A rule naming it is a licensing declaration rather than an access rule. ClaudeBot, disallowed at the root in 75 files, is documented by Anthropic as the training crawler in a family of three, alongside Claude-User for user-directed fetches and Claude-SearchBot for search indexing, on a page carrying 7 April 2026 that says its bots honour industry standard robots.txt directives. A site blocking ClaudeBot and leaving Claude-SearchBot unnamed has made the same unintended trade the previous section describes.

None of this says what any crawler actually did. These are the site's instructions, and the gap between an instruction and a fetch is its own measurement: Googlebot and GPTBot got different answers on 81 of 1,008 files, and a rule is only as good as the crawler's compliance with it.

What refused this scanner was the network, not the token

232 of the 930 hosts whose own robots.txt permitted this crawler at the root then refused the home page request, 212 of them with a 403 and 7 with a 429. Taken alone that number invites the obvious conclusion, and the obvious conclusion is the one measurement this post exists to contradict. Each of the 232 was requested again the same day with a desktop Chrome user agent string, the same identity the scanner's renderer presents, from the same address. 196 of the 232 refused that too, 177 of them with a 403 both times. 34 served the browser a 2xx having refused LantadBot, and 2 failed in the client.

So the refusal is overwhelmingly not about the declared token. It is about the network the request came from, which here is a cloud address, and about whatever else the edge fingerprints before it reads a user agent. Only 34 of 930 permitted hosts, under four percent, demonstrably treated a declared crawler differently from a browser on this evidence. Those 34 skew towards ecommerce with 10 and government and travel with 6 each, and they include boots.com, chicago.gov and noaa.gov. That is the honest size of the user-agent-sensitive population measured here, and it is much smaller than the raw 232.

This bounds our own earlier work as much as anyone else's. When we published that 103 of 1,089 sites served an unknown bot and refused GPTBot and that 79 of 115 refused a crawler their robots.txt allows, the controlling variable in those counts deserves the same scrutiny applied here. It also bounds every third-party study that reports a percentage of sites blocking a named AI crawler on the basis of requests sent from a data centre. Without a browser control from the same address, such a figure cannot separate a site that refuses GPTBot from a site that refuses everything arriving from a cloud provider, and the second population is larger.

Two things follow for anyone reading a number like this, including the ones in this post. A real AI crawler is not in this position: OpenAI and Anthropic publish address ranges for their fetchers and operate from them with a reputation attached, and reputation is exactly what a cloud scanner lacks, which is why a user agent is a claim, not an identity and why verification runs the other way, through the published ranges. Our own attempt to confirm those ranges found 0 of 2,344 addresses OpenAI publishes answered a reverse DNS lookup. What an external scan can report honestly is what a declared, unverified crawler received from one address at one moment. It cannot report what GPTBot received, and this post does not.

What to check in your own robots.txt

The useful work here is small and it is not a rewrite. Four checks follow directly from the counts above, and each one takes a minute on a file you already have.

Start with the tokens you have named, because naming is what switches off the wildcard group for that crawler. If your file names GPTBot and nothing else from OpenAI, then OAI-SearchBot is being governed by your asterisk group, and 40 of the 55 files in this corpus that look like a deliberate training against search split got there by accident that way. Decide whether that is what you meant, then write it. The same applies to ClaudeBot against Claude-SearchBot, and the vendor documentation linked above is the list of names worth checking against rather than a blog roundup.

Then read your wildcard group as though you were an unnamed crawler, because that is what most AI fetchers are on most sites: 616 of the 750 files here named none of the seven tokens tested. Whatever that group says is your AI policy, whether or not it was written as one, which is how four sites in this corpus came to refuse a training crawler they never mention. Our earlier measurement of a trailing star changing no verdict in 168 comparisons is a reminder that pattern intuition is unreliable here, so check the verdict rather than reading the pattern, which is what the robots.txt names the wildcard, not the AI crawler measurement found people getting wrong.

Third, separate the root from the rest. Every verdict in this post is at the root path, and a file can leave the root open while closing the directories that hold your actual content. Check a real article or product URL, not just the slash.

Fourth, do not stop at the file. 232 hosts here permitted this crawler in robots.txt and then refused it at the edge, and even after the browser control cut that to 34 demonstrably token-sensitive cases, the file and the server disagreed on all 232. If what you want to know is what a crawler receives rather than what you told it, the file is the wrong place to look. What GPTBot sees reports the fetch, and the two answers are not the same measurement.