BlogFindings
Is my site blocking AI crawlers: 79 of 115 refused one their robots.txt allows
Lantad requested the home page of 1,015 hostnames five times each on 13 September 2026, once as LantadBot and once as each of GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. On 891 hostnames all five requests came back. 700 of those answered the baseline with a 2xx, and 115 of the 700 refused at least one crawler user agent. Re-reading each of those 115 robots.txt files the same day, 79 of them allow at the site root a crawler their server turned away.
On 13 September 2026 we asked all 1,027 hostnames in this repository's committed industry corpus frame for /robots.txt, identifying as LantadBot. Eleven of them disallow LantadBot at the site root and a twelfth answered that request with a 529, which this scanner reads as a temporary disallow, so we stopped at all twelve and did not fetch their home page. The remaining 1,015 each received five requests for the home page in sequence from the same client: one identifying honestly as our own crawler, then one for each of the four published AI crawler user agents this scanner sends. 891 hostnames answered all five. What came back was not one answer per site. It was frequently four different answers.
In short
- Is my site blocking AI crawlers is two questions, because there are two layers. Of 700 hostnames that answered a baseline bot with a 2xx on 13 September 2026, 115 refused at least one of four AI crawler user agents, and on 79 of those 115 the site's own robots.txt allows at the site root a crawler the server turned away.
- Across 2,800 host and crawler pairs, 312 were refused. Splitting those 312 against each site's robots.txt re-read the same day, 194 were refusals of a crawler the file permits, 96 matched a disallow already written in the file, and 22 sat behind a robots.txt that would not parse on the re-read.
- 83 of the 115 refused some crawlers and answered others in the same sequence of requests, and on 53 of those 83 the robots.txt permits all four at the site root. cdc.gov, canada.ca, bls.gov, nyc.gov and admin.ch each returned HTTP 200 to LantadBot, 403 to GPTBot and 200 to ClaudeBot, on the sample run and on a second check hours later.
- News sites refuse the most and software companies the least: 37 of 99 news hostnames refused at least one crawler user agent against 6 of 113 software hostnames, with government at 16 of 83, education 16 of 98, healthcare 15 of 93, travel 10 of 69, ecommerce 8 of 57 and finance 7 of 88.
- None of these refusals proves a deliberate block, and the reason is written into this scanner: a server that verifies crawlers by IP address will refuse an unverified user agent from any scanner, and every request here came from an address in no vendor's published range.
Is my site blocking AI crawlers somewhere other than robots.txt?
Two layers decide whether a crawler reads a page, and they are owned by different people. RFC 9309 defines the first: a crawler fetches /robots.txt, finds the group matching its product token, and applies the most specific rule. Nothing in that document compels obedience, and nothing in it stops a request. It is a published preference, and the crawler decides.
The second layer is the server, and it does stop requests. A content delivery network, a web application firewall or a bot management product sits in front of the origin and decides, per request, whether to pass it on. Cloudflare's own AI Crawl Control documentation, last updated 14 August 2026, describes setting allow or block rules for individual crawlers, which is enforcement rather than request. That is the layer this post measures, and it is the one a site owner is least likely to have looked at, because it is not a file in the repository and changing it does not show up in a diff.
The two layers can disagree in both directions, and both directions appeared in this sample. A site can publish a robots.txt that welcomes every AI crawler and sit behind an edge rule that refuses them. A site can disallow a crawler in the file and serve it anyway. Checking one and assuming the other is how a team ends up confident about access it does not have. Our own robots.txt tester reads the file and only the file, which is why it is one check in a report rather than the report.
Neither layer is a statement about whether an answer engine will cite you. That depends on what the page says once a crawler reaches it, which is a separate measurement with its own methodology. This post is only about the door.
Flow: Crawler requests a page (advisory) to robots.txt rule; robots.txt rule (if honoured) to CDN or bot management; CDN or bot management (if allowed) to Origin server; CDN or bot management (if blocked) to HTTP 4xx or 5xx; Origin server to Page is read.
What we sent, and what a refusal here does not prove
The frame is a fixed list of 1,027 hostnames committed to this repository, grouped into eight sectors of roughly equal size: government at 129 hosts, education at 130, healthcare at 123, news at 128, software at 130, ecommerce at 130, travel at 130 and finance at 127. It is weighted toward large organisations and is not a random sample of the web, so every proportion below describes this frame rather than the internet.
Each hostname was asked for /robots.txt first, as LantadBot, with redirects followed and a twenty second timeout, and the file was evaluated for our own token at the site root with the matcher this scanner runs in production. Twelve were left alone at that point, which is the policy we publish on our crawler page rather than one invented for this study. The other 1,015 received five sequential requests for the home page with identical headers, differing only in the user agent string: our own, then the published strings for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot as recorded in this repository's bot registry.
Now the part that decides how much any of this is worth. A user agent is a claim, and we have written before that a user agent is a claim rather than an identity. Every operator here publishes a list of IP addresses its crawler uses: OpenAI's crawler documentation points at openai.com/gptbot.json, Anthropic's crawler article, dated 7 April 2026, points at claude.com/crawling/bots.json, and Perplexity's bot guide points at its own file. We read all five files on the day and confirmed that the addresses our requests left from, sampled four times and falling inside one /24 each time, appear in none of them. So a site that verifies crawlers by IP address was correct to refuse every one of these requests, and this scanner carries that sentence as a fixed caveat in its code rather than as a footnote: servers that verify crawler IPs will block a spoofed user agent from any scanner, and the real crawler may pass.
One more limit, and it is specific. Anthropic publishes crawler tokens and no full user agent string, which the bot registry in this repository records in its own comment as the one part of the file taken on trust. The ClaudeBot string we sent is widely reported and not vendor published, so any comparison that ranks ClaudeBot against the others carries an asterisk the other three do not. Those lists also age at different rates, which we counted in an earlier post on how crawler IP files carry two dates that disagree.
GET /robots.txt then GET / five times, same client, 13 September 2026
- GET https://cdc.gov/robots.txt ua: LantadBot/1.0 200, no AI crawler token in the file
- evaluateRobots(file, LantadBot, /) allowed, wildcard group
- GET https://cdc.gov/ ua: LantadBot/1.0 200
- GET https://cdc.gov/ ua: GPTBot/1.4 403
- GET https://cdc.gov/ ua: OAI-SearchBot/1.4 403
- GET https://cdc.gov/ ua: ClaudeBot/1.0 200
- GET https://cdc.gov/ ua: PerplexityBot/1.0 403
How many sites refused a crawler user agent
891 of the 1,015 hostnames answered all five requests. The 124 that did not were re-tested once the same day to find out why, and the failures were mostly transport rather than policy: 36 HTTP/2 stream errors, 19 requests refused by the egress path this scan ran from before they reached the site, 17 connections reset, 14 timeouts at twenty seconds, 13 TLS failures including six certificate chains that would not validate, three single oddities such as an HTTP/0.9 reply, and 22 that simply answered on the re-test. None of the 124 is counted anywhere below.
Of the 891, the baseline request got a 2xx on 700. On those 700 the comparison is clean, because the same client reached the same page seconds earlier under our own name. 115 of the 700, which is 16.4 percent, then refused at least one of the four crawler user agents. 32 refused all four. 83 refused some and answered others.
Per crawler, counted over the same 700, GPTBot was refused 92 times, OAI-SearchBot 75, PerplexityBot 75 and ClaudeBot 70. That ordering matches the one we found in the files rather than at the server, where GPTBot was the most named token in robots.txt disallow rules, and it should be read with the string caveat above attached to the ClaudeBot column.
The other direction is worth its own line. On 191 of the 891 the baseline did not get a 2xx, and on 70 of those at least one crawler user agent did. Generic bot defence refusing an unknown crawler while passing a named one is a real configuration, and it means an unnamed tool checking access on your behalf can report a block that the crawlers themselves do not meet.
| Crawler | Refused | robots.txt allows it | robots.txt disallows it | File would not parse |
|---|---|---|---|---|
| GPTBot | 92 | 59 | 27 | 6 |
| OAI-SearchBot | 75 | 54 | 14 | 7 |
| PerplexityBot | 75 | 46 | 24 | 5 |
| ClaudeBot | 70 | 35 | 31 | 4 |
| All four | 312 | 194 | 96 | 22 |
Why one site answers ClaudeBot and refuses GPTBot
The 83 hostnames that refused some crawlers and answered others are the most informative group in the sample, because they rule something out. Every one of the five requests to a given host came from the same client, within seconds, carrying identical headers except the user agent, and from an address in no published crawler range. Verification by IP cannot explain a difference between them, because all four claims were equally unverifiable. What is left is a rule that reads the user agent string.
On 53 of those 83, the site's robots.txt permits all four crawlers at the site root. Five of them are worth naming because we checked them twice, hours apart, and got the same answers both times. cdc.gov, canada.ca, bls.gov, nyc.gov and admin.ch each returned HTTP 200 to our baseline, 403 to GPTBot and 200 to ClaudeBot. The cdc.gov robots.txt carries four user-agent lines, the wildcard plus Roverbot, EmailSiphon and MindSpider, names no AI crawler token at all, and its wildcard group permits the site root, so the file expresses no policy on any of this and the server expresses a detailed one.
The mirror image also occurs, and it is the reassuring case. theguardian.com disallows ClaudeBot and PerplexityBot in its file, and its server returned 403 to exactly those two and 200 to GPTBot and OAI-SearchBot. nature.com disallows GPTBot, ClaudeBot and PerplexityBot, and returned 406 Not Acceptable to exactly those three and 200 to OAI-SearchBot, which is a curious code for the job given we sent an identical Accept header on every request. cnn.com disallows all four and answered three of them with 451. Those are sites whose two layers agree, which is what you want to be able to demonstrate about your own.
It is also worth saying what this does not show. A named group in robots.txt replaces the wildcard group rather than adding to it, a mechanic we counted across this same frame when measuring how sites write rules for AI crawlers, and several of these sites are carrying edge rules that predate any AI crawler. An old rule that blocks unfamiliar user agents catches GPTBot without anyone having decided anything about GPTBot.
| Hostname | Baseline | GPTBot | OAI-SearchBot | ClaudeBot | PerplexityBot | robots.txt at / |
|---|---|---|---|---|---|---|
| cdc.gov | 200 | 403 | 403 | 200 | 403 | allows all four |
| canada.ca | 200 | 403 | 403 | 200 | 403 | allows all four |
| bls.gov | 200 | 403 | 403 | 200 | 403 | allows all four |
| admin.ch | 200 | 403 | 403 | 200 | 403 | allows all four |
| nyc.gov | 200 | 403 | 403 | 200 | 403 | allows all four |
| un.org | 200 | 403 | 200 | 200 | 200 | allows all four |
| cornell.edu | 200 | 403 | 403 | 403 | 200 | allows all four |
| theguardian.com | 200 | 200 | 200 | 403 | 403 | disallows those two |
| nature.com | 200 | 406 | 200 | 406 | 406 | disallows those three |
| cnn.com | 200 | 451 | 200 | 451 | 451 | disallows all four |
Which sectors refuse AI crawlers the most
The frame is stratified by sector, so the same comparison runs eight ways. News is the outlier and nothing else is close: 37 of 99 news hostnames that answered our baseline refused at least one crawler user agent, which is 37.4 percent. Software companies are at the other end with 6 of 113, or 5.3 percent, and they are the sector most likely to want to be quoted by an assistant and least likely to have a licensing negotiation running.
Between those two the spread is narrow. Government sat at 16 of 83, education at 16 of 98, healthcare at 15 of 93, travel at 10 of 69, ecommerce at 8 of 57 and finance at 7 of 88. The pattern is not obviously about technical sophistication, since government and finance sites carry heavy bot defence and still refuse crawlers at roughly half the news rate. A more likely reading is that news organisations are the ones who decided this on purpose, and everybody else inherited a setting.
Sector also changes which crawler gets turned away. In news, ClaudeBot was refused on 28 of 99 hostnames and GPTBot on 24, the only sector where our ClaudeBot string was refused more often than GPTBot. In ecommerce the gap runs the other way and is wider: GPTBot refused on 8 of 57, ClaudeBot on 2. Anyone using one crawler as a proxy for all of them is measuring the wrong thing, which is the same conclusion we reached measuring how few robots.txt files name OAI-SearchBot while naming its training twin.
None of these numbers says anything about AI visibility as an outcome. A site that refuses every crawler here may still be cited from a third party page, and a site that answers all four may be sending a crawler an empty document. Access is the precondition, not the result.
The status codes a crawler actually gets
Across the 4,455 requests that were answered, the codes were 3,271 of 200, 1,007 of 403, 44 of 402, 27 of 429, 21 of 404, 13 each of 202 and 406, 12 each of 401 and 451, 10 of 426, five each of 400, 405 and 498, three each of 416 and 503, and two each of 470 and 509. The long tail is the interesting part, because a crawler has to decide what each of them means.
403 Forbidden is the plain refusal and it dominates. The 402s are stranger. 402 Payment Required is described by MDN as a nonstandard code reserved for future use, with no standard use convention, and sixteen hostnames returned it to at least one crawler user agent while answering our baseline with a 200. Twelve of the sixteen are news publishers and two are healthcare publishers: acog.org, medscape.com, clarin.com, economist.com, forbes.com, independent.co.uk, hollywoodreporter.com, japantimes.co.jp, nzz.ch, slate.com, sfchronicle.com, theatlantic.com, usatoday.com, variety.com, totvs.com and flysafair.co.za. We did not establish which product emits them and do not attribute them to one.
Then the codes that are not refusals at all until you read the body. 582 hostnames returned 200 to all five requests, and on 171 the byte count still differed between the five, on 25 by a factor of 1.5 or more, and on 22 the page title differed. Six of the 582 returned a 200 carrying a challenge page rather than content, and on three of those, inria.fr, geisinger.org and practo.com, the baseline got the real page and the crawler user agents got the challenge. A checker that reads the status code and stops would record all six as healthy, which is the same trap we found counting X-Robots-Tag headers that turned out to be captchas.
Worth remembering alongside all of this: a code is what a server said to a request claiming to be a crawler, not what a crawler did. Several assistants never request robots.txt at all, which we measured separately when asking whether AI crawlers respect robots.txt.
-
200 with content3,271 responses Reached the page. On 171 of 582 all-200 hostnames the byte count still varied by user agent. -
200 with a challenge6 hostnames Status says yes and the body is a bot check. Three served it only to crawler user agents. -
403 Forbidden1,007 responses The plain refusal, and the code a verifying server correctly returns to an unverified claim. -
402 Payment Required16 hostnames Nonstandard and reserved per MDN. Twelve of the sixteen are news publishers. -
451 Unavailable12 responses Three hostnames, all news. cnn.com sent it to three crawlers and 200 to the fourth. -
429 Too Many Requests27 responses One request per user agent per host, so this is rate limiting by identity rather than by rate.
How to check whether your own site is blocking AI crawlers
The check is cheap and almost nobody runs it, because the tooling everyone reaches for reads the file rather than the door. Four steps, in order.
Read your robots.txt and evaluate it per token rather than by eye, since a named group replaces the wildcard group entirely and a rule you think applies may not. Then request your own home page with each crawler's published user agent and compare the status code against a request with no special user agent at all. Then read the body, not just the code, because a 200 carrying a challenge page is the failure most likely to be mistaken for success. Then look at where the difference is configured: if the file allows a crawler and the edge refuses it, the fix is in the bot management product and not in the repository.
That last step is where this usually lands. Nothing in a codebase records that someone turned on a managed rule three years ago, and the people who would notice the consequence are not the people with access to the dashboard. If your own site appears in a report like this one, the useful next question is not who blocked it but which product is holding the rule.
Our scanner does the first three steps on a single URL and reports what each crawler received, including the caveat that a verifying server refuses any scanner, and what GPTBot sees is the narrow version of the same check. The platform pages for getting cited by ChatGPT, by Claude and by Perplexity cover what happens after a crawler is through the door, and the crawlability study is where the standing measurements live. What is measured here is narrow on purpose: one page per site, one day, one network location, five user agents, and no observation of any real crawler at all.
- Evaluate robots.txt per crawler token Not by eye. A named group replaces the wildcard group rather than adding to it.
- Request the page as each published user agent Five requests per host here, identical but for the user agent string.
- Compare against a baseline request Without it a 403 is just a 403, and 191 of 891 hostnames refused our baseline too.
- Read the body, not only the status Six hostnames returned 200 carrying a challenge page rather than content.
- Confirm the crawler's identity by IP Not done here and not possible from outside: our addresses are in no published range.
Lantad
Published .
Ask whether a crawler can reach your pages and almost every answer you will find is about robots.txt. It is the wrong place to stop looking. A robots.txt rule is a request a crawler chooses to honour, and it is written by whoever edits the site. A 403 from the edge is a refusal nobody can argue with, and it is usually configured by somebody else, in a bot management product, months ago, under a rule nobody has opened since. So the question is my site blocking AI crawlers has two answers, and the file most people check holds only one of them.
Common questions
Is my site blocking AI crawlers if robots.txt allows them?
It can be. Of 700 hostnames that answered a baseline bot with a 2xx on 13 September 2026, 79 refused at least one AI crawler user agent that their own robots.txt permits at the site root. The refusal comes from the server layer, usually a CDN or bot management rule, which robots.txt has no visibility into and no control over.
Does a 403 to GPTBot mean the site blocked GPTBot on purpose?
No, and this is the caveat that matters most. A server that verifies crawlers against the operator's published IP ranges will refuse any request that claims a crawler user agent from an address outside those ranges, which includes every request in this study. A 403 here is evidence of user agent level enforcement, not evidence of intent, and the real crawler may pass where our probe did not.
Why did ClaudeBot get through on sites that refused GPTBot?
We observed it on 83 hostnames but cannot explain it, and one limit is ours: Anthropic publishes crawler tokens without a full user agent string, so the ClaudeBot string sent here is widely reported rather than vendor published. Any per crawler ranking carries that asterisk, and the honest finding is the asymmetry itself rather than a league table of which crawler is blocked most.
Which sector blocks AI crawlers most?
News, by a distance. 37 of 99 news hostnames refused at least one crawler user agent, against 6 of 113 software company hostnames, with government, education, healthcare, travel, ecommerce and finance falling between 8.0 and 19.3 percent. News publishers appear to have decided this deliberately while other sectors inherited a bot defence setting.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
A user agent is a claim, not an identity
Every per-crawler number you have ever seen, including ours, counts requests that said they were a crawler. Verifying the claim needs an IP check, and most tools do not do one.
Allow AI crawlers in robots.txt: 89 of 145 files ruled only on the whole site
Lantad requested /robots.txt once from each of 1,027 hostnames on 13 September 2026. 706 responses parsed into at least one group, and 145 of those name at least one of the 15 AI crawler tokens this scanner evaluates. On 89 of the 145, every rule in those groups governs the whole site. Fifty six write at least one path rule, and on 29 of the 145 deleting the AI rules entirely would have changed no verdict we tested.
Most blocked AI crawler: GPTBot in 82 of 718 robots.txt files, and its search twin in 24
Lantad requested /robots.txt once from each of 1,027 hostnames across eight industry sectors on 12 September 2026, and 718 of the responses parsed as robots.txt. GPTBot was disallowed at the site root by 82 of them, seven more than the fourth placed token. The wider gap is inside each vendor: 59 of the 718 refused GPTBot while leaving OAI-SearchBot allowed.
