CDN blocking AI crawlers: 103 of 1,089 sites served an unknown bot and refused GPTBot

What a refused request proves, and what it does not

This measurement sent a user agent string. It did not send GPTBot. That distinction governs every figure in this post and it belongs before the figures rather than in a footnote after them, because the honest reading of a 403 here is narrower than the headline invites.

Every request in this run left from 160.79.106.133. OpenAI publishes the addresses its crawlers use as a JSON file, and openai.com/gptbot.json held 18 prefixes when it was read on 26 September 2026, with a creationTime of 2026-09-22T02:00:07. The scanner's address is in none of them, which was checked rather than assumed. So a server that refuses this request is refusing an unverified claim to be GPTBot, and refusing an unverified claim is what bot verification is for.

Cloudflare states the test plainly. Its verified bots documentation, last updated 1 July 2026, requires "Honest self-identification" and defines it as declaring "who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS." A user agent string on its own satisfies none of those three. This is the same gap recorded when this blog argued that a user agent is a claim and not an identity, and it is why the count of published crawler IP ranges matters more than the count of tokens in a robots.txt file.

Two things survive that limit, and they are what the rest of this post rests on. A refusal is still evidence that the site treats the two strings differently, because the browser request and the bot request went to the same URL seconds apart from the same address. And a site owner reading this cannot verify their own AI crawler access from outside either: they have the same problem this scanner has, which is the practical finding underneath the numbers.

Is your CDN blocking AI crawlers, or is it your robots.txt?

On 76 of the 1,089 sites the two layers gave opposite answers on the same day. The robots.txt file permits GPTBot at the site root, evaluated under RFC 9309 group selection and longest match, and the server then refused the GPTBot user agent while serving both a browser and an unknown bot token from the same address moments earlier. Nineteen more refusals came from sites whose robots.txt disallows GPTBot anyway, where the two layers agree, and eight came from sites whose robots.txt could not be read well enough to say.

The status codes are worth reading rather than summarising. Of the 103 refusals, 74 were 403, eight were 402, five were 429, four were 503, three were 400, two each were 404, 406 and 502, and one each were 416, 451 and 509. A 403 is a refusal. A 429 is a rate limit, which on a single request from a cold address is a refusal wearing different clothing. A 400 and a 416 are the more interesting shapes, because a malformed-request error in response to a legal GET that differs only in its user agent header is a filter reporting itself badly rather than a server having trouble parsing anything.

Twelve of the named cases reproduced exactly on a second independent request later the same day: justice.gov, nyc.gov, loc.gov, seattle.gov, ontario.ca and un.org each returned 200 to a browser, 200 to LantadBot and 403 to the GPTBot string, as did salesforce.com, sap.com, nike.com, blackrock.com, wiley.com, cornell.edu and imperial.ac.uk. The robots exclusion protocol has nothing to say about any of it, because none of these refusals is a robots.txt outcome. This is the second layer described in the two layers that decide if AI can read your site, measured here at a scale that lets the layers be compared, and it extends the earlier finding that 79 of 115 sites refused a crawler their robots.txt allows by asking which vendor's product returned the refusal.

Which providers refused the GPTBot string most often

Grouping the 1,089 sites by the infrastructure fingerprint on the browser response gives a rate per provider rather than a raw count, which is the only way the comparison means anything when Cloudflare fronts four times as many sites in this corpus as Akamai does. The fingerprint is read from headers that identify the vendor rather than the customer's configuration: a cf-ray header or a cloudflare server header for Cloudflare, x-amz-cf-id or a cloudfront via for Amazon CloudFront, an akamai server header or x-akamai-transformed for Akamai, x-fastly-request-id for Fastly, x-vercel-id for Vercel, x-nf-request-id for Netlify.

Two columns are needed, not one. The GPTBot column counts refusals of the documented string. The LantadBot column counts refusals of a token that appears in no vendor's published list and that no bot manager ships a rule for, so it stands in for how the same infrastructure treats automation it has never heard of. The gap between the columns is the part that is about the name rather than about bots.

On Cloudflare that gap is the whole story: 3 of 344 sites refused the unknown token and 37 refused GPTBot, so the refusal is switched on against a named crawler by customers who are otherwise serving anything that asks. That is consistent with the controls described when a Cloudflare default turned on ads while blocking AI crawlers and with the moving token list recorded in Cloudflare's generated robots.txt. Fastly at 1 and 7 of 77 has the same shape. Akamai does not: it refused the unknown token 17 times, so a fifth of the Akamai fronted sites in this corpus turn away automation as a class before the name is considered at all. GPTBot remains the most refused token measured on every provider where the two columns differ.

Akamai fronted 87 of these sites and refused 32 of them

Thirty-seven per cent against eleven is a large enough difference to be worth stating and a small enough sample to be worth qualifying twice. Akamai fronted 87 of the 1,089 sites and refused the GPTBot string on 32 of them. Cloudflare fronted 344 and refused 37.

The obvious objection is that these are not the same sites. Akamai's customers in this corpus are airlines, banks, retailers and government departments, and those are exactly the organisations that would block an AI crawler whoever sold them their edge. The corpus makes a partial test possible, because it is stratified, so the comparison can be run inside a single stratum where the kind of organisation is held roughly constant. It survives in four strata and reverses in two. Among travel sites, Akamai refused 9 of 20 against Cloudflare's 2 of 15; in ecommerce, 9 of 19 against 0 of 22; in software, 3 of 3 against 6 of 50; in government, 2 of 4 against 1 of 13. In finance it reverses, 3 of 24 against 0 of 11, and in healthcare it reverses more sharply, 2 of 4 against 8 of 25.

Half those cells hold fewer than 15 sites, which is too few to carry a rate, and the honest summary is therefore narrower than the headline figure: the association holds in the strata where both providers have enough sites to compare, and this corpus cannot separate the vendor's defaults from its customers' intentions. Nobody outside the two companies can, because the configuration is private. What can be said is that a site owner who finds an AI crawler refused has to look at the product in front of their origin, not only at a text file, and that the same infrastructure also turns away real browsers running under automation, as recorded when headless Chromium was blocked on 15 per cent of sites. Which organisations block at all continues to split by sector rather than by size, the pattern found when this corpus asked who blocks AI crawlers.

The site builders refused almost nobody

The sharpest contrast in the run is not between two content delivery networks. It is between the enterprise edge and the managed platforms that small businesses actually use.

Five strata of this corpus are drawn from hosted site builders and application platforms: Wix and Squarespace, Webflow, Shopify storefronts, Framer, and single-page application startups. Two hundred and five of those sites answered a browser with 200 on 26 September 2026 and two of them refused the GPTBot string. Both were in the Framer stratum. Wix and Squarespace refused none of 54, Webflow none of 42, Shopify none of 34, and the single-page application startups none of 44. The industry corpus, drawn from named organisations in eight sectors, refused 110 of 706.

The reason is not that these platforms have thought harder about AI crawlers. It is that a customer on a hosted builder has no bot manager to configure, so there is nothing to switch on, and the platform's own default is to serve. That cuts both ways and the earlier platform measurements show which way: robots.txt on these hosts is generic to the point of silence, with 0 of 69 Wix robots.txt files naming any AI crawler and 20 of 40 Webflow files holding no rule at all. A site that cannot block is not a site that has decided to allow. It is a site with no control either way, which is a different and less comfortable position than the access numbers alone suggest.

For a reader on one of these platforms the practical consequence is that the access half of the problem is already solved and nothing they do to robots.txt will change it. For a reader behind an enterprise edge it is the reverse: the file is likely fine and the refusal is somewhere they may not have looked.

46 sites disallowed GPTBot in robots.txt and served it anyway

A post that only counted servers refusing what the file permits would be telling half of a symmetrical story, and the other half is the more awkward one for anyone who treats robots.txt as a control.

Sixty-six of the 1,089 sites carry a robots.txt that disallows GPTBot at the site root. Forty-six of those 66 answered the GPTBot user agent with HTTP 200. The file said no and the server handed over the page, which is exactly what the specification describes: robots.txt is a request to a well-behaved crawler and not an access control. Among the 46 are amsterdam.nl, wa.gov, unesco.org, jstor.org, helsinki.fi, healthline.com and abc.net.au. Nothing in that list is a misconfiguration. It is what a robots.txt directive is, observed working as designed, and it is the same asymmetry recorded when a GPTBot ban served a 200 anyway.

Eight of the refusals took a third position. They answered 402, the status Cloudflare's pay-per-crawl uses, which this blog has already noted runs on an undefined status code. Two of the eight, japantimes.co.jp and totvs.com, carry a robots.txt that permits GPTBot while the edge asks for payment, and both reproduced on a second request. That is not a contradiction so much as a division of labour, with the file left as published policy and the commercial decision moved to the edge, and a crawler reading only the file would not learn that a toll exists.

Taken together the two directions say the same thing about measurement. Reading a robots.txt file tells you what a site has written down. It does not tell you what the site will serve, in either direction, and the only way to know is to ask the server, which is why asking 1,056 sites for their robots.txt and reading the file are two different measurements.

What to check on your own site, and what you cannot check

The useful conclusion is procedural rather than a number. Four checks separate the layers, and one of them cannot be done from outside at all.

Request your own home page twice from the same place, once with a browser user agent and once with the GPTBot string, and compare the status codes. If they differ, the refusal is at the server or in front of it and no robots.txt edit will move it. If they match, the file is the only thing left to read. What GPTBot sees runs that comparison, and this scanner publishes its own crawler token and address behaviour so a site owner can tell its requests apart from anyone else's in a log.

Then find out what is actually in front of your origin, because the response headers name the vendor even when they do not name the rule. If a cf-ray, an x-amz-cf-id or an akamai header comes back, the bot control worth auditing is in that vendor's dashboard and not in your repository. This is where the 33 sites with no CDN fingerprint matter: a third of the refusals in this run came from an origin with no recognisable edge in front of it, so the answer is not always a CDN and the check has to be run rather than assumed.

The third check is the one that cannot be completed from outside, and it is the honest limit on this entire post. Neither this scanner nor a site owner can send a request that a bot manager will accept as GPTBot, because the address will be wrong. OpenAI's crawler documentation publishes the IP ranges and the current user agent string, which reads GPTBot/1.4 while this run sent GPTBot/1.2, and the gap between claiming a name and proving it is precisely what web bot authentication exists to close. Until that is widely deployed, the only place the truth is written down is your own server logs, filtered to the published ranges. Everything measured here is a claim about what a site does with a string, on one date, from one address.