234 of 592 sites that ban GPTBot in robots.txt served it a 200

What HasData measured, and on how many domains

The index is built from two segments pulled into one sample. HasData states it took the Tranco top 10,000, naming the list as 46XLX dated 9 July 2026, and added Ben Welsh's news-homepages set of 1,148 sites, deduplicating the union to 10,894 registrable domains. Naming the list revision and its date matters more than it looks. Tranco regenerates, so a top 10,000 without a list identifier is not a reproducible population, and a study that gives one can be re-run against the same set later.

The first half of the work is the part every robots.txt survey does: fetch each domain's file and parse what it declares. On that reading HasData reports 10.3 percent of the whole web segment blocking any AI crawler, against 56.4 percent of the news publishers. For GPTBot specifically the split is 7.9 percent against 50.5 percent. The ratio between the two populations is roughly five to one on both measures, which is the finding most of the coverage has repeated, and it is the least surprising thing in the study. News publishers have been the loudest constituency on AI crawling for two years, and their files say so.

The same fetch also counted llms.txt files. HasData puts adoption at 7.4 percent across the full 10,894 domain sample, splitting as 7.9 percent in the Tranco segment and 3.2 percent among publishers. Publishers write more robots.txt restrictions than anyone and fewer llms.txt files than anyone, which is coherent rather than odd: one file is a restriction and the other is an invitation. It also lands close to the ground this blog has covered before, where the question was not how many sites publish the file but whether anything reads it, and the evidence on llms.txt is still that most published files are never fetched. Adoption and effect are separate measurements and this study only speaks to the first.

The second half is the part that is rare, because it costs requests rather than a parse. HasData ran its enforcement questions against a 2,096 domain subset, described as the top 1,000 open web domains plus every publisher in the sample. Each of those domains received two requests from a US datacenter IP pinned to the same exit: one carrying the official GPTBot user agent, one carrying a standard Chrome browser string. Holding the network path fixed and varying only the declared identity is the right shape for the question, because it isolates the header as the variable rather than confounding it with the address the request came from.

234 of 592 declared bans returned a 200

Here is the number the study exists to produce. HasData reports that of 592 sites in the enforcement subset that disallow GPTBot in robots.txt, 234, or 39.5 percent, still served GPTBot a live 200 when it knocked. Expressed against the whole tested subset rather than against the sites that declared a ban, that is 234 of 2,096 domains, which HasData gives as 11.2 percent and labels paper-only bans.

Both denominators are worth keeping straight, because the same 234 sites support two very different sentences. Two in five declared GPTBot bans were not applied at the server. One in nine tested domains carries a ban it does not apply. The first is the interesting rate and the second is the population share, and a summary that quotes one figure while implying the other is where this finding will get distorted as it travels.

The split by segment runs against the direction most people would guess. HasData reports that open web sites ignore their own robots.txt more often, at 46.7 percent, than publishers do, at 38.9 percent. So the constituency that blocks AI crawlers five times more often in its files is also the constituency whose files are more likely to be backed by something at the server. That is consistent with publishers having bought infrastructure for the job, and it is the one part of the result that reads as intent rather than accident. Neither figure is given with a raw count on the page, so both are quoted here as percentages only.

What the number does not establish is worth stating with equal force. A 200 to a request carrying the GPTBot name does not prove that OpenAI's crawler is being served, that the site is unaware of its own file, or that any content was taken. It establishes that a request presenting that identity, from that network path, on that date, received a page rather than a refusal. Every one of those qualifiers is doing work. This site has argued before that the two layers deciding whether AI can read your site answer independently, and this study is a measurement of the gap between them from the outside, at one moment, from one vantage point.

It is also worth remembering how much of robots.txt behaviour is decided by the response code rather than the rules, which is why a 404 and a 503 on robots.txt mean opposite things under RFC 9309. A study that reads files can only see files that were served. Domains whose robots.txt was unreachable at fetch time do not declare anything to count, and they are not the same as domains that declared nothing.

An unenforced ban is not a contradiction

The framing that comes naturally here is hypocrisy: sites saying one thing and doing another. That framing is wrong, and getting it right is the whole reason a scanner should write about this study rather than repeat its headline.

RFC 9309, the Robots Exclusion Protocol, was published on the Standards Track in September 2022 and it settles the question in its first section. The specification states that these rules are not a form of access authorization, and its security considerations add that the protocol is not a substitute for valid content security measures. A Disallow line is a request addressed to the crawler, not a control applied to the server. A site that publishes one and leaves its origin open is doing exactly what the standard describes, and a site that publishes one and blocks at the edge is doing something the standard never asked for.

That distinction changes what the 39.5 percent means. Against a crawler that reads robots.txt and obeys it, the Disallow line is the entire mechanism and the open door is irrelevant, because a compliant crawler never turns the handle. The open door only matters against a client that ignores the file. So the figure is not a measure of how often GPTBot gets content it was told not to take. It is a measure of how much of the population has a second line of defence if something disregards the first, and those are different questions with different answers.

Which crawler you are trying to stop decides whether any of this bites. The published tokens are not interchangeable and several of them cannot be separated at all: six of the nine vendors in Lantad's registry publish a single crawler token, so for those vendors a robots.txt file cannot express allow search and block training even in principle. Three of the tokens in that registry send no request user agent of their own, which is why some crawler tokens will never appear in your logs no matter what your file says. OpenAI's crawler documentation is the reference for which of its names does what, and it lists GPTBot as the training crawler rather than the one that fetches pages to answer a live question.

There is a related pattern already covered here, where the instruction and the enforcement live in the same file and still do different jobs: Cloudflare's Content-signal line asks while the Disallow lines under it block. A declaration of preference and a mechanism of refusal are separate things even when they are adjacent on disk. HasData's study is the first measurement this blog has seen that puts a population rate on the distance between them.

The request said GPTBot, which is not the same as being GPTBot

HasData's method sends the official GPTBot user agent string from its own datacenter IP. That is the correct way to test a server's response to a header, and it is not a test of what OpenAI's crawler receives. The User-Agent field is chosen by the client and nothing in HTTP verifies it, which is the argument this blog has made at length in noting that a user agent is a claim, not an identity. The claim cuts both ways. It lets a scraper wear a crawler's name, and it lets a researcher measure one.

Verification is available and it costs a lookup. OpenAI publishes the address ranges its crawlers fetch from, and a request from outside those ranges is not that crawler whatever its header says. Counting them is a small exercise this blog has already done, in four OpenAI crawler IP lists, opened and counted, and the GPTBot range file is public. HasData's requests did not come from those ranges, so a site running verification correctly should refuse them, and a refusal in that case is the site behaving well rather than badly. The study cannot distinguish that outcome from a plain block, and neither can any external scan.

HasData ran a second pass that makes the identity effect visible directly. It reports testing five bots against 250 publisher domains from a residential vantage, described as a single machine with one consistent exit IP, and measuring what share of requests came back successfully. A standard browser string passed on 77.2 percent of those domains. GPTBot passed on 44.4 percent, PerplexityBot on 45.2 percent, Meta-ExternalAgent on 38.0 percent, CCBot on 37.6 percent and ClaudeBot on 36.8 percent. Same machine, same address, same set of sites, with the declared name as the only difference, and roughly half the browser's success rate.

That is the clearest thing in the study, and it points at something a site owner can act on. Publishers are not mostly filtering by network origin here, because the network origin was held constant. They are filtering on the name in the header. Every one of the five names tested sits in Lantad's own crawler registry, which the AI crawler checker exposes, and the spread across them is narrow: about eight points separates the most and least successful of the five, against a gap of more than thirty points from any of them to the browser. Whatever these sites are doing, they are treating declared crawlers as a class rather than picking favourites among them.

What you can check on your own domain

Population rates are not verdicts about any one site, and this one is less transferable than most. The sample is the Tranco top 10,000 and a news publisher set, so it describes large and heavily defended properties. If you run a smaller site, the useful part of this study is the method rather than the percentages.

The method reduces to three questions you can answer today. First, what does your robots.txt actually say, per crawler, rather than per wildcard. That is a parse and nothing more, and the robots.txt tester will evaluate a file against the tokens individually, which matters because a wildcard group does not reach every agent a vendor operates. Second, what does your server return to a request that is not a browser. Third, whether those two answers agree, which is the whole of what HasData measured at scale.

Lantad answers the first and the third only partially, and the limit should be stated rather than glossed. The scanner fetches with its own declared identity, LantadBot, not by impersonating anyone else's crawler, so it can tell you what your file permits GPTBot to do and what your server does for a non-browser client, but it does not send GPTBot's name to find out how you treat that specific string. The crawler view of your page is the honest version of that question: what a JavaScript-blind fetch actually gets back. Impersonating a named vendor's crawler to test a site is a defensible thing for a one-off study to do and a bad default for a tool that runs continuously.

That gap is also why this site withholds a grade rather than converting a partial observation into a confident number, and why the methodology page states what each component is fetched with. A measurement is only as good as the identity it was taken under, and reporting the identity is cheaper than defending a figure taken without one.

The broader point for anyone working on generative engine optimization is an ordering one, and it keeps arriving from different directions. Whether a page is reachable is settled before anything about how it is written matters. HasData's study says that for a meaningful share of large sites, the file and the server give different answers to that question, and that nobody finds out which answer counts until a request arrives.