robots.txt allowed all 60 AI crawler probes, and 11 got a different answer

What a user agent probe actually does, and what 60 of them covered

The scanner fetches the page once as itself, honestly identified, and that response is the baseline. It then sends one further request per crawler token, changing exactly one thing: the user agent header. OpenAI's crawler documentation publishes the full strings, giving GPTBot as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, and the same page documents OAI-SearchBot, OAI-AdsBot and ChatGPT-User with their own strings. It carries no date, which is worth noting whenever anybody quotes a token from it. The probe sends the published string and nothing else, because a user agent is a claim rather than an identity and there is no way to send a real one.

The registry holds 15 tokens. Three of them, anthropic-ai, Google-Extended and Applebot-Extended, exist only as names in robots.txt and have no request user agent behind them at all, and every fixture records them with the note that there is no request user agent to probe. That leaves 12 tokens that can be sent. Of the six real captures, allbirds.com answered the honest baseline fetch with HTTP 429, and the pipeline skips probing entirely when the baseline is already an error, so that capture stores 15 rows all reading not probed. Five pages times 12 tokens gives 60 probes that actually ran.

Each probe compares two things. First the status code, against the baseline status. Then, when the statuses match, the visible text of the two responses, normalised the same way prose parity normalises it, compared by containment in both directions with the lower of the two figures taken as the similarity. A raw byte hash would fire on every nonce and CSRF token on the page, so the comparison runs over extracted text rather than bytes. Above a floor of 0.9 the two count as the same page. Below it, with an HTTP 200 and a baseline of at least 200 visible tokens, the verdict is that a different page was served. Below the floor with too short a baseline, the scanner declines to judge and records why, because on a page of forty words a single clock or counter drops the similarity far enough to look like evidence when it is noise. The raw response for one of these tokens is the same view the GPTBot preview tool shows for any URL.

robots.txt permitted 60 of 60, and 11 responses did not match

Across the five probed captures, the robots allowed field reads true on all 60 rows. Not a single one of those pages disallowed a single one of those tokens on the day of capture. astro.build reached that answer explicitly, storing the matched rule as Allow: / against every token. The other four store no matched rule at all, which is the shape of permission by silence: nothing in the file applied to the token, so nothing refused it.

That is the correct outcome under the specification. RFC 9309, published in September 2022, describes rules that crawlers are requested to honour when accessing URIs, and it settles the question of what those rules are in a single clause that anyone quoting robots.txt as a control should read out loud: these rules are not a form of access authorization. Permission to request is not delivery of content. A file that grants a crawler access grants it nothing that the server in front of the file has to honour, and the server does not consult it.

The 60 responses split three ways. Forty nine matched the baseline in both status and text. Four returned HTTP 403 where the honest baseline had returned 200. Seven returned HTTP 200 with visible text matching the baseline at 87 percent, below the 0.9 floor, on a page whose baseline carried far more than the 200 token minimum. Eleven of 60 is 18.3 percent of the probes that ran, and every one of those 11 sits on a page whose robots.txt raised no objection whatsoever.

The direction of that disagreement is the reverse of the one measured elsewhere in this field. A third party study of 592 sites that disallow GPTBot in robots.txt found 39.5 percent of them served it a live 200 anyway: the file said no and the server said yes. Here the file said yes and the server said no or served something else. Both readings point at the same structural fact, which is that two layers decide whether AI can read your site and only one of them is the file you edited. If you want to see what your own file resolves to per token before you look at the responses, the robots.txt tester evaluates it the way this scanner does.

Two tokens from the same vendor got opposite answers on the same page

The 11 disagreements are not spread evenly. They sit on two of the five pages, and within those two pages they sit unevenly across tokens from the same company.

gymshark.com returned a matching page to GPTBot and to Claude-SearchBot and Claude-User. It returned text at 87 percent similarity to OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Perplexity-User, Meta-ExternalAgent and Bytespider. It returned HTTP 403 to CCBot and to Amazonbot. So on one page, in one four minute window, OpenAI's training crawler received the baseline page and OpenAI's search crawler did not, and Anthropic's training crawler received a different page while Anthropic's search and user tokens received the baseline. The stored note is identical on all seven of the mismatches, giving the same similarity figure and the same baseline hash, which means all seven probes differed from the baseline in the same way rather than each seeing something of its own.

en.wikipedia.org is the cleaner case and the sharper one. Ten of its 12 probes matched. ClaudeBot received HTTP 403. Meta-ExternalAgent received HTTP 403. Claude-SearchBot and Claude-User, from the same vendor as ClaudeBot, received 200 with matching text. A per vendor summary of that page would be wrong in both possible directions: Anthropic is neither blocked nor unblocked there, it is one token blocked out of three.

This is the practical argument for keeping a matrix rather than a verdict per company. Vendors publish separate tokens for training, search and user initiated fetches, a site can act on one and not the others, and on these captures at least one site did exactly that. It is also the argument against reading a platform as a policy: gymshark.com and allbirds.com both run on Shopify, they produced entirely different outcomes on the same day, and the Shopify guide treats the bot defence sitting in front of the store as the variable rather than the store. Whether the differences here were configured deliberately is not something a probe can establish, and this post does not claim it. The tokens that never appear in your logs are a reminder that the same matrix can be sparse for reasons that have nothing to do with intent.

Our own scale prices a 403 and does not price a different page

Both of those captures stored an access sub-score of 0.9, and both stored the same confidence note naming the tokens that were blocked and the deduction taken. The deduction is 0.10 in both cases. It comes from a fixed penalty of 0.2 in ACCESS_PENALTIES for observed server level enforcement, multiplied by a confidence scale that reads 0.5 for a low confidence observation. Access is weighted at 25 of the 100 composite points, so 0.10 of access is 2.5 points of the composite.

The seven mismatched bodies on gymshark.com cost nothing. They raise a defect, titled in the code as different content served to AI crawler user agents and marked major, and it appears in the report. It carries no number. The comment in core/src/gains.ts says so directly, that a suspected content difference is flagged rather than priced, and gives the reason: any figure attached to it would be invented. We agree with the reasoning and we are publishing the consequence anyway, because the consequence is that a page which served seven crawler tokens something other than what it served us scored 94.3 and graded A, with the mismatch visible only in a defect line.

That is the same shape as an audit published here earlier, in which a robots.txt blocking every citation crawler still graded B, because access is 25 of the 100 composite points and a full block does not take all of them. A grade compresses. Anything that compresses can hide the row you most needed to read, which is why the matrix is in the report next to the number rather than behind it, and why the withheld grade exists as a state at all.

There is a defensible version of pricing nothing here. HTTP 403 is a definite act by the server: MDN defines it as the server understanding the request and refusing to process it, with re-authentication making no difference, so the refusal is observable and unambiguous even when the reason is not. An 87 percent text similarity is a measurement of difference, and difference has many innocent causes. Pricing it would mean putting a number on a suspicion. What we can say without inventing anything is that the number 0.10 exists for one of these and no number exists for the other, and a reader comparing two reports should know which side of that line each finding sits on.

What a probe cannot tell you, and why the confidence is low

The probe sends a header. That is the whole of its authority, and every limit below follows from it.

A site that verifies crawlers properly does not trust the header either. It resolves the connecting address, and the vendors publish address ranges to make that possible: when this scanner read OpenAI's published IP range files it found four of them, one per token. Our probe is not OpenAI's crawler and does not fetch from any address in them. So a site running verification sees a request claiming to be GPTBot arriving from an address OpenAI does not publish, and refusing it is correct behaviour rather than a decision about GPTBot. The scanner encodes that judgement in the one place it can: when robots.txt already disallowed the token, a 403 is consistent policy and the observation is recorded at high confidence; when robots.txt allowed the token, as it did on all 60 of these probes, the observation is recorded at low confidence and the deduction is halved. Every one of the four 403s in this data set is a low confidence observation for exactly that reason.

The content mismatch carries a limit of its own that is easier to miss. The baseline is fetched once, at the start of the scan, and every probe is compared against that single capture. Nothing in the run re-fetches the baseline to establish how much a page varies between two ordinary requests. On a storefront rotating promotional copy, or a page carrying a live counter, some part of an 87 percent similarity may be that variation rather than anything conditioned on the user agent. The stored evidence is consistent with a user agent effect, since three tokens on the same page in the same window did match, but it does not isolate one, and this scanner reports a suspicion rather than a finding for that reason. Correctly labelling a request is an ordering problem with named failure modes, and this is the mirror of it: correctly labelling a response.

The sample is the last limit and the flattest. Six URLs, one page each, one day, chosen to calibrate a scoring engine rather than to represent anything. Sixty probes is a count and not a rate, 11 of 60 is arithmetic over those 60 and not a claim about the web, and no percentage in this post should be carried past the fixtures it came from. What five pages can do is establish that the disagreement occurs at all, on ordinary well run sites, in a direction most tooling does not look for.

How to run the same two checks on your own site

Both halves of this are reproducible with a terminal and about a minute, and the point of running them yourself is that the disagreement, if you have one, lives between two systems that are usually owned by different people.

Read the file first, and read it per token rather than as a whole. A single wildcard group, a group naming one token and not its siblings, or a stale group naming a token that has since been renamed will all resolve differently for different crawlers, and a renamed token leaves the old rule matching nothing at all. The reachability of the file matters as much as its contents: a 404 and a 503 on robots.txt are opposites under the specification, one meaning no rules apply and the other meaning a careful crawler should stay away entirely, so a file that fails to serve is a different situation from a file that permits everything.

Then send the requests. Fetch your page once with an ordinary browser user agent and once with each published crawler string, and compare the status codes before you compare anything else. A status difference is the unambiguous signal and needs no similarity threshold to interpret. If the statuses match, compare the visible text rather than the bytes, because analytics identifiers, CSRF tokens and timestamps will differ on any two fetches of the same page and will make a byte comparison useless. If your infrastructure sits behind a CDN with a managed bot rule set, the rule that produced a 403 will be visible in that dashboard and not in your robots.txt, which is the whole point.

What you are looking for is not a bad grade. It is a row where the file and the response disagree, because that row is the one nobody is responsible for: the file was written by whoever owns SEO and the response is produced by whoever owns the edge, and neither of them is looking at the other's output. The current list of crawler tokens is the input to both checks, and it changes, which is a reason to run this more than once a year.