robots.txt testen: six files, and not one AI crawler named

robots.txt testen: what the test actually decides

A robots.txt test takes three inputs and returns one verdict. The inputs are the file, a product token naming the crawler you are asking about, and a path. Everything difficult sits between them, in two decisions the file itself never states and the person reading it rarely makes consciously.

The first decision is group selection. A robots.txt file is a sequence of groups, each headed by one or more user-agent lines and followed by rules. Asking whether GPTBot may fetch a page means first asking which group governs GPTBot, and a file that names GPTBot nowhere sends the question to the wildcard group instead. The second decision is rule precedence inside the winning group. A group holding both a broad disallow and a narrow allow needs a tie-break, and the tie-break is not obvious from reading the lines in order.

Neither decision is visible in the file. This matters more than it sounds, because it means two testers can read the same bytes, apply two defensible rulesets, and hand a site owner two different answers without either of them being broken. That is the failure mode this post is really about, and how this scanner reaches a verdict is written down for the same reason.

The verdict a test returns is also narrower than the question most people are asking. Allowed means the file grants permission to a client that identifies itself with that token and chooses to honour the file. It does not mean the crawler came, that it obeyed, or that it could read anything once it arrived. This scanner publishes the user agent it sends and the address it sends from precisely so a site owner can separate the permission from the behaviour in their own logs, which is the only place the two can be told apart.

How many of six real sites name an AI crawler in robots.txt?

Zero of six. That is the whole finding, and it survived every way we looked at it. The six files declare 48 user-agent groups between them: six at allbirds.com, one at astro.build, six at gymshark.com, one at developer.mozilla.org, one at webflow.com and 33 at en.wikipedia.org. Searching the raw bytes of all six for any of the strings that would name an AI crawler, including gpt, claude, anthropic, perplexity, extended, bytespider, ccbot and amazonbot, returns nothing on any of them. This is a different measurement from the one we published when we fetched robots.txt across all 14 hostnames of the same six sites and found three of the six serving a different file on a second hostname. That post was about which file you get. This one is about what is inside it.

Two of the six are the same file with different content. allbirds.com and gymshark.com declare an identical set of seven user-agent tokens, being the wildcard plus AdsBot-Google, Nutch, AhrefsBot, AhrefsSiteAudit, MJ12bot and Pinterest, which is the platform default rather than a decision either merchant made. The files are not byte identical, at 5,495 and 3,123 bytes, so the rules diverge while the token list does not. Anyone running a Shopify storefront is starting from that list unless they have changed it, and the list predates every AI crawler on the web.

Wikipedia is the interesting case because it disproves the obvious explanation. Its file is 28,275 bytes and names 32 crawlers, so this is not a site that never got around to writing rules. Every one of the 32 is an old scraper, a mirroring tool or an SEO crawler: HTTrack, Teleport, WebZIP, Xenu, larbin, libwww, wget, MJ12bot, SemrushBot and their contemporaries. The file is a careful, maintained document that was aimed at a previous decade's traffic. A robots.txt that a platform generates for you moves its token list on its own schedule, but a hand-maintained file moves only when a person edits it, and nobody edited these for the crawlers in our public crawler directory.

RFC 9309 and Google disagree about which group wins

Two documents describe how a crawler picks its group, and they do not say the same thing. RFC 9309, published in September 2022, states that crawlers MUST use case-insensitive matching to find the group that matches the product token, and that if there is more than one group matching the user-agent, the matching groups' rules MUST be combined into one group. Only when no group matches does it send the crawler to the wildcard.

Google's robots.txt documentation, last updated 31 August 2026, describes something else. Google's crawlers, it says, find the group with the most specific user agent that matches the crawler's user agent, that only one group is valid for a particular crawler, and that other groups are ignored. It adds that all non-matching text is ignored, giving googlebot/1.2 and googlebot* as both equivalent to googlebot. On rule precedence it is more specific than the RFC as well: the most specific rule by the length of the rule path wins, and where rules conflict Google uses the least restrictive one.

This scanner implements Google's version. Group selection picks the single longest matching token and merges only the groups sharing it, and rule precedence is longest pattern wins with allow taking ties. That is a decision, made because it describes what the largest crawler operator says it actually does, and it is recorded in the header comment of the file that implements it. It is not a measurement, and nothing here establishes that any AI crawler follows Google's rules rather than the RFC's. The vendors publish tokens, not parsers.

The decision had no consequence on this sample, and saying so is more useful than leaving it implied. We evaluated 15 tokens against three paths on six sites, which is 270 evaluations, under both rulesets. They agreed 270 times out of 270. They could not do otherwise: a divergence needs a file that names one crawler in two separate groups, and no file here names one in even a single group. The same thing happened when we tested whether a trailing wildcard in a disallow pattern changed any verdict and it changed nothing. Parser edge cases are real, and they are not what is costing sites their access. Our standing measurements keep landing on the plainer failures instead.

Which crawler tokens should a robots.txt test check?

A per-crawler test needs a list of crawlers, and the list is a choice somebody has to make and keep current. This scanner's registry holds 15 tokens. That number is a setting, not a finding, and two things we read on 5 September 2026 show how quickly such a list goes out of date.

OpenAI's crawler documentation names four bots: OAI-SearchBot for surfacing sites in ChatGPT's search features, GPTBot for the foundation models, ChatGPT-User for certain user actions, and OAI-AdsBot for validating the safety of pages submitted as ads. Our registry carries three of them and not OAI-AdsBot. Anthropic's support article documents three tokens, being ClaudeBot for model training, Claude-SearchBot for search quality and Claude-User for pages fetched when a person asks. Our registry carries those three plus a legacy anthropic-ai token that the vendor page does not document at all. So on the two vendors we checked, the list is short by one in one direction and long by one in the other.

One entry on OpenAI's page carries a caveat that a test result cannot express. The page states that ChatGPT-User is not used for crawling the web in an automatic fashion, and that because these actions are initiated by a user, robots.txt rules may not apply. A tester reporting ChatGPT-User: allowed is reporting the file's contents accurately and telling you less than it appears to, because the vendor has said in its own documentation that the file may not govern that traffic.

The tokens also have to survive being matched. Group selection here treats a group token as matching when it is a prefix of the bot token, so a group headed Claude would catch ClaudeBot, Claude-SearchBot and Claude-User together. None of the 15 tokens in the registry is a prefix of another, so that behaviour costs nothing internally, and it is still a rule a file author cannot see. When we last checked which of these tokens publish a request user agent at all, the split mattered for a different reason: a robots-only token has nothing to detect in a log. If you are working out what to allow for ChatGPT specifically, the search bot and the training bot are separate decisions, and declaring a bot is not the same as verifying one.

What the wildcard group actually allows on these six sites

Since every AI crawler on all six sites lands in the wildcard group, that group is the only thing a per-crawler test on these sites is really reading. It is worth stating what it holds, because the volume of rules turns out to have nothing to do with whether an AI crawler gets in.

The wildcard groups carry 47 disallow rules and no allow rules at allbirds.com, 34 disallow and 2 allow at gymshark.com, 3 disallow at developer.mozilla.org, 5 disallow and 2 allow at webflow.com, 428 disallow and 4 allow at en.wikipedia.org, and at astro.build a single Allow: / under a comment welcoming our new robotic overlords. Every one of the six allows the site root, for all 15 tokens.

Of the 270 evaluations, 30 came back disallowed, and all 30 were the same path on the two storefronts: /search at allbirds.com and at gymshark.com, decided by the same Disallow: /search line in the wildcard group, fifteen times identically on each. That is the whole finding in one number. A file with 432 rules and a file with one rule return the same verdict for GPTBot, and where a verdict does change it changes for all fifteen crawlers at once, because the rule that changed it was never written about any of them.

Wikipedia's file also shows what a robots.txt looks like when nobody is enforcing the grammar. RFC 9309 states that a product token MUST contain only letters, underscores and hyphens. Four of Wikipedia's user-agent values do not: offline explorer and download ninja carry spaces, and sitecheck.internetseer.com and microsoft.url.control carry dots. A fifth, mediapartners-google with a trailing asterisk, is the exact case Google's documentation says is ignored as non-matching text. None of this breaks anything, and all of it is the kind of thing a parser has to decide about silently.

The practical reading is that a wildcard-only file is a policy of not having a policy, which is a legitimate choice and is increasingly not the choice the platform makes for you. Cloudflare's own change means new domains onboarding to it get defaults that depend on whether a page carries ads from 15 September 2026, and a proposed well-known ai.txt that defaults training to deny would flip the default the other way. Both would govern sites whose robots.txt still says nothing at all.

What a robots.txt test cannot tell you

The honest limits are worth naming, because a green result on a file that says nothing is easy to read as a clean bill of health and it is not one.

It cannot tell you what the crawler did. The file is a statement of permission by the site and nothing in it observes a request. A crawler that ignores the file produces exactly the same test result as one that honours it, and the only place the difference exists is your own access log.

It cannot tell you what was served to the crawler at the other end. That was the point of the fourteen-hostname measurement referenced earlier: the file governing the page a crawler is fetching is the file at that page's hostname, and three of these six sites answered differently on a second hostname.

It cannot read a field it does not know. Exactly one of the six files carried anything our parser could not classify, and it was webflow.com, whose second line is a Content-Signal field declaring ai-train, search and ai-input all yes, sitting inside the wildcard group between the user-agent line and the rules. An RFC 9309 parser records it as an unknown field and ignores it, which is correct behaviour and also means the site's clearest statement about AI use is invisible to the test. That mechanism has its own coverage problem, since only one of ten Cloudflare crawl endpoints reads those directives.

And it cannot tell you whether there was anything to read. Permission to fetch a page is not the same as text in the response, which is why what a crawler receives before any script runs is a separate measurement from this one, and why five of five canonical tags pointing at themselves was worth checking separately too. On this sample the robots layer was not the constraint on any of the six sites. Everything expensive was further down.