AI crawler robots.txt: ChatGPT's file disallows a token no crawler can match

What should an AI crawler robots.txt file contain?

The format is defined by RFC 9309, the Robots Exclusion Protocol, and the parts that decide an AI crawler's fate are short enough to read in full. A file is a sequence of groups. A group opens with one or more user-agent lines and continues with allow and disallow rules until the next user-agent line begins a new one. A crawler reads the file, finds the group that names it, and obeys that group only.

Three rules in that specification do the work, and each of them is a place a file can go wrong. First, the name itself. Section 2.2.1 of RFC 9309 defines a product token by an ABNF production that reads identifier = 1*(%x2D / %x41-5A / %x5F / %x61-7A). Those four ranges are the hyphen-minus, the capital letters, the underscore and the lower case letters. Nothing else is a product token. Second, the matching is case-insensitive, which is why amazon.com writing meta-externalagent in lower case still binds Meta's crawler. Third, and this is the one that matters here, the specification says that when no group explicitly matches the crawler's identifier it must obey the group whose user-agent value is an asterisk, if one is present.

That third rule is a fallback, and a fallback is the opposite of an error. A crawler that cannot find its own name does not stop, log a warning or refuse to proceed. It quietly reads the wildcard group instead and behaves exactly as a well behaved crawler should, which is why a typo in a user-agent line produces no symptom anywhere. We have written before that a renamed crawler token leaves your robots.txt group matching nothing, and the failure mode here is the same one arriving from the other direction: the vendor changed nothing and the file was wrong when it was written.

The reason AI crawler groups exist separately at all is that most vendors now ship more than one token per product, split by purpose. Our AI crawler entry sets out the vocabulary, and the split matters commercially rather than technically: a training token returns nothing to the site it read, while a search or user token is the path a citation can travel back along. That asymmetry is the whole argument for writing per-token rules instead of one blanket line, and it is also why we have said that six of the nine vendors we track publish exactly one crawler token even though the products behind them do several jobs.

One more property of the format is worth stating before the results, because it explains why we requested 20 hostnames rather than 9 companies. A robots.txt file governs one origin. It is fetched from the host being crawled, and a rule written on one hostname says nothing about another, which is the finding behind three of six sites serving a different file on a second hostname. A company with a marketing domain and a product domain has two policies whether it meant to have two or not.

What twenty AI company robots.txt files actually say

The 20 hostnames were openai.com, chatgpt.com, anthropic.com, claude.ai, claude.com, perplexity.ai, google.com, gemini.google.com, microsoft.com, copilot.microsoft.com, apple.com, meta.com, bytedance.com, commoncrawl.org, amazon.com, x.ai, grok.com, you.com, mistral.ai and deepseek.com. Seven of the requests were answered after one redirect to a www host, and all 20 returned HTTP 200 with a plain text body. Nothing here required a browser, a rendering step or an account.

Five of the 20 files name at least one of the 15 tokens in this scanner's registry: amazon.com names 12, grok.com and x.ai name 6 each, chatgpt.com names 5 and claude.ai names 4. The other 15 files name none of them. That set of 15 includes openai.com at 98 bytes, anthropic.com at 71 bytes, claude.com at 107 bytes, mistral.ai and deepseek.com at 70 bytes each, you.com at 59 and copilot.microsoft.com at 42, all of which consist of a wildcard group allowing the site with a handful of exclusions. It also includes commoncrawl.org, whose 4,508 byte file names no AI crawler token at all, including its own CCBot.

Three of the 20 refuse at least one of the 15 tokens at the site root, and the counts are 12 for amazon.com, 5 for chatgpt.com and 4 for claude.ai. Amazon's file is the outlier by size and by intent: 7,887 bytes across 101 user-agent lines, most of them a token followed by a bare Disallow of the whole site, and the list runs well past the tokens we track to names such as TavilyBot, xAI-Grok, webzio-extended and WRTNBot. The three of our 15 that amazon.com does not name are anthropic-ai, Applebot-Extended and Amazonbot, the last of which is Amazon's own.

The distribution is the first result worth stating plainly, because it is the opposite of what a reader might expect from a set of companies this technical. Three quarters of these files treat AI crawlers exactly as an ordinary site does, which is to say not at all, and the ones that do write per-token rules are mostly writing them about somebody else's crawler. Whether that is the right policy for an ordinary site is a separate question, and the honest answer has always been that a robots.txt block does not reliably stop a chatbot and that sites banning GPTBot in robots.txt served it a 200 anyway at scale. The file is a request, and enforcement lives at the edge. What it does do, reliably, is tell a compliant crawler what you want, and the rest of this post is about the cost of getting that sentence wrong.

One character turned a Disallow group off

The file at chatgpt.com is 4,302 bytes, was served by Cloudflare with a Last-Modified header of 6 September 2026, and is actively maintained rather than left over from 2023. It opens with a run of short groups, most of them naming one crawler and refusing it the whole site. Among them, in the order they appear, are CCBot, img2dataset, Google-Extended, anthropic-ai, Claude-Web, Omgilibot, Omgili, FacebookBot, Bytespider, magpie-crawler and PerplexityBot, and then one more group written for Perplexity's user-initiated token. That last one is where the file stops working.

The user-agent value on that line is not Perplexity-User. Between the word Perplexity and the word User sits U+2011, the non-breaking hyphen, rather than U+002D, the hyphen-minus. We checked this rather than assumed it: we fetched the file twice, once as LantadBot and once with a desktop Chrome user-agent string, and the two responses were byte identical by SHA-256. We then scanned the whole file for characters outside the ASCII range, and that one is the only one in all 4,302 bytes. The two byte difference between the 4,302 byte Content-Length and the 4,300 characters the file decodes to is the same character showing up again, since U+2011 costs three bytes in UTF-8 and one in the count.

Read the ABNF against that. A product token is built from hyphen-minus, letters and underscore, and U+2011 is none of those, so the value on that line cannot equal any conformant product token. Perplexity's own crawler documentation names exactly two: PerplexityBot, which it describes as surfacing and linking sites in Perplexity's search results and not crawling content for foundation models, and Perplexity-User, which it describes as visiting web pages when a user asks a question and as not being used for training. The token Perplexity publishes carries an ordinary hyphen. The group written for it at chatgpt.com does not.

So the two Perplexity tokens, written one after the other and evidently meant to be treated the same way, get opposite answers. PerplexityBot matches its group and is refused everything. Perplexity-User matches no group, falls to the wildcard group under the rule quoted above, and inherits its 165 rules. We evaluated five paths against both. PerplexityBot is refused all five. Perplexity-User is allowed the home page, the pricing page, the shared conversation paths under /s/ and the custom GPT paths under /g/, and is refused only a conversation URL under /c/. Those allowed paths are not incidental: they are the ones the wildcard group was written to open up.

None of this says anything about whether Perplexity-User actually visits. It is a statement about what the file instructs, which is the only thing a robots.txt file can ever be, and it is the reason our robots.txt tester evaluates a real token against a real file rather than reporting whether a rule looks present. A rule that looks present and matches nothing is the exact output a string search cannot tell apart from a rule that works.

Two of the Anthropic tokens named there are ones Anthropic does not document

The same file carries a second, quieter version of the same problem, and this one needs no typo. Two of its groups name Anthropic crawlers: anthropic-ai and Claude-Web. Both are refused the whole site. Neither of them appears in the vendor's current documentation.

Anthropic's crawler help article, which carries an updated date of 7 April 2026, documents three user agents and no others: ClaudeBot for collecting web content used in model development, Claude-User for user-initiated access when somebody asks Claude a question, and Claude-SearchBot for indexing content to improve search result quality. Read the two lists against each other and the overlap is empty. The file names two Anthropic tokens, neither of which the vendor documents, and none of the three the vendor does.

The consequence follows the same path as the hyphen. ClaudeBot, Claude-User and Claude-SearchBot match no group at chatgpt.com, so all three take the wildcard group and reach the home page, the pricing page, /s/ and /g/, which is what the file's other groups were written to refuse. The intent is legible. The instruction is not the intent.

This is not a criticism that only lands on one company, and it is the most common robots.txt defect we see anywhere. A token list written once ages against a vendor's release notes, and nothing anywhere tells the site owner when a name it depends on stops existing or a new one starts. It is why we published that three AI crawler tokens will never appear in your logs and why our AI crawler directory carries the tokens rather than leaving them in prose. The practical rule is that a robots.txt group is a dependency on somebody else's naming decisions, and dependencies need re-reading.

There is a second-order effect worth naming for anyone writing rules for getting cited by Claude or by ChatGPT. Blocking a search token and blocking a training token are commercially opposite decisions, and a stale name means you do not know which one you made. A file that refuses anthropic-ai and Claude-Web while admitting Claude-SearchBot has, in effect, taken the decision that a reader would have described as the reverse of the one written down.

Which AI crawlers do the AI companies refuse?

Counting across all 20 files gives a ranking of which tokens these companies bothered to name. Google-Extended leads it, named in 5 of the 20 and refused at the site root by 3 of them, which makes it both the most named and the most refused token in the set. GPTBot, ChatGPT-User and PerplexityBot are each named in 4 files, ClaudeBot in 3, and OAI-SearchBot and Applebot-Extended in 2. Four tokens appear in exactly one file each, and Amazonbot appears in none of the 20, which includes the file served at amazon.com.

Set the refusals side by side and the reciprocity is direct. The file at claude.ai names GPTBot, OAI-SearchBot, ChatGPT-User and Google-Extended in a single group and refuses all four the whole host. The file at chatgpt.com refuses PerplexityBot, Google-Extended, CCBot, Bytespider and the two Anthropic tokens discussed above. Each of the two largest assistant products, in other words, has written a rule against the other's crawler, and both have written one against the token that governs training for Google's models.

Two qualifications keep that from being a bigger claim than the data supports. The first is that a refusal at the root is a refusal to a compliant crawler and nothing more, which is the same limit that applies to every robots.txt finding on this site and the reason our methodology page separates what a file declares from what a server enforces. The second is that these are consumer product hostnames whose useful content sits behind a login, so refusing a training crawler there costs the company very little. The corporate hostnames, where the marketing pages and the documentation live, tell a different story: openai.com, anthropic.com and claude.com name no AI crawler at all and allow the site to everything.

That split between hostnames is worth dwelling on, because it is the same thing an ordinary business gets wrong for entirely innocent reasons. claude.ai carries a 281 byte file that refuses four AI crawlers. claude.com carries a 107 byte file that refuses none and allows everything. Both serve the same product to a reader, and a crawler fetching one learns nothing about the other. If you run a marketing site and an app on separate hostnames, or a documentation subdomain, you have at least two AI crawler policies, and the odds that both were written on purpose are not good. Our what GPTBot sees tool takes one URL for exactly this reason: the answer is a property of the host, not the brand.

Two files name six crawlers each and refuse none of them

The remaining two files that name AI tokens are the most interesting, because they show the two ways a named group can be written without refusing anything.

The file at x.ai names GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended and Applebot-Extended in one group, and gives that group two rules: allow the site, and disallow /tools/. Its wildcard group above carries exactly the same two rules. The named group is therefore a no-op. Whatever a named AI crawler is permitted, an unnamed one is permitted identically, and the file would behave the same if the group were deleted. It is not a mistake, but it is a group that reads as a policy and is not one. The same file also carries two Content-Signal lines declaring ai-train=no, which is a preference rather than an access rule, and the point we made when Cloudflare's content signal line asked while the Disallow lines blocked applies in reverse here: the declaration says no to training while the rules above it say yes to the crawlers that do it.

The file at grok.com names the same six tokens and does something genuinely different. Their group carries nine allow rules, for the home page, /chat, /highlights, /imagine, /plans, /plans/business, /project, /automations and /supergrok/imagine, and then a bare disallow of the site. Under the longest-match rule, and the tie-break that an allow beats an equally specific disallow, those six crawlers may fetch the listed paths and nothing else. The wildcard group above them allows the site and excludes six path prefixes. So a named AI crawler at grok.com is given strictly less of the site than an anonymous one, which is a deliberate and legible position, and the reverse of the usual assumption that naming a crawler in robots.txt is how you welcome it.

The tie-break is worth knowing because it decides this case. RFC 9309 says the most specific match wins, measured in octets, and that where an allow rule and a disallow rule are equivalent the allow should be used. That is why the bare disallow of the site does not close the home page against the allow that names it. It is also why we have measured that a trailing wildcard on a path changed nothing across 168 comparisons: the rules that decide a verdict are usually not the ones an author is looking at.

None of the three files that refuse an AI crawler tells the reader what a refusal costs. That is the part a site owner has to reason about, and the two directions are not symmetric: refusing a training token withholds a page from a model, while refusing a search or user token withholds it from the answer a person is reading right now. Our generative engine optimization entry sets that trade out, and the two layers that decide whether AI can read your site covers the half of it that robots.txt does not reach at all.

What to check in your own robots.txt today

Four checks follow from the 20 files, and all four are cheap. None of them requires a tool, though a tool is faster than a careful read.

Check the spelling against the vendor's own page rather than against a blog post or a template. OpenAI's crawler documentation and the Anthropic and Perplexity pages linked above are the authorities on their own tokens, and a token that arrived in your file from a listicle has an unknown provenance. Copy and paste is the safest method, and it is also how the non-breaking hyphen at chatgpt.com most plausibly arrived, so paste into a plain text editor and not through a rich text field or a word processor that substitutes typographic characters.

Check for characters outside the ASCII range in the whole file. This is the check that would have caught the case above in one second, and it catches the whole family of related faults: a non-breaking space after a colon, a curly quote, a soft hyphen invisible at any font size. Any robots.txt file that needs a character outside plain ASCII in a user-agent or path line has a defect, because the specification does not define one.

Check that each named group actually differs from your wildcard group. A group that repeats the wildcard rules, as x.ai's does, is not wrong, but it records a decision that the file does not implement, and the next person to read it will believe a policy exists. If the intent was to grant more or less than the default, the rules have to say so.

Check the file on every hostname you serve, not just the apex. That is the claude.ai and claude.com split, and it is the most common version of this problem outside the AI industry entirely: an app subdomain, a documentation host and a marketing site that were configured by different people in different years. If you want one number for whether an AI crawler can read a given page, our robots.txt tester evaluates a token against a live file, and a full scan reports what the crawler receives after the file has been obeyed. If you would rather read our own conduct rules first, they are on the bot page.

Finally, remember what the check can and cannot tell you. A robots.txt file is a set of instructions to a compliant client, and every finding in this post is a statement about instructions. We did not observe a single crawler request, we cannot say whether Perplexity-User has ever fetched chatgpt.com, and nothing here should be read as evidence that any crawler did or did not obey anything. We measured what 20 companies wrote down on one day, and in one of those files a character that a reader cannot see makes the writing and the meaning disagree. That gap is the whole subject, and it is invisible from inside the file.