Allow AI crawlers in robots.txt: 89 of 145 files ruled only on the whole site

How do you allow AI crawlers in robots.txt for some pages and not others?

Three mechanics decide this, and two of them surprise people. The first is grouping. RFC 9309, the Robots Exclusion Protocol, defines a group as one or more user-agent lines followed by one or more rules, terminated by the next user-agent line or the end of the file. A crawler picks the group whose product token matches it and falls back to the group headed by an asterisk only if none does. The named group replaces the wildcard group rather than adding to it, so the moment you write a group for GPTBot that crawler stops reading the rules you wrote for everybody else.

The second is what happens when nothing matches. Section 2.2.2 is explicit: if no match is found amongst the rules in a group for a matching user-agent, or there are no rules in the group, the URI is allowed. That single sentence is why an Allow-only group is usually decoration. A group that lists ten Allow paths and no Disallow permits those ten paths and every other path on the site, because the paths you did not mention were never forbidden. An allow list in robots.txt only functions as a list when a Disallow sits beside it for the rest to fall through to.

The third is precedence, and it is the one worth checking before you edit a line. RFC 9309 says the most specific match found MUST be used, that the most specific match is the match that has the most octets, and that where an allow rule and a disallow rule are equivalent the allow rule SHOULD be used. Google's own robots.txt documentation, last updated 2026-08-31 UTC, states it as the most specific rule based on the length of the rule path, and adds that in case of conflicting rules, including those with wildcards, Google uses the least restrictive rule. Its worked examples show both halves operating: against /page.htm the pair Allow: /page and Disallow: /*.htm resolves to the disallow because the rule path is longer, while against /page.php5 the pair Allow: /page and Disallow: /*.ph resolves to the allow because the two are the same length and the least restrictive wins.

Pattern syntax is the smallest part of it. Only two characters do anything in a path: the asterisk for zero or more of any character and the dollar sign for the end of the match. Both are in section 2.2.3 as characters crawlers MUST support. Neither is as powerful as it looks, because a robots.txt pattern is already anchored at the start of the path, which is why we found that a trailing asterisk changed no verdict in 168 comparisons while a trailing dollar changed 36 of 112. If you want to see how your own file resolves for each token rather than reading it as a mood, the robots.txt tester evaluates one path against every token separately.

What did 1,027 hostnames return for /robots.txt?

The frame is a committed file in this repository rather than a list assembled for this post, which matters because every figure below can be recomputed against the same 1,027 hostnames. It holds eight sectors of roughly equal size: government at 129 hosts, education at 130, healthcare at 123, news at 128, software at 130, ecommerce at 130, travel at 130 and finance at 127. It is not a random sample of the web and it is deliberately weighted toward large organisations, so a proportion measured here describes this frame and not the internet. The same frame produced the count behind GPTBot being disallowed at the root by 82 of 718 files the day before.

Nine hundred and eighty five hostnames answered and 42 did not, 23 of those timing out at fifteen seconds and 19 failing to connect at all. Of the responses, 743 were HTTP 200 and three were HTTP 202. One hundred and sixty eight returned 403, which is a bot defence rather than a robots policy, and 35 returned 503, which under the protocol is the one status that means the opposite of a 404: an unavailable robots.txt imposes no restrictions while a failing one is commonly treated as disallow-all, a distinction we set out in full when we established that a 404 on robots.txt allows every crawler and a 503 blocks them all. Twenty five returned 404 and four returned 429.

Twenty seven of the 2xx bodies were an HTML or XML document rather than a robots file, which is a 200 that means nothing, and those are excluded from every count that follows. That leaves 706 files. Between them they carry 6,296 user-agent groups and 67,873 allow and disallow rules. The median file holds 20 rules and one group; the largest holds 12,227 rules and another holds 642 separate groups. Thirty two of the 706 parsed with a group and no rules at all, which under section 2.2.2 allows everything. Seven hundred and one of the 706 carry a group headed by an asterisk, so the fallback almost always exists.

The five files without a wildcard group are worth a moment, because a file with no wildcard group and no group for your token imposes nothing on you: not a permission the site granted, just a gap. What this scan counts and what it refuses to infer is set out on the methodology page.

How many robots.txt files write a path rule for an AI crawler?

One hundred and forty five of the 706 files name at least one of the 15 tokens the scanner evaluates. The naming is lopsided in the way it always is: GPTBot appears in 113 of the 145 and ClaudeBot in 102, while OAI-SearchBot appears in 60 and Claude-SearchBot and Claude-User in 32 each. That gap is the same one behind our finding that 16 of 140 files named OAI-SearchBot while 25 named GPTBot, and it means a large share of these files have addressed the training crawler and said nothing to the crawlers that fetch a page to answer a question about it, which is the traffic a site actually wants if it wants any. The vendor pages behind each token are collected in the AI crawler reference.

Inside those 145 files, 89 write nothing but whole-site rules in the AI-named groups. Sixty eight of the 89 only disallow, which is a closed door and a coherent position. Thirteen only allow, which is usually an exemption from a restrictive wildcard group and occasionally nothing at all. Seven do both, allowing some tokens the whole site and refusing others. One names a token and writes no rule under it, which permits everything. Fifty six files write at least one rule naming a path, and those 56 hold 213 AI-named groups between them.

So on the narrow question this post asks, the answer is 56 of 145, or 8 percent of the 706 files that parsed. Everyone else who has thought about AI crawlers at all has thought about them as a single switch. That is a defensible way to answer a question about training. It is not a way to answer the question a site asks when it wants its documentation quoted and its faceted search left alone, and it is the shape that makes AI visibility hard to reason about from a robots file: a yes at the root tells you the crawler may fetch, and nothing about what it will find worth quoting.

The 15 tokens are not equally likely to be governed by a path rule, and the ordering inverts the naming. GPTBot is named in 113 files and sits in a group holding a path rule in 37 of them, a third. OAI-SearchBot is named in only 60 files and sits in a group holding a path rule in 36, which is 60 percent and the highest share of the fifteen. Bytespider, CCBot and Amazonbot are lowest at 26 percent each. Read together: a site that names the training crawler usually names it to close a door, and a site that names the search crawler is more often choosing where it may go. That second group is the one our guides on getting cited by ChatGPT and the Claude crawlers are written for.

Did writing the AI rules change any verdict?

A rule that names a crawler feels like an act. Whether it is one depends entirely on what the rest of the file already said, so we tested it directly. For each of the 145 files we removed every group headed by an AI crawler token, leaving the file otherwise intact, and evaluated each token the file had named against twelve paths in both versions: the root, an about page, a blog post, a documentation page, a search URL with a query string, a product URL with a tracking parameter, a PDF, a JPEG, a markdown file, /llms.txt, a markdown content negotiation URL, and a JSON API path. Every evaluation used the matcher that ships in this scanner, and the comparison is between two readings of the same file rather than between our reading and a crawler's.

On 116 of the 145 files at least one verdict moved, so the AI groups are doing work. On 29 they did not: every token the file names received exactly the same answer on all twelve paths with the AI groups deleted. Among those 29 are canonical.com, whose group headed by 14 tokens carries 16 Allow and 13 Disallow lines, cloudflare.com, gov.uk, netlify.com, redis.io, retool.com, supabase.com and sonarsource.com. In most of those cases the reason is simple and not a mistake: the named group repeats what the wildcard group already said, so a reader of the file learns the site's intent while a crawler learns nothing new.

The direction of the 116 that did move is the finding a site owner should sit with. Ninety eight of them only ever restricted an AI crawler relative to their own wildcard group, 13 only relaxed it, and 5 did both depending on the path. So among the files in this frame that bothered to write AI-specific rules and made a difference by doing it, the overwhelming use of the mechanism is to give an AI crawler less access than an ordinary crawler gets, not more. Whether that is right is a business decision. It is worth knowing that it is the default behaviour of the population, because the advice circulating about generative engine optimization assumes the opposite.

One file shows what a real allow list looks like. figma.com serves 384,745 bytes of robots.txt. Its groups for GPTBot, ClaudeBot and CCBot hold one rule each, Disallow: /. Its group headed by ChatGPT-User, OAI-SearchBot, PerplexityBot, Claude-SearchBot and Claude-User holds 8,848 rules: one Disallow and 8,847 Allow lines, 8,842 of them anchored with a trailing dollar sign, enumerating the exact pages those five crawlers may fetch. That is the shape the specification actually rewards, and it is 8,848 lines long because there is no shorter way to say it. Compare visitdubai.com, whose group headed by 26 tokens holds two rules, Allow: /*.md$ and Allow: /llms.txt, and permits every other path on the site as well, because nothing under that group forbids anything. Both files look like allow lists. Only one is.

The 2,159 rule values that are not valid path-patterns

RFC 9309 gives the grammar for a rule value in its ABNF, and it is one line: path-pattern is a forward slash followed by any number of characters. A value with no leading slash does not fit that production. Across the 706 files, 2,159 allow and disallow values fail it, spread over 156 files. Two thousand one hundred and seventeen of them begin with an asterisk instead, which is somebody reaching for a substring match that robots.txt does not offer, because a pattern is already anchored at the front of the path. The remaining 42 begin with something else entirely: a query string, a bare filename, and on one ecommerce site the literal text of a User-agent line written as a Disallow value.

The specification does not say what a crawler must do with a value that fails the grammar, and that silence is the whole problem. Two readings are available and both are defensible. A lenient parser treats the value as a pattern anyway, which is what the matcher in this repository does. A strict parser discards the line as malformed, leaving the group with one fewer rule. On a file whose group is otherwise permissive the two readings give opposite verdicts, and the site owner has no way to know which behaviour any given crawler implements, because no AI crawler vendor publishes one. OpenAI's crawler documentation names four crawlers and their user agent strings and recommends allowing OAI-SearchBot, and says nothing at all about pattern syntax or which specification it follows.

Two hundred and sixty six of those 2,159 values sit inside a group headed by an AI crawler token, across 11 files, and confluent.io is the clearest case of intent defeated by syntax. Its wildcard group carries Disallow: *.pdf, and it then writes three separate groups, for GPTBot, Google-Extended and ClaudeBot, each holding the single rule Allow: *.pdf. The intent could not be plainer: PDFs are closed to crawlers in general and open to those three. All four values lack the leading slash, so under a strict reading all four vanish. It happens not to matter here, because a named group with no rules allows everything and the three crawlers end up permitted either way, but the file only works by accident.

Two further habits show up in the same 67,873 rules and neither is in the grammar. Character classes appear on one apparel site, which writes eight consecutive instances of a hexadecimal class inside a product path, 16 rules of that shape in total, and one of those lines runs straight into a Sitemap directive with no newline between them, producing a single 144 character pattern that matches nothing. Regular expression syntax appears on one marketplace, which writes the same rule twice as a digit class followed by a plus and a dollar sign in the middle of the path. And of all 67,873 values, not one uses the percent-encoded forms the specification recommends in section 2.2.3, %2A and %24, for matching a literal asterisk or dollar in a URL. Three rules in the corpus are a bare asterisk and nothing else, which is the same construction we found on a social network when a robots.txt named the wildcard and not the AI crawler.

What to check in your own robots.txt

Four checks fall out of this scan, in the order they change an outcome. None needs a scanner and all four can be run against your own file in a few minutes.

First, check that every named group contains a Disallow. Seventeen of the 145 files here have AI-named groups with no Disallow anywhere in them, and a group of nothing but Allow lines permits every path on the site rather than only the ones listed. If the intent was a list, the list needs a Disallow for everything else to fall through to. If the intent was an exemption from a restrictive wildcard group, an Allow line is the right tool and the group is doing exactly what you meant.

Second, delete your AI groups on a copy and see whether anything changes. That is the test that separated 116 files from 29 here, and it takes one evaluation per path you care about. If the answers are identical, your file documents an intention to a human reader and communicates nothing to a crawler. That is not automatically wrong. It is worth knowing before you rely on it.

Third, check the leading slash on every value. Two thousand one hundred and fifty nine values in this corpus do not have one, and the specification does not say what happens next. A value like an asterisk followed by a file extension is the single most common form of this, and rewriting it with a leading slash and an internal asterisk costs one character and removes the ambiguity entirely. The same goes for character classes and regular expression fragments, which match themselves literally and nothing else.

Fourth, separate the fetch question from the content question. A permission in robots.txt says a crawler may make the request. What it receives when it does is a different measurement, and one that 32 of 44 documentation sites answered differently at /llms-full.txt than at /llms.txt shows can diverge sharply from what a browser sees. What GPTBot sees renders the crawler side of a single URL, llms.txt covers the file most often proposed as an alternative, and our guides for Perplexity and Google AI Overviews cover the platform-specific half.

What this post did not measure should be stated plainly. We requested one path from each of 1,027 hostnames on one day, 13 September 2026, from one network location, as the single user agent documented on our bot page. We observed no AI crawler requesting any of these sites, we read no access log, and no figure here reports what any crawler did. Every verdict is our matcher's reading of a file, and a reading is not a behaviour: the 29 files whose AI groups changed no verdict were tested on twelve paths, and a thirteenth path could separate them. The frame is a fixed list weighted toward large organisations in eight sectors, so no proportion here describes the web. Earlier scans against the same frame and its sibling are indexed on the research page.