# robots.txt Allow: 343 of 716 files wrote no line that changed a verdict, and 24 rules won on a length tie

> Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 22 October 2026 and parsed the 1,061 files that resolved into at least one user-agent group. Those files carried 18,427 Allow rules against 60,514 Disallow rules. Taking each Allow line out and re-running the shipped evaluator against the line's own path, 15,591 of the 18,427 changed the answer and 2,714 did not. At file level the split is much closer: of the 716 files whose Allow lines could be probed, 343 held not one line that changed anything.

- Canonical page: https://lantad.co/blog/robots-txt-allow-343-of-716-changed-no-verdict
- This file: https://lantad.co/blog/robots-txt-allow-343-of-716-changed-no-verdict.md
- Last substantive update: 2026-10-22

## Key facts

- **Published:** 2026-10-22
- **Category:** Findings
- **Author:** Lantad
- **Length:** 4576 words
- **Takeaway 1:** A robots.txt Allow line can only do work where a Disallow already reaches, and in this corpus about half the files that write one have nothing for it to reach: of the 716 files Lantad probed on 22 October 2026, 343 held no Allow line that changed any verdict.
- **Takeaway 2:** 111 of the 1,061 files parsed on 22 October 2026 carry at least one Allow rule and no Disallow rule anywhere in the file. Not one of those 111 has an Allow line that changes an answer, because under RFC 9309 a path with no matching rule is already allowed.
- **Takeaway 3:** 24 Allow rules across 14 hostnames won only because RFC 9309 says an Allow and a Disallow of equal length resolve in the Allow's favour. nytimes.com writes Allow: /athletic/search/$ beside Disallow: /athletic/search/*, which are the same number of octets.
- **Takeaway 4:** Two files in the corpus state both positions for one crawler and let the tie rule settle it. motherjones.com names ChatGPT-User twice, once with Allow: / and once with Disallow: /, and because groups sharing a token merge, the shipped evaluator returns allowed.
- **Takeaway 5:** Whether an Allow line does anything is mostly decided by the platform that generated the file. All 28 readable Framer files carry an Allow and 1 has one that changes a verdict; 30 of 33 Shopify files carry one and 29 of those 30 do.

## Summary

The Allow line is the only directive in robots.txt that grants rather than withholds, and it is the one most often written in the belief that permission has to be granted at all. It does not: under the protocol, a path nobody has disallowed is already open. That makes Allow a narrow instrument. It can carve an exception out of a Disallow that reaches too far, and outside the reach of a Disallow it is inert, however deliberate the comment above it.

On 22 October 2026 this scanner requested the robots.txt of all 1,419 hostnames in the two committed corpus seed files in this repository, as LantadBot/1.0 with redirects followed and no JavaScript executed, and parsed the 1,061 files that resolved into at least one user-agent group. Rather than count Allow lines, this run tested each one: take the line out, ask the shipped evaluator about the path that line covers, and see whether the answer moves. What follows is how many moved, which files write Allow lines with nothing to override, where the protocol's tie-break rule turns out to be load bearing, and what any of it means for the fifteen [AI crawler](https://lantad.co/glossary/ai-crawler) tokens this scanner tracks. Any single file can be put through the same parser on the [robots.txt tester](https://lantad.co/tools/robots-txt-tester).

## Does a robots.txt Allow line override a Disallow?

Sometimes, and the condition is precise. [RFC 9309, the Robots Exclusion Protocol, published in September 2022 by Koster, Illyes, Zeller and Sassman](https://www.rfc-editor.org/rfc/rfc9309.html), says that the most specific match found must be used, and defines the most specific match as the match that has the most octets. It then adds one sentence that settles everything else: if an allow rule and a disallow rule are equivalent, then the allow rule should be used. The same document states that where no rule in the matching group matches the path, the path is allowed.

Put those together and the shape of the directive follows. Length decides, not order in the file, and not which keyword was written. An Allow of twelve octets beats a Disallow of eight that covers the same path, and loses to a Disallow of twenty. Where the two are the same length the Allow wins, which is the only place the keyword itself decides anything. And because an unmatched path is allowed by default, an Allow line that sits outside every Disallow's reach cannot change an answer, since the answer was already yes.

[Google's robots.txt documentation, last updated 2026-08-31 UTC](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt), describes the same procedure in its own words: crawlers use the most specific rule based on the length of the rule path, and in case of conflicting rules, including those with wildcards, Google uses the least restrictive rule. It also states that when no path is specified the rule is ignored, and that Google enforces a robots.txt file size limit of 500 kibibytes. The largest file in this corpus, boe.es at 495,022 bytes, sits under seventeen kilobytes beneath that ceiling.

This is the procedure the product implements. evaluateRobots in core/src/robots.ts selects the group, merges every group sharing the winning token, scores each matching pattern by length and breaks a tie in favour of the Allow. The pattern matcher behind it handles the asterisk and the end-of-path dollar sign, and what those two characters actually change was measured separately when [appending a trailing asterisk changed nothing across 168 comparisons](https://lantad.co/blog/wildcard-robots-txt-trailing-star-changed-nothing). Group selection is the other half, and it is the half that decides which rules a given crawler ever sees, as the count showing [559 of 581 lost paths were closed by a rule that never named GPTBot](https://lantad.co/blog/blocked-by-robots-txt-559-of-581-by-a-wildcard) established.

## What was measured on 22 October 2026, and what was not

The population is the 1,419 hostnames in the two committed corpus seed files in this repository: worker/seeds/corpus-seeds-platform.json with 392 hostnames across ten platform strata, and worker/seeds/corpus-seeds-industry.json with 1,027 across eight industry strata. Each hostname was asked once for its robots.txt over HTTPS, as the user agent described on the [bot page](https://lantad.co/bot), from one network location, with redirects followed and a fifteen second timeout. No other URL was requested and no page was rendered.

The funnel sets every denominator below. 1,144 hostnames answered HTTP 200, the only 2xx status seen in this round. 39 of those served an HTML document at the robots.txt address rather than a robots file, a defect this blog has [counted across the same corpus before](https://lantad.co/blog/robots-txt-validator-232-of-1059-carried-a-defect), and they were excluded. 44 more parsed into no user-agent group at all, usually a file holding only a Sitemap line or a comment, and a file with no group carries no rules, so those went too. That leaves 1,061 parsed files. Of the remainder 114 answered 403, 48 answered 404, 4 answered 429, 2 each answered 426 and 401, and single instances of 418 and 498 arrived. 103 requests returned no status: 33 failed at TLS connection, 21 at an HTTP/2 stream, 20 on receive, 14 at certificate verification, 14 timed out and 1 exceeded the redirect limit.

An earlier read of the same corpus the same day, published as the [Bingbot count of 62 of 1,019 files](https://lantad.co/blog/bingbot-robots-txt-62-of-1019-named-it), reached 1,019 parsed files rather than 1,061. The two differ because availability moves between rounds and because that run saw 42 files answer 503 where this one saw none. Both numbers are what the network returned at the hour they were asked, and the counts in this post are not comparable with that post's beyond the direction of travel.

The classification needs one definition. For each Allow rule, a concrete probe path is built from the rule's own pattern by dropping a trailing dollar sign and replacing each asterisk with a single literal character, then checked against the shipped matcher to confirm the rule still matches it. The group's rules are evaluated for that path twice, once as written and once with that single Allow rule removed. If the path is allowed with the line and disallowed without it, the line changed the answer. If it is allowed both ways, it did not. 122 rules produced no probe path, because their patterns name no path at all: aucklandcouncil.govt.nz writes Allow: .css and Allow: .js, and hel.fi writes 18 of its 19 Allow lines in forms beginning with an asterisk, such as Allow: */core/*.css$. Those are reported separately rather than counted either way.

Three limits apply throughout. No crawler was observed doing anything, because there are no server logs here, so nothing below says what any crawler did with these files; it says what the protocol and the shipped parser make of them. One URL per hostname was requested, at the apex, so nothing here covers subdomains, which [differ on 73 of 277 hosts that have them](https://lantad.co/blog/robots-txt-subdomains-73-of-277-differ). And these 1,419 front doors are two sampling frames built for another purpose, so no proportion here generalises past them. The approach is the one set out on the [methodology page](https://lantad.co/methodology).

## 343 of 716 files held no Allow line that changed an answer

724 of the 1,061 files carry at least one Allow rule, and 875 carry at least one Disallow. Across them sit 18,427 Allow rules and 60,514 Disallow rules, inside 96,737 directive lines in total. Removing each Allow rule and re-asking the evaluator about its own path, 15,591 of the 18,427 changed the answer, 2,714 did not, and 122 could not be probed. Not one Allow rule in the corpus was beaten at its own path by a longer Disallow, which is a property of the probe rather than a finding about the web: the probe path is built from the Allow's own pattern, so a Disallow would have to be longer and still match it.

That ratio, 15,591 of 18,427, is the wrong number to carry away, because one host writes nearly half the rules. figma.com's file is 100,930 bytes and holds 8,800 Allow rules against 38 Disallow rules, a long enumeration of permitted paths under a short set of broad exclusions, and 8,749 of its Allow lines change an answer. Take that one file out and 9,627 Allow rules remain, of which 6,842 change an answer. Take out the next two heaviest, decathlon.fr with 1,013 and elastic.co with 890, and 7,724 remain with 5,112 changing an answer. The rule-level figure is a statement about three files.

The file is the unit that matches how robots.txt is actually written, and at file level the picture is much flatter. The median file carrying an Allow writes two of them, and 332 of the 724 write exactly one. 716 files had at least one probeable Allow. 373 of those carry at least one line that changes an answer. The other 343 carry none at all: every Allow line in them is a statement of permission for something nothing had withheld. That is 48 percent of the 716.

The clearest version is the 111 files that carry an Allow rule and no Disallow rule anywhere. Not one of the 111 has an Allow that changes anything, and it could not be otherwise, since there is nothing in the file to override. groundcover.com is the instructive case because the intent is explicit and commented. Its file heads one group with GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot under the comment reading allow AI search crawlers, follows it with Allow: /, then heads a second group with ChatGPT-User and Perplexity-User under a comment about user-triggered agents and follows that with Allow: / as well. The file contains no Disallow line. Every one of those crawlers was already allowed every path, so the three Allow lines leave the file saying exactly what an empty file would say. The same holds for the 28 readable Framer sites, whose entire file is typically User-agent: *, Allow: / and a Sitemap line.

arxiv.org is the largest single instance of the pattern in a file that does restrict things. It writes 59 Allow rules, among them Allow: /abs, Allow: /pdf and Allow: /list, alongside 249 Disallow rules including Allow's apparent counterparts Disallow: /user and Disallow: /form, and a Crawl-delay of 15 seconds. None of the 59 changes a verdict, because none of the Disallow patterns reaches any path the Allow lines name. The file is written as though robots.txt were an allowlist that opens what it names. It is not, and the counting of the directive that does get honoured, [Crawl-delay on 157 of 1,014 files](https://lantad.co/blog/crawl-delay-robots-txt-157-of-1014-set-one), is a separate question.

## 24 rules won only because the Allow and the Disallow were the same length

The one sentence in RFC 9309 that gives Allow a preference over Disallow is easy to read as a tidy-up for an edge case nobody hits. In this corpus 24 Allow rules across 14 hostnames are decided by it, and in each the site has written two patterns of identical length that disagree about the same path.

The most common shape pairs a dollar-anchored Allow against an asterisk Disallow of the same width. nytimes.com writes Allow: /athletic/search/$ beside Disallow: /athletic/search/*, so the search index page is reachable and everything beneath it is not, expressed in two patterns of exactly the same number of octets. theatlantic.com does the same for one 2001 magazine article, viarail.ca for two offer pages in English and French, figma.com for a single product path, and kayak.com for its things-to-do index. kayak.com and nordstrom.com each write their pair twice, once in the wildcard group and once for Googlebot, which is where four of the 24 come from. In every one of these the site plainly intends the Allow to win, and it does, but it wins on the preference sentence rather than on specificity, and a parser that implemented only the longest-match rule would resolve all of them the other way.

wayfair.com and agoda.com reach the tie through wildcards instead. wayfair.com writes Allow: /*/sb0/, Allow: /*/sb1/ and Allow: /*/sb2/ against Disallow: /*/sb*/, where each Allow is the same length as the Disallow it is carving into. agoda.com produces four of them from pairs such as Allow: /place/*/ against Disallow: /place/*$, in both its wildcard group and its Googlebot group. tripadvisor.com contributes three, and its pairs are literally identical strings: under User-agent: applebot the file carries both Allow: /media/photo-f and Disallow: /media/photo-f, and the same for photo-l and photo-o. That is the plainest form of the case the RFC sentence exists for, and the deference to Applebot it implies sits oddly beside the finding that [64 files defer to Googlebot for Applebot](https://lantad.co/blog/applebot-robots-txt-64-files-defer-to-googlebot).

Two files go further and contradict themselves about an entire site for one named crawler. Because groups sharing a token are merged rather than read in order, a token named twice in one file ends up with both sets of rules. motherjones.com heads a group with ChatGPT-User and writes Allow: / under it, then twenty-two lines later heads another group with ChatGPT-User and writes Disallow: /. Allow: / and Disallow: / are the same length, so the merged rule set resolves to allowed. illinois.gov does the same for Baiduspider, with Allow: / near the top of the file and Disallow: / forty-eight lines below. These are the only two instances in 1,061 files, and in both the site's own file holds both positions while a single sentence of a specification settles which one a crawler acts on. Whether the later line was meant to replace the earlier one is not knowable from the file; what is knowable is that merging makes order irrelevant. OpenAI's user-triggered token is the one most likely to be handled inconsistently, as the count of [38 files refusing GPTBot while allowing ChatGPT-User](https://lantad.co/blog/block-chatgpt-38-files-refuse-gptbot-allow-chatgpt-user) showed from the other direction.

## What an Allow line does for GPTBot and the other AI crawler tokens

145 of the 1,061 files write at least one Allow rule inside a group that names one of the fifteen AI crawler tokens in this scanner's registry, which is published on the [AI crawler reference](https://lantad.co/tools/ai-crawlers). 84 of those 145 have at least one such rule that changes an answer. The other 61 do not. That is 58 percent against the corpus-wide 373 of 716, so the habit arrives at the crawlers the product exists to measure in much the same proportion it arrives everywhere else.

The per-token counts separate two quite different habits. 204 of the 1,061 files name GPTBot, 107 of those write an Allow inside its group, and on 61 of the 107 one of those lines changes an answer. ClaudeBot is almost identical at 197, 102 and 62, and Google-Extended at 177, 106 and 66. Those three are the tokens most often named at all, and when a site names them it tends to be writing a real rule set, which is why the Allow lines inside those groups do work about three times in five. The training tokens behave as a group: anthropic-ai reaches 60 of 81, CCBot 59 of 76, Applebot-Extended 56 of 76, Meta-ExternalAgent 55 of 73, Bytespider 54 of 69 and Amazonbot 53 of 70.

The search and user-triggered tokens are where the Allow line stops working. OAI-SearchBot is named on 99 files, 70 write an Allow in its group, and only 27 of those 70 have one that changes an answer. PerplexityBot is 132, 75 and 26. ChatGPT-User is 126, 76 and 32. Perplexity-User is 73, 36 and 13, the smallest count of the fifteen. The reason is visible in the files: a group naming a search or user-triggered crawler very often contains nothing but Allow: /, written as a welcome rather than as an exception. The site is making a statement, and the statement has no mechanical effect, because nothing was stopping that crawler in the first place. The honest reading is that the gesture costs nothing and achieves nothing, which is a different objection from the one usually made about these lines.

None of this says anything about what the crawlers do. A rule is a request, and whether it is honoured is a separate measurement that this run did not make and that the series has approached from the other end, finding that [six of ten engines never asked for robots.txt at all](https://lantad.co/blog/do-ai-crawlers-respect-robots-txt-six-of-ten-never-asked). What the file does settle is what a compliant crawler is told, and the finding here is narrow and worth keeping narrow: for the tokens a site is most likely to want to welcome, the welcome is usually written in a line that changes nothing. Where a site genuinely wants to readmit an AI crawler to a path it has closed to others, the Allow line is the correct instrument and it works; it is simply not needed for a path that was never closed, and the files naming [ClaudeBot on 159 of 1,014 hosts](https://lantad.co/blog/claudebot-robots-txt-159-of-1014-named-it) show the same mixture of considered rules and decoration.

## The platform that generated the file decides whether Allow does anything

Split the 1,061 files by stratum and the pattern stops being about editorial care and starts being about defaults. In the Framer stratum all 28 readable files carry an Allow rule and exactly one, americanelitepainters.com, carries one that changes an answer. In the Webflow stratum 10 of 18 carry an Allow and none changes an answer. In the static documentation stratum 11 of 19 carry one and none changes an answer. Among the single-page application startups 33 of 34 carry an Allow and one changes an answer. These are the strata where the file is generated, the generated file contains no Disallow, and the Allow line is a default rather than a decision. The crawler behaviour of two of them has been measured directly, in the reads showing [35 of 36 Framer sites served markdown](https://lantad.co/blog/framer-ai-crawlers-35-of-36-served-markdown) and [20 of 40 Webflow sites held no rule](https://lantad.co/blog/webflow-ai-crawlers-20-of-40-held-no-rule).

Shopify is the exact inverse and for the same reason. 30 of the 33 readable Shopify files carry an Allow rule and 29 of those 30 have one that changes an answer, because the generated Shopify file is a real exception list: 35 Allow rules against 50 Disallow rules, in which Disallow: /checkout is carved into by Allow: /products/checkout and the same pattern repeats for account, orders and the localised variants. 18 of those 35 Allow rules change an answer and 17 do not, identically across store after store, which is what a template produces. That platform's handling of crawler access was read separately when [1,112 Shopify products arrived as open JSON](https://lantad.co/blog/shopify-ai-crawlers-1112-products-as-open-json).

The Wix and Squarespace stratum sits between the two and is the largest single block of generated files here: 53 of 54 carry an Allow and 35 of those have one that changes an answer. The generated Wix file heads the wildcard group with Allow: / followed by Disallow: *?lightbox= and then writes narrower groups for the Google Ads crawlers and a refusal for PetalBot, so whether the Allow does work depends on which of several template variants the site received. The inverse case, where naming a crawler changes nothing because the name lands in a group that already governs everyone, was counted when [zero of 69 Wix sites named one](https://lantad.co/blog/wix-ai-crawlers-zero-of-69-named-one).

The hand-written strata behave differently again, and more sensibly. In news, 80 of 116 files carry an Allow and 50 of those have one that changes an answer. In ecommerce it is 67 of 81 with 42. In education 56 of 92 with 41, in WordPress small business 26 of 39 with 23. Finance is the outlier among them at 47 of 90 carrying an Allow and only 17 having one that works, which fits the pattern established when [2 of 97 finance robots.txt files disallowed GPTBot](https://lantad.co/blog/which-industries-block-ai-crawlers-41-of-69-news-files): a stratum that writes few restrictions has little for an Allow to carve into. None of this changes a site's [AI visibility](https://lantad.co/glossary/ai-visibility) by itself, and the measurable consequence of a file this size is elsewhere, in the delivered page rather than the rule, as the count of [25 of 1,093 home pages passing two megabytes](https://lantad.co/blog/page-size-limit-25-of-1093-passed-2mb) shows.

The practical reading is short. If the file has no Disallow, delete the Allow lines and nothing changes. If the file does restrict something, the Allow line is the only way to open an exception inside it, and it has to be longer than the Disallow it is carving into, or exactly as long. The 61 files in this corpus that write an Allow inside a group naming an AI crawler and hold no such line that changes anything are not making a mistake that costs them access; they are making a statement no parser acts on, which is a cheaper error, and the files that write [a rule naming the wildcard rather than the crawler](https://lantad.co/blog/robots-txt-names-the-wildcard-not-the-ai-crawler) are making the expensive one.

## Questions and answers

**Does Allow override Disallow in robots.txt?**

Only when the Allow pattern is at least as long as the Disallow pattern that matches the same path. RFC 9309 says the most specific match must be used and defines it as the match with the most octets, then adds that where an allow and a disallow rule are equivalent the allow rule should be used. Google's robots.txt documentation, last updated 2026-08-31 UTC, describes the same procedure and says it uses the least restrictive rule when rules conflict. Order in the file does not matter, and neither does which keyword was written, except at an exact length tie.

**Do I need an Allow line to let a crawler read my site?**

No. RFC 9309 states that where no rule in the matching group matches the path, the path is allowed, so a site with no Disallow is already fully open. Lantad measured the consequence on 22 October 2026: 111 of 1,061 robots.txt files carried at least one Allow rule and no Disallow rule at all, and not one of those Allow rules changed the answer the shipped evaluator returns for the path it names.

**Is it harmful to write an Allow line that does nothing?**

Nothing measured here says it costs access. An inert Allow line is read, parsed and ignored, and 343 of the 716 files Lantad probed on 22 October 2026 held only lines of that kind. The cost is that it reads as a control when it is not one, so a site owner who writes Allow: / for GPTBot and believes the question is settled has not checked the thing that would actually close the path, which is a Disallow elsewhere in the file or a block in front of the server.

**What happens if one robots.txt file names the same crawler twice with opposite rules?**

Under RFC 9309 the groups are merged rather than read in order, so the crawler receives both sets of rules and the length and tie rules settle the result. Lantad found two instances in 1,061 files on 22 October 2026: motherjones.com names ChatGPT-User once with Allow: / and once with Disallow: /, and illinois.gov does the same for Baiduspider. In both the two patterns are one octet long, so the Allow preference in the specification applies and the shipped evaluator returns allowed.

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot