# Is robots.txt case sensitive? 84 mis-cased crawler tokens all bound, and 9 disallowed pages answered anyway

> Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 14 October 2026. 1,052 returned a text file, carrying 88,364 directive lines and 74,626 allow and disallow rules. 84 user-agent lines on 58 files spelled a documented AI crawler token in the wrong case and this scanner's evaluator bound every one of them. 15,864 rule paths on 418 files carried a capital letter, and on the half of the file where case does decide the answer, nine pages across five hosts answered a crawler at the lowercase address their own Disallow line failed to cover.

- Canonical page: https://lantad.co/blog/is-robots-txt-case-sensitive-84-tokens-bound-9-pages-open
- This file: https://lantad.co/blog/is-robots-txt-case-sensitive-84-tokens-bound-9-pages-open.md
- Last substantive update: 2026-10-14

## Key facts

- **Published:** 2026-10-14
- **Category:** Findings
- **Author:** Lantad
- **Length:** 4640 words
- **Takeaway 1:** The question is robots.txt case sensitive has two answers inside one file: RFC 9309 section 2.2.1 states that crawlers MUST use case-insensitive matching to find the group that matches the product token, while section 2.2.2 states that the path matching SHOULD be case sensitive, which is a recommendation and not a requirement.
- **Takeaway 2:** Case in a user-agent line is harmless in practice. Across the 1,052 robots.txt files Lantad parsed on 14 October 2026, 84 lines on 58 files named one of 15 documented AI crawler tokens in a case the vendor does not publish, 21 distinct spellings between them, and this scanner's shipped evaluator in core/src/robots.ts bound all 21 to the right crawler.
- **Takeaway 3:** Case in a path is where the answer changes. 15,864 of the 74,626 allow and disallow rules in those 1,052 files carried an uppercase ASCII letter, spread over 418 files, and 2,734 of the 15,864 carried a lowercase letter immediately followed by an uppercase one.
- **Takeaway 4:** 6,746 literal uppercase Disallow paths on 198 hosts left their own lowercase address allowed to GPTBot when re-evaluated by the shipped matcher on 14 October 2026, though 5,206 of the 6,746 belong to a single host, boe.es, and the remaining 197 hosts account for 1,540 between them.
- **Takeaway 5:** Of a sample of 626 of those lowercase addresses, five per host at most and only ones the site's own file leaves open, 81 answered HTTP 200 and nine pages on ama.com.au, bca.co.id, discover.com, github.com and nationwidechildrens.org survived every check for a soft 404 or a homepage catch-all on 14 October 2026.

## Summary

Almost every rule in a robots.txt file is a string comparison, and a string comparison has to decide what to do about letter case. The standard decides it twice, in opposite directions, within two sections of each other. A crawler reading the user-agent line is required to ignore case entirely. The same crawler reading the Disallow line beneath it is told to respect case. Both halves sit in the same file, usually written in the same sitting by the same person, and nothing in the file's syntax hints that the rules change halfway down.

So the two halves were measured separately. On 14 October 2026 this scanner requested the robots.txt of all 1,419 hostnames in the two committed corpus seed files in this repository, as LantadBot with no JavaScript executed, and parsed the 1,052 that returned a text file with the same [robots.txt parser](https://lantad.co/tools/robots-txt-tester) that ships in the product. The question was not whether people make case mistakes, because they clearly do: it was whether a case mistake costs anything, and the answer turns out to depend entirely on which line it lands in. One half of the file forgives everything. The other half quietly stops protecting the thing the rule names, and nine pages in this corpus show what that looks like when a real [AI crawler](https://lantad.co/glossary/ai-crawler) arrives.

## Is robots.txt case sensitive? The specification answers twice, and differently

[RFC 9309, the Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html), was published in September 2022 by Koster, Illyes, Zeller and Sassman, and it is the first time the format had a standards track document rather than a convention. Section 2.2.1 covers the user-agent line and is unambiguous about case: crawlers MUST use case-insensitive matching to find the group that matches the product token. The requirement level is MUST, the strongest the document has, so a crawler that skipped a group because the token was typed in lowercase would be violating the specification rather than exercising a choice.

Section 2.2.2 covers the allow and disallow lines and reverses the position: the matching SHOULD be case sensitive. SHOULD is weaker than MUST, and the difference is not a formality. A crawler may match paths case-insensitively and still be conformant, which means the behaviour of the path half is not actually fixed by the standard at all. The same section anchors every pattern at the front, stating that the matching MUST start with the first octet of the path, and describes the comparison in terms of octets rather than characters: the match evaluates positively if and only if the end of the path from the rule is reached before a difference in octets is encountered. Octets are bytes, and two bytes differing only in case are still two different bytes.

Google's robots.txt specification page, which carries Last updated 2026-08-31 UTC, removes the ambiguity for the one crawler whose behaviour most site owners are actually writing for. It states that field names are case-insensitive, giving User-agent and user-agent as equivalent examples, and that both the user-agent field name and its value are case-insensitive. On the rules it draws the line explicitly: the field name disallow is case-insensitive, but its value is case-sensitive, and the path value must start with a slash to designate the root and the value is case-sensitive. Its worked example for the pattern /fish adds the same note again. The [robots.txt documentation Google publishes](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) is therefore firmer than the RFC it implements, turning a SHOULD into a plain statement of behaviour, and it is the only one of the two that a person editing a file is likely to have read.

The matcher that ships inside this product follows the same split, and it is worth saying that it is a choice the code makes rather than a law of nature. In core/src/robots.ts the parser lowercases every user-agent token as it builds a group, and the group selector lowercases the bot token before comparing, so token case cannot affect a verdict. The pattern matcher compares the rule and the path one character at a time without normalising either. An earlier post on this blog established those semantics against Google's own worked examples, finding that [a trailing wildcard changed no verdict in 168 comparisons](https://lantad.co/blog/wildcard-robots-txt-trailing-star-changed-nothing) while the end anchor changed 36 of 112. That post tested the matcher. This one asks what the asymmetry does to real files.

## What was measured on 14 October 2026, and what was not

The population is the 1,419 hostnames in the two committed corpus seed files in this repository, worker/seeds/corpus-seeds-platform.json with 392 hostnames in ten platform strata and worker/seeds/corpus-seeds-industry.json with 1,027 hostnames in eight industry strata. Each was asked for its robots.txt once, as LantadBot/1.0, from one network location, with redirects followed, a twenty second timeout and no browser involved. 1,093 answered HTTP 200. 1,052 of those delivered something that parsed as a plain text robots file and form the population for everything below. 33 delivered an HTML page served at the robots.txt address, which is a defect this blog has [counted across the same corpus before](https://lantad.co/blog/robots-txt-validator-232-of-1059-carried-a-defect), and 8 returned an empty body. 180 answered 403 and 50 answered 404, and the gap between a file that permits a crawler and a server that serves one is wide enough that [79 of 115 hostnames previously measured refused a crawler their own file allowed](https://lantad.co/blog/is-my-site-blocking-ai-crawlers-79-of-115).

Every one of the 1,052 files was then read twice. The first pass counted directive lines by their written spelling, which is how the field name and token figures below were produced. The second pass handed the body to parseRobotsTxt and walked the resulting groups and rules, which is how the path figures were produced. Both passes used the shipped code rather than a throwaway regular expression, so the counts describe what the product would do with these files and not what a one-off script thought they meant. The method is the same one behind the [crawlability study](https://lantad.co/research/crawlability-study) and is written up in full on the [methodology page](https://lantad.co/methodology).

Three limits matter before any number below is read. No AI crawler was observed doing anything, and no request was sent pretending to be one: every verdict here is this scanner's own evaluation using the GPTBot product token, and a crawler that matches paths case-insensitively, which section 2.2.2 permits, would make the entire second half of this post moot for itself. Lantad does not know which crawlers do that, because no vendor documentation read for this post states a position either way. Second, only one URL per hostname was requested for the robots file, and these are 1,419 front doors drawn from two committed sampling frames rather than a sample of the web, so no proportion here generalises beyond them. Third, and most important for the live checks in the last two sections, the cased paths were never fetched. A rule saying Disallow followed by a capitalised path disallows that exact address, and this scanner honours it, so the only addresses requested were the lowercase ones the site's own file leaves open. That is a narrower test than it could be, and it is narrow on purpose.

## 58 files named an AI crawler in the wrong case, and all 84 lines still bound

The product token is the name a crawler answers to in robots.txt, and vendors publish an exact spelling for it. Fifteen of those spellings are held as a named constant in core/src/bots.ts, each with the documentation URL it came from, covering OpenAI, Anthropic, Perplexity, Google, Apple, Meta, ByteDance, Amazon and Common Crawl. Across the 1,052 files, 1,629 user-agent lines on 203 hosts named one of those fifteen tokens. 84 of those lines, spread over 58 files, spelled the token in a case the vendor does not publish, in 21 distinct variants.

One variant accounts for more than half of them on its own. meta-externalagent, written entirely in lowercase against the documented Meta-ExternalAgent, appears on 48 lines, which is a strong hint that a single widely copied template or platform default is responsible rather than 48 independent typing errors. The pattern of names Meta publishes has caused trouble in this corpus before: only [6 of 1,065 files named the citation token rather than the training one](https://lantad.co/blog/meta-ai-crawler-6-of-1065-named-the-citation-token). The rest of the variants are the ordinary shapes of hand editing. AmazonBot appears four times against the documented Amazonbot, which is the kind of mistake a reader makes because the capital B looks more correct than the real spelling does. Anthropic-AI and Anthropic-ai appear six times between them against the lowercase anthropic-ai that Anthropic actually documents. Claudebot and claudebot appear five times, CCbot and ccbot four, and bytespider, OAI-Searchbot, google-extended and perplexitybot twice each. Nine variants appear once: gptbot, chatgpt-user, CHATGPT-User, ChatGPT-user, applebot-extended, claude-user, perplexity-user, amazonbot and ByteSpider.

None of it costs anything. Each of the 21 variants was fed to the shipped parser as a one group file with a single Disallow, and the shipped evaluator was then asked for a verdict using the vendor's documented spelling as the bot token. All 21 bound: the group was selected, the rule applied, and the verdict came back disallowed. That is the RFC 9309 requirement working exactly as written, and it is the reason this half of the file is a non-finding rather than a problem. A site owner who typed gptbot has still blocked GPTBot.

Which is worth saying plainly, because the failure modes that do cost something in a user-agent line are different ones, and they are easy to confuse with a case mistake. Naming a token that no longer exists costs the whole rule, and [renamed crawler tokens leave stale rules behind](https://lantad.co/blog/renamed-crawler-tokens-leave-stale-robots-rules) in files nobody has revisited. Naming a token no crawler answers to costs the whole rule too. Across this corpus, [1,004 robots.txt files named 2,209 distinct user-agent tokens](https://lantad.co/blog/crawler-user-agent-2209-tokens-in-1004-files), and the great majority of those tokens belong to nothing that is still crawling. Case is the one thing in that line the standard guarantees will be forgiven.

## 15,864 rule paths carry a capital letter, and that is the half that decides

The 1,052 files hold 74,626 allow and disallow rules between them. 15,864 of those rules, on 418 files, carry at least one uppercase ASCII letter in the path. That is 21 percent of every rule in the corpus, and it is not in itself a mistake: a path that genuinely contains a capital needs a capital in the rule, and plenty of them do. The interesting subset is narrower. 2,734 of the 15,864 carry a lowercase letter immediately followed by an uppercase one, which is the camelCase shape that application frameworks generate and that a human transcribing a URL by hand is most likely to get wrong in one direction or the other.

To separate a working rule from a decorative one, each uppercase disallow path was put back through the shipped evaluator twice: once as written, and once lowercased. Patterns containing a wildcard or an end anchor were excluded, so only literal path prefixes were considered, because a wildcard pattern covers a family of addresses rather than one. Where the file blocked the path as written and allowed the same path in lowercase, the rule was recorded as covering only one of the two spellings of its own target. 6,746 rules on 198 hosts came out that way.

That number needs a caveat stated before it is quoted anywhere, because it is dominated by a single site. boe.es, the Spanish official state gazette, contributes 5,206 of the 6,746 on its own, with a very long file of individually listed document paths. The remaining 197 hosts account for 1,540 between them, a median of three each. The honest form of the finding is therefore about hosts rather than rules: 198 of the 418 files that use a capital in a path have at least one rule that protects one spelling of an address and leaves the other open, and the rule count is a property of how verbose one gazette's robots.txt is.

A good deal of this is harmless for a reason that has nothing to do with case. A large share of the uppercase paths are the README and licence files that content management systems list by default, where the lowercase address simply does not exist and the rule is guarding a file that was never there. That is the same class of inherited boilerplate that made [79 of 86 Drupal robots.txt files name no AI crawler at all](https://lantad.co/blog/drupal-ai-crawlers-79-of-86-named-none), and it is noise rather than exposure. Distinguishing the noise from a real gap cannot be done by reading the file, which is why the next section stops reading and starts fetching. It is also worth noting what a case gap is not: it is not the much larger problem of a wildcard that sweeps up more than its author intended, where [559 of the 581 pages GPTBot lost were closed by a rule that never named it](https://lantad.co/blog/blocked-by-robots-txt-559-of-581-by-a-wildcard).

## Nine pages a Disallow line named were open to a crawler at the other spelling

A rule that fails to cover an address only matters if something is served at that address. So a sample of the 6,746 was requested. The sample rule was fixed in advance and is not random: for each of the 198 hosts, the first five qualifying paths in file line order, which gives 626 requests spread evenly enough that boe.es contributes five rather than 5,206. Every request was the lowercase address, sent as LantadBot with redirects followed and no JavaScript executed, and every one of them was an address the site's own robots.txt leaves open to GPTBot by the evaluator's reading. The cased originals were not requested at all, because those are disallowed and this scanner obeys the file it is reporting on. The [bot page](https://lantad.co/bot) states that commitment and this run kept it.

620 of the 626 completed. 444 answered HTTP 404, which is the expected result and the measure of how much of the uppercase path population is inherited boilerplate guarding nothing. 71 answered 403, 81 answered HTTP 200, and the remaining 24 were spread across 503, 429, 502, 410 and a handful of single statuses, with six requests failing or timing out.

81 responses of 200 is not 81 open pages, and the gap between those two numbers is the part of this measurement worth copying. 72 of the 81 delivered HTML, on 41 hosts. 37 of those carried at least 200 words of extracted text, on 22 hosts. 18 of the 37, on 8 hosts, still had the lowercase path in the final URL after redirects, rather than having been bounced to the cased original by a server that normalises case on its own. Then every one of the 18 was fetched again and compared against its own host's home page, because a site that serves its home page for any unrecognised address produces a 200 with plenty of words and no page behind it. One was dropped on exactly that test, netcare.co.za, where the lowercase address returned the home page byte for byte. Eight more were dropped as soft 404s: five addresses on ethiopianairlines.com return a page titled Page Not Found under a 200, two on ama.com.au return a document titled 404, and revolve.com had moved to a 503.

Nine pages on five hosts survived. ama.com.au disallows /Sitemap.aspx and serves a 212 word site map at /sitemap.aspx. bca.co.id, an Indonesian bank, accounts for three: a 2,447 word terms and conditions page for its KeyBCA token under a rule written for /Syarat-dan-Ketentuan/keybca-softtoken/, a 498 word promotion page under a rule written for /id/Campaign/promo-imlek/, and a 311 word event page under /id/Forms/Ext/event-bca. nationwidechildrens.org accounts for two, both over 2,100 words, behind rules whose paths contain a literal space as well as capitals. github.com accounts for two, a repository and a user profile, where the rules name /ekansa/Open-Context-Data and /Explodingstuff/ and the lowercase forms serve the same content without redirecting. discover.com is the one case where only the query string differs: the rule names a credit card article with the parameter channel set to LINK, and the same article answers with channel set to lowercase link.

Nine is a floor on a 626 request sample, not an estimate of anything. A wider sample of the same 198 hosts would find more, and the 404 rate suggests it would not find proportionally many more. What the nine establish is that the failure is real rather than theoretical, and that it is concentrated in exactly the places the specification predicts: camelCase application paths, localised paths with capitalised words, and query parameter values. None of these nine pages is a secret, and nothing here says any of them was crawled, indexed or cited by anything. They are pages whose owners wrote a rule to keep a crawler out and whose rule does not cover the address the crawler can reach, which is a different and smaller claim than a leak.

## What to check on your own file, in the order that pays

The practical conclusion is short, and it is almost the opposite of the advice case sensitivity usually attracts. Stop worrying about the case of your crawler names. If a file says gptbot, Claudebot or meta-externalagent, that rule works, the standard requires it to work, and this scanner's evaluator confirmed all 21 variants found in the corpus bind correctly. Rewriting them to match the vendor's capitalisation is tidiness, not a fix, and the time is better spent checking the token still exists at all.

Then take the path half seriously, because that is where a rule silently stops doing its job. Three checks, in descending order of how often they found something here. First, any path with a capital that came from an application framework rather than from a person: the camelCase shapes, the aspx file names, the localised segments with capitalised words. For each, decide whether your server treats the two spellings as the same resource, and if it does, write both rules or lowercase the whole thing. Second, query parameter values, which are part of the path for matching purposes and which discover.com shows can differ in case alone. Third, paths containing a space or other character that needs encoding, which is where the two nationwidechildrens.org cases sit and where the encoding question compounds the case question.

A caution on the fix, because the obvious one has a cost. Adding a lowercased duplicate of every capitalised rule doubles the length of that part of the file, and length is not free: Google enforces a robots.txt file size limit of 500 kibibytes and ignores everything after it, which its specification page states directly. A file already near that limit cannot absorb a mechanical doubling. Lowercasing the paths your server actually serves in lowercase is the cheaper change, and verifying it is a matter of requesting both spellings and seeing whether the same page comes back, which is the whole of the test run above.

Beyond case, the checks that move more are elsewhere in the same file, and this blog has measured most of them against this corpus. Whether the file is served at all: [92 of 1,056 sites refused a crawler the file itself](https://lantad.co/blog/can-gptbot-read-your-robots-txt-92-of-1056). Whether it is syntactically sound: [232 of 1,059 carried a defect](https://lantad.co/blog/robots-txt-validator-232-of-1059-carried-a-defect), separately from anything in this post. Whether the apex and the subdomains agree, since [73 of 277 gave an AI crawler a different answer from the apex](https://lantad.co/blog/robots-txt-subdomains-73-of-277-differ). Whether the fields it uses are ones anything reads, since [7 of 1,057 files still wrote the noindex rule Google retired in 2019](https://lantad.co/blog/robots-txt-noindex-7-of-1057-files-wrote-it). Whether the rules name anything current, which the [AI crawler reference](https://lantad.co/tools/ai-crawlers) tracks as tokens appear and are renamed, and which matters more for a [generative engine optimization](https://lantad.co/glossary/geo) outcome than any of the case questions here. And whether the file is obeyed, which no amount of reading it can establish: [six of ten crawlers never asked for it](https://lantad.co/blog/do-ai-crawlers-respect-robots-txt-six-of-ten-never-asked) in an earlier observation, which is the limit sitting under every figure in this post.

## Questions and answers

**Is robots.txt case sensitive?**

Partly, and the split runs through the middle of every file. The user-agent line is not case sensitive: RFC 9309 section 2.2.1 states that crawlers MUST use case-insensitive matching to find the group that matches the product token, so writing gptbot instead of GPTBot costs nothing. The Allow and Disallow path value is case sensitive: section 2.2.2 says the matching SHOULD be case sensitive, and Google's robots.txt specification page states flatly that the path value is case-sensitive. Field names such as Disallow are also case-insensitive.

**If I typed my crawler name in the wrong case, is the rule broken?**

No. Across 1,052 robots.txt files read on 14 October 2026, 84 user-agent lines on 58 files named a documented AI crawler token in a case the vendor does not publish, in 21 distinct variants, and this scanner's shipped evaluator bound all 21 to the right crawler. The rules that do break are the ones naming a token that was renamed or never existed, which is a different problem from capitalisation.

**What happens if my Disallow path has the wrong case?**

The rule covers the spelling you wrote and no other, so an address differing only in case is left allowed. On 14 October 2026, 6,746 literal uppercase Disallow rules across 198 hosts left their own lowercase address allowed to GPTBot when re-evaluated, and a 626 request sample of those lowercase addresses found nine pages on five hosts that answered with real content. Most of the rest answered 404, because the lowercase address did not exist.

**Should I just write every rule twice, once in each case?**

Rarely worth it as a blanket policy. Google enforces a robots.txt file size limit of 500 kibibytes and ignores anything after that point, so doubling every rule in an already long file risks losing rules at the end. The cheaper fix is to lowercase the paths your server serves in lowercase, and to test by requesting both spellings and checking whether the same page comes back.

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot