BlogFindings
Robots.txt special characters: 22 rules carried a raw non-ASCII path, and our matcher bound none of them encoded
Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 23 October 2026 and parsed every answer with the shipped parser in core/src/robots.ts. 1,063 files parsed, carrying 75,739 Allow and Disallow rules. 51 files held a byte outside ASCII somewhere, 46 confined it to a comment or a Sitemap line, and 5 put one inside a rule path. Those 5 wrote 22 such rules, and the shipped matcher bound none of the 22 against the percent-encoded form of the path they name.
The measurement was taken on 23 October 2026. One GET went to https://<host>/robots.txt for each of the 1,419 hostnames in the two committed seed files at worker/seeds/corpus-seeds-platform.json and worker/seeds/corpus-seeds-industry.json, sent as LantadBot/1.0 (+https://lantad.co/bot) from one network location, redirects followed, a twenty second timeout, no JavaScript and no browser. 1,091 hostnames answered HTTP 200. 28 of those served an HTML document at the robots.txt address and were excluded, leaving 1,063 files parsed with parseRobotsTxt from core/src/robots.ts, carrying 75,739 Allow and Disallow rules between them. Two of the 1,063 arrived gzip-compressed under a text/plain content type although the request advertised no compression, and were decompressed before parsing. The reasoning this site applies to a figure it cannot fully support is set out on the methodology page.
In short
- Robots.txt special characters are rare and concentrated: of the 1,063 robots.txt files Lantad parsed from its committed corpus on 23 October 2026, 51 carried a byte outside ASCII anywhere in the file, 46 of those kept it inside a comment or a Sitemap line, and 5 put one inside an Allow or Disallow path.
- Those 5 files wrote 22 rules with a raw non-ASCII path, and all 22 sit in a group whose tokens include the wildcard, so each one addresses every crawler that reads the file, while none of the 5 names an AI crawler in the group carrying the rule.
- Google's robots.txt documentation, carrying Last updated 2026-08-31 UTC, states that a rule path may be raw UTF-8 or percent-encoded UTF-8 and prints Disallow: /foo/bar/ツ and Disallow: /foo/bar/%E3%83%84 as the same rule, while RFC 9309 section 2.2.2 says octets outside ASCII MUST be percent-encoded before comparison.
- Lantad's shipped matcher compares a pattern and a path as literal text, so on 23 October 2026 it bound none of the 22 raw rules against the percent-encoded form of their own path, and a Disallow written one way left the other way allowed in both directions.
- 73 of the 1,063 files wrote a percent escape in a rule path across 638 rules, and 43 of them wrote the same octet in both hex cases, 59 doubled octets in all, though RFC 3986 section 2.1 says two URIs differing only in that case are equivalent; 33 of those 43 are Shopify storefronts.
| Measure | Files | Of | What the count means |
|---|---|---|---|
| Hostnames asked | 1,419 | 1,419 | Both committed corpus seed files |
| Answered HTTP 200 | 1,091 | 1,419 | Of which 28 served HTML, excluded |
| Parsed as robots.txt | 1,063 | 1,419 | The denominator below |
| A non-ASCII byte anywhere | 51 | 1,063 | Comments and Sitemap lines included |
| Only in a comment or Sitemap line | 46 | 51 | No rule path affected |
| Inside an Allow or Disallow path | 5 | 1,063 | The files this post is about |
| A percent escape in a rule path | 73 | 1,063 | The encoded form, written deliberately |
| Both forms in one file | 1 | 1,063 | Raw and escaped rule paths together |
What do robots.txt special characters do to a rule?
Three published documents bear on this and they do not say the same thing. RFC 9309, the Robots Exclusion Protocol, states in section 2.2.2 that octets in the URI and robots.txt paths outside the range of the ASCII coded character set, and those in the reserved range defined by RFC 3986, MUST be percent-encoded as defined by RFC 3986 prior to comparison. Read literally that makes the encoded form the only comparable form: a crawler holds a percent-encoded request path, and a rule written in raw UTF-8 has not been brought into the same alphabet before the comparison happens. The same section adds that a percent-encoded ASCII octet in the URI must be decoded before comparison unless it is reserved or outside the unreserved range, and section 2.3 of RFC 9309 says the file itself MUST be UTF-8 encoded.
Google documents something more forgiving. Its robots.txt documentation, carrying Last updated 2026-08-31 UTC, says the path in the rules can be either raw UTF-8 characters or percent-encoded UTF-8 strings, and that its crawlers automatically canonicalize raw UTF-8 rule paths into percent-encoded forms. It then prints the two lines Disallow: /foo/bar/ツ and Disallow: /foo/bar/%E3%83%84 and states they are treated identically. That is a vendor promising to do the normalising work the specification assigns to the file author, and it is the behaviour most site owners have in mind when they type an accent into a Disallow line. The same page says the file must be a UTF-8 encoded plain text file, that characters outside the UTF-8 range may be ignored, that the Unicode byte order mark is an invalid line and ignored, and that internationalised domain names are equivalent to their punycode versions.
RFC 3986 settles a narrower question that turns out to matter here more than the first. Section 2.1 states that the uppercase hexadecimal digits A through F are equivalent to the lowercase digits a through f, and that if two URIs differ only in the case of hexadecimal digits used in percent-encoded octets, they are equivalent. It adds that for consistency, producers and normalizers should use uppercase digits. So %2B and %2b are the same octet by the definition of RFC 3986, and any matcher that compares them as text will disagree with that definition. The two crawler vendors whose tokens this scanner evaluates say nothing at all. OpenAI's crawler documentation, read on 23 October 2026, documents OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User and does not mention percent-encoding, UTF-8 or non-ASCII paths anywhere. Anthropic's crawler help article, read the same day, documents ClaudeBot, Claude-User and Claude-SearchBot, says the bots honour industry standard directives in robots.txt, offers the non-standard Crawl-delay extension, and is silent on path encoding too. A site owner wanting to know which of the two readings GPTBot or ClaudeBot implements cannot find out from the vendor.
| Document | Date it carries | What it says about the raw form |
|---|---|---|
| RFC 9309 section 2.2.2 | September 2022 | MUST be percent-encoded prior to comparison |
| RFC 3986 section 2.1 | January 2005 | Escapes differing only in hex case are equivalent |
| Google robots.txt documentation | Last updated 2026-08-31 UTC | Raw or encoded, canonicalized to encoded |
| OpenAI crawler documentation | Read 23 October 2026 | No mention of path encoding |
| Anthropic crawler help article | Read 23 October 2026 | No mention of path encoding |
How many robots.txt files write a non-ASCII rule path?
Across the 1,063 parsed files, 51 carry at least one byte outside ASCII somewhere in the file. That number falls apart usefully when you ask where the byte sits. On 46 of the 51 it never reaches a rule: it is inside a comment, a human sentence after a hash mark that the parser discards before it reads a field, or inside a Sitemap line, which is a URL rather than a path pattern and is not matched against anything. Only 5 files put a byte outside ASCII inside an Allow or Disallow path, which is where the comparison in section 2.2.2 actually happens. Those 5 files wrote 22 such rules between them.
The 5 are tiki.vn with 11, skyscanner.net with 8, and selfridges.com, zalando.de and ethiopianairlines.com with one each. What they are blocking is mostly a search or filter surface in the site's own language. tiki.vn disallows patterns holding Vietnamese phrases for gambling and prize games, along with the full-width brackets and the star and circle symbols that spam listings use in their titles. skyscanner.net disallows result paths written in Spanish and French, including two spellings of the same route with and without a space, such as one path containing "Los vuelos" with a space and another with hyphens. zalando.de disallows a path ending in the two Chinese characters for WeChat. Every one of the 22 rules sits in a group whose token list includes the wildcard, so each rule addresses every crawler that reads the file, AI crawlers included. Not one of the 5 files names an AI crawler token in the group that carries the rule: skyscanner.net names AhrefsBot and ethiopianairlines.com names SemrushBot and SiteAuditBot, and both of those groups also carry the wildcard. That pattern, a rule aimed at everything because nothing was aimed at anyone, is the same one this blog measured when it found that 559 of the 581 pages GPTBot lost were closed by a rule that never named it.
Two of the 22 name a character that cannot have come from a working URL. ethiopianairlines.com disallows a path for its Italian accessibility page in which the accented character has been replaced by U+FFFD, the Unicode replacement character, baked into the file as valid UTF-8. No request path can contain that codepoint as a literal, so the rule matches nothing under any reading. selfridges.com disallows a pattern ending in U+00C2, the capital A with circumflex that appears when a non-breaking space is decoded as Latin-1 and re-encoded as UTF-8. Both are the residue of an editing pipeline rather than a decision, and neither is visible to the site owner in any report that only counts whether a robots.txt file exists. Checking for that kind of damage is the reason this site ships a robots.txt tester rather than a presence check, and it is the same class of defect behind the finding that 232 of 1,059 files carried a defect.
What Lantad's own matcher does with each form
The scanner's rule matcher is robotsPatternMatches in core/src/robots.ts. It walks the pattern and the path together one character at a time, with a single greedy backtrack for the asterisk, and it does no normalising of any kind before it starts. There is no percent-decoding step, no percent-encoding step and no case folding of escape digits. The function was written that way for a reason recorded in its own header, which is that an earlier version compiled the pattern to a regular expression and could be made to backtrack exponentially by a crafted robots.txt, so the rewrite traded a general matcher for a bounded one. Nothing in that trade concerns encoding, and the encoding behaviour is a consequence nobody chose.
Running the shipped function over the exact example Google publishes gives four results. A raw pattern matches a raw path. An encoded pattern matches an encoded path. A raw pattern against the encoded path does not match, and an encoded pattern against the raw path does not match. Taking the 22 raw rules found in the corpus and building from each one the percent-encoded form of the path it names, the shipped matcher bound 0 of the 22. End to end through evaluateRobots the consequence is a verdict that flips on spelling alone: a file reading User-agent: * and Disallow: /café/ returns disallowed for the raw path and allowed for the percent-encoded one, and a file writing the same rule percent-encoded returns the two the other way round. Under Google's documented canonicalisation both of those are one rule with one answer. Under this scanner they are two rules with two answers, and whichever form the crawler sends decides which one applies.
This is a defect in our matcher rather than a finding about the web, and it is worth being exact about its size. It can only change a verdict on a path that carries a character outside ASCII, which in this corpus is 5 files out of 1,063, and the 22 rules involved are search and filter surfaces rather than content a site would want read. The post changes no code, so the gap stands as measured; what it rules out is the comfortable reading, which is that a literal matcher is merely stricter than the specification. It is not stricter. It is differently wrong in both directions, reporting a path open that a conforming crawler would treat as closed, and closed where a conforming crawler would treat it as open. Readers checking their own file against what a specific crawler sees can use what GPTBot sees, and the per-crawler token list this scanner evaluates is published on the AI crawlers page and in the bot page for our own fetcher.
Pattern written raw
- pattern /foo/bar/ツ against path /foo/bar/ツ: match
- pattern /foo/bar/ツ against path /foo/bar/%E3%83%84: no match
- Disallow: /café/ leaves /caf%C3%A9/x allowed
- 0 of the 22 corpus rules bound their encoded path
Pattern written encoded
- pattern /foo/bar/%E3%83%84 against /foo/bar/%E3%83%84: match
- pattern /foo/bar/%E3%83%84 against /foo/bar/ツ: no match
- Disallow: /caf%C3%A9/ leaves /café/x allowed
- Google documents both lines as one rule
Why 43 files write the same escape twice
The encoded form is the commoner one and it is written on purpose. 73 of the 1,063 files carry a percent escape inside a rule path, across 638 rules. Sorting those 73 by the case of the hex digits they use produces a result that reads as a verdict on the specification rather than a stylistic spread. 13 files use only escapes whose digits are numerals, so case does not arise. 17 use uppercase letters only, which is what RFC 3986 recommends. None uses lowercase only. The remaining 43 carry both cases, and in every one of those 43 the two cases are not two different octets: the same octet is written twice, once each way. 59 doubled octets appear across the 43 files.
33 of those 43 files are Shopify storefronts, drawn from the Shopify stratum of the platform seed file, with 9 more in the hand-built ecommerce stratum and 1 in SaaS. The generated Shopify robots.txt writes Disallow: /collections/*%2B* and Disallow: /collections/*%2b* as two adjacent rules, and then repeats the pair under a locale prefix. By the definition in RFC 3986 section 2.1 the second line is redundant, because the two differ only in the case of a hex digit and are therefore the same URI. The platform writes it anyway, which is the most informative thing in this measurement: a vendor serving a generated file to a large number of stores does not rely on crawlers normalising escape case, and pays two lines per pattern to avoid finding out. Run the shipped matcher over it and the caution is justified against this scanner at least, because a pattern holding %2B does not match a path holding %2b, nor the reverse. The same platform's handling of what a crawler can read is covered in the Shopify stack guide and in the finding that 38 stores served 1,112 products as open JSON.
Escape case is a smaller instance of a pattern this blog has measured twice before in robots.txt. Rule paths are case-sensitive while user-agent tokens are not, which produced the finding that 84 mis-cased crawler tokens all bound while 9 disallowed pages answered anyway. Length rather than order decides which of two matching rules wins, which produced the finding that 343 of 716 files wrote no Allow line that changed a verdict. In each case the file looks like a set of instructions and behaves like a string comparison with rules attached. Six files in this corpus also open with a UTF-8 byte order mark, being sciencespo.fr, ama.com.au, 1177.se, albertahealthservices.ca, bhf.org.uk and lung.org. That is the one encoding question where the shipped parser and Google's documentation agree: the parser strips the mark before reading, and Google's page calls it an invalid line and ignores it.
| Escape case in the file | Files | Rules or octets | What it implies |
|---|---|---|---|
| Numerals only, so no case | 13 | Escapes such as %20 and %3F | |
| Uppercase letters only | 17 | The form RFC 3986 recommends | |
| Lowercase letters only | 0 | Nobody writes it this way alone | |
| Both cases present | 43 | 59 doubled octets | The same octet written twice |
| All files with an escape | 73 | 638 rules | Of 1,063 parsed files |
| Of the 43, Shopify storefronts | 33 | A generated file, identical across stores |
What this does not measure, and what to check on your own file
No crawler was observed doing anything here. There are no server logs behind this measurement, so nothing in it says how GPTBot, ClaudeBot or PerplexityBot resolves a raw rule path, and the vendor pages read on 23 October 2026 do not say either. Every statement about matching is a statement about what two specifications require, what Google publishes, and what this scanner's own code does. One URL was requested per hostname at the apex, so nothing covers subdomains, which matters because this blog has already found that 73 of 277 hostnames serve a different robots.txt on a subdomain. The 1,419 hostnames are two committed sampling frames built for another purpose, not a sample of the web, so none of these proportions generalises past them, and a corpus weighted toward English-language sites will understate how often a non-ASCII rule path occurs on the web as a whole. The German-language search demand that reaches this site, measured in the post on robots.txt testen, is a reminder that the population searching for robots.txt help is less English than this corpus is.
Availability also moves between rounds, which is why the denominator differs from other posts drawn from the same corpus. Of the 1,419 hostnames, 1,091 answered 200 and 328 did not: 158 answered 403, 54 answered 503, 49 answered 404, 6 answered 202, 4 answered 429, 2 answered 401 and single instances of 307, 406, 418 and 498 arrived, while 51 requests returned no status at all, being 20 DNS failures, 15 that redirected the request to a plain HTTP address this client refused to follow, 9 timeouts, 6 connection resets and 1 oversized header. A file that cannot be fetched has no rules to encode, which is the first thing to establish before any of this applies and the subject of the earlier finding that fourteen hostnames yielded eight files.
What is worth doing on a site whose URLs carry characters outside ASCII is narrow and cheap. Write the rule path percent-encoded, in uppercase hex, because that form is comparable under RFC 9309, is accepted by Google's documented canonicalisation, and is the only form that does not depend on a crawler normalising for you. Check the file for a replacement character or a stray capital A with circumflex, either of which means the path was damaged in an editor rather than written. Do not write the same octet in two cases unless you have a specific crawler in mind that needs it, and if you do, record why. And check the result against the crawler you care about rather than against the file, because the gap this post found in our own matcher is exactly the gap a file cannot show you: both the raw and the encoded line look correct, and only one of them binds. The wider corpus work this measurement comes from is collected in the crawlability study, and how the scoring treats a blocked path is set out under AI visibility and in the GEO glossary entry.
-
Percent-encoded, uppercase hexComparable Required form under RFC 9309 and accepted by Google; 17 files use only this -
Raw UTF-8 in the rule pathDepends on the crawler Google canonicalises it; our matcher does not, and bound 0 of 22 -
Same octet in both hex casesRedundant by the spec 43 files write it anyway, 33 of them Shopify storefronts -
Replacement or mis-decoded characterMatches nothing 2 rules in the corpus name a codepoint no URL can carry
Lantad
Published .
Robots.txt special characters are the part of the file nobody checks, because most files never need them. A rule path is usually ASCII, and the question of what a crawler does with an accent, a Vietnamese tone mark or a Chinese character in a Disallow line never comes up. It comes up on sites whose URLs carry those characters, which is most sites outside the English-speaking web, and the answer the specification gives is not the answer the largest crawler vendor gives. This post counts how often the situation arises across a committed corpus, reads what three primary documents say about it, and then reports what this scanner's own AI crawler matcher does with each form, which turns out to be neither of the published behaviours.
Common questions
Do robots.txt special characters have to be percent-encoded?
Under the specification, yes. RFC 9309 section 2.2.2 says octets in the URI and robots.txt paths outside the range of the ASCII coded character set, and those in the reserved range defined by RFC 3986, MUST be percent-encoded prior to comparison. Google is more forgiving in practice: its robots.txt documentation, carrying Last updated 2026-08-31 UTC, says a rule path can be either raw UTF-8 or percent-encoded UTF-8 and that its crawlers canonicalize the raw form into the encoded one, printing Disallow: /foo/bar/ツ and Disallow: /foo/bar/%E3%83%84 as the same rule. Writing the encoded form satisfies both readings, which is why it is the safer choice.
Does GPTBot or ClaudeBot handle a raw non-ASCII rule path?
Neither vendor says. OpenAI's crawler documentation, read on 23 October 2026, documents OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User without mentioning percent-encoding, UTF-8 or non-ASCII paths. Anthropic's crawler help article, read the same day, documents ClaudeBot, Claude-User and Claude-SearchBot, says the bots honour industry standard directives in robots.txt, and is also silent on path encoding. Lantad has no server logs for those crawlers, so this post makes no claim about what either one does.
Are %2B and %2b the same thing in robots.txt?
By the definition in RFC 3986 section 2.1 they are: the uppercase hexadecimal digits A through F are equivalent to the lowercase a through f, and two URIs differing only in that case are equivalent. Not every matcher implements that. Lantad's shipped matcher compares escape text literally, so on 23 October 2026 a pattern holding %2B did not match a path holding %2b or the reverse. 43 of the 73 corpus files writing an escape in a rule path write the same octet in both cases, 33 of them Shopify storefronts, which suggests the platform does not rely on crawlers normalising it either.
How common is a non-ASCII path in robots.txt?
Rare in this corpus and worth stating with its limits. Of the 1,063 robots.txt files Lantad parsed on 23 October 2026 from its two committed corpus seed files, 51 carried a byte outside ASCII anywhere, 46 of those kept it inside a comment or a Sitemap line, and 5 put one inside a rule path, writing 22 such rules between them. Those seed files are weighted toward English-language sites, so the figure understates how often the situation arises on the web, and it says nothing about sites outside the 1,419 hostnames asked.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
Is robots.txt case sensitive? 84 mis-cased crawler tokens all bound, and 9 disallowed pages answered anyway
Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 14 October 2026. 1,052 returned a text file, carrying 88,364 directive lines and 74,626 allow and disallow rules. 84 user-agent lines on 58 files spelled a documented AI crawler token in the wrong case and this scanner's evaluator bound every one of them. 15,864 rule paths on 418 files carried a capital letter, and on the half of the file where case does decide the answer, nine pages across five hosts answered a crawler at the lowercase address their own Disallow line failed to cover.
robots.txt Allow: 343 of 716 files wrote no line that changed a verdict, and 24 rules won on a length tie
Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 22 October 2026 and parsed the 1,061 files that resolved into at least one user-agent group. Those files carried 18,427 Allow rules against 60,514 Disallow rules. Taking each Allow line out and re-running the shipped evaluator against the line's own path, 15,591 of the 18,427 changed the answer and 2,714 did not. At file level the split is much closer: of the 716 files whose Allow lines could be probed, 343 held not one line that changed anything.
robots.txt validator: 232 of 1,059 files carried a defect, and 33 sites answered with a web page
Lantad asked 1,419 hostnames for /robots.txt on 19 September 2026 and checked every answer against RFC 9309 and Google's published parsing rules. 1,104 returned HTTP 200. 33 of those returned HTML rather than robots.txt text, 12 returned nothing at all, and of the 1,059 that held real directives, 232 carried at least one defect a validator can name.
Wildcard robots.txt: the trailing star changed nothing in 168 comparisons
Run against Lantad's shipped robots.txt matcher on 27 August 2026, appending a trailing asterisk to six rule patterns changed the verdict on none of 168 path comparisons. Appending a trailing dollar sign to four patterns changed 36 of 112.
