BlogFindings
robots.txt validator: 232 of 1,059 files carried a defect, and 33 sites answered with a web page
Lantad asked 1,419 hostnames for /robots.txt on 19 September 2026 and checked every answer against RFC 9309 and Google's published parsing rules. 1,104 returned HTTP 200. 33 of those returned HTML rather than robots.txt text, 12 returned nothing at all, and of the 1,059 that held real directives, 232 carried at least one defect a validator can name.
So on 19 September 2026 Lantad requested /robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files, as LantadBot, following redirects and executing no JavaScript, and checked every response against RFC 9309 and against Google's published parsing rules. 1,104 answered HTTP 200 with a body. 33 of those returned an HTML page instead of robots.txt text. 12 returned a file with nothing in it. That leaves 1,059 responses holding real directives, and 232 of them, 21.9 percent, carry at least one defect that changes or could change what a parser reads. The rest of this post is what those defects are, in the order of how much damage they do, and you can run the same checks against a single host with the robots.txt tester rather than reading your own file by eye. None of this is a statement about any AI crawler actually visiting these sites, and the last section says what that leaves unmeasured.
In short
- A robots.txt validator run against 1,419 hostnames on 19 September 2026 found 1,104 answering HTTP 200, and 232 of the 1,059 responses that held real robots.txt directives carried at least one defect against RFC 9309 or Google's published parsing rules.
- 33 of the 1,104 files were HTML rather than robots.txt text, and only one of the 33 contained the string user-agent anywhere, so the other 32 hand a compliant crawler a file with no rules in it.
- Every one of the eight responses larger than the 500 kibibyte parsing limit was an HTML page rather than a robots.txt file, the largest being 1,344 kibibytes served by technion.ac.il.
- 224 of the 1,059 files used a field outside the four Google documents supporting, with crawl-delay in 158 of them, and fifteen distinct unsupported field names in total including six different invented ways to point at an llms.txt.
- 107 files named a user-agent value containing a space or punctuation that RFC 9309's product token grammar does not permit, across 1,394 such lines, and no crawler's product token can match one of them.
| What the check found | Files | Share of 1,059 |
|---|---|---|
| No defect this check can name | 827 | 78.1% |
| A field Google does not support | 224 | 21.2% |
| A user-agent value the RFC grammar forbids | 107 | 10.1% |
| A UTF-8 byte order mark | 6 | 0.6% |
| A line with no colon at all | 3 | 0.3% |
| A rule before the first user-agent line | 3 | 0.3% |
| Compressed without being asked | 1 | 0.1% |
What does a robots.txt validator actually check?
Two documents decide, and they do not agree about everything, so a validator has to say which one it is quoting. The first is RFC 9309, the Robots Exclusion Protocol, published by the IETF in September 2022. It defines the grammar and the fetch semantics. The second is Google's robots.txt specification, which carries a last updated date of 31 August 2026 and describes what one very large crawler actually implements. Where they differ, a site owner is governed by whichever crawler is at the door, which is why this check reports the two separately rather than merging them into one verdict.
RFC 9309 is explicit that a parser does not stop at a bad line. "Crawlers MUST try to parse each line of the robots.txt file. Crawlers MUST use the parseable rules," it says, which means a malformed line is skipped and the file is otherwise honoured. Google says the same thing from the implementation side: it "ignores invalid lines in robots.txt files, including the Unicode Byte Order Mark (BOM) at the beginning of the robots.txt file, and use only valid lines." Both behaviours are reasonable and both have the same consequence for a site owner, which is that a broken rule produces no error anywhere. The file still parses. The rule is simply not there.
That is the reason a defect count is worth collecting at all. A syntax error in a configuration file usually announces itself by breaking something. A syntax error in robots.txt announces itself by quietly widening access, and the only way to find it is to parse the file the way a crawler would and compare the result against what the author meant. This check parses, but it cannot read intent, so throughout this post a defect means a line whose handling is specified and surprising, never a guess about what somebody wanted.
Four of the checks rest on Google's documentation rather than on the RFC, and it is worth naming which. Google supports exactly four fields, "user-agent", "allow", "disallow" and "sitemap", and states in the same sentence that other fields "such as crawl-delay aren't supported". Google enforces a maximum file size of 500 kibibytes and says "Content which is after the maximum file size is ignored". The RFC approaches that same number from the other direction: it tells crawlers to impose a parsing limit and says "The parsing limit MUST be at least 500 kibibytes". A floor in one document and a ceiling in the other land on the same figure, which is convenient, and it means a file above that size is in undefined territory for everyone. Our own scanner's parsing rules are set out in the methodology.
| Check | What it rests on | What happens when it fails |
|---|---|---|
| Response holds robots.txt text, not HTML | Google: parses HTML for rules and ignores everything else | Rules are absent, not wrong |
| Body is under 500 kibibytes | Google enforces it as a maximum; RFC 9309 sets it as a minimum parsing limit | Everything past the limit is dropped |
| Every line carries a colon | RFC 9309: crawlers MUST use the parseable rules | That one rule does not exist |
| Rules follow a user-agent line | RFC 9309: crawlers SHOULD ignore rules not in any group | The rule applies to nobody |
| Field name is one of the four Google supports | Google supports user-agent, allow, disallow and sitemap only | Line is read by nothing named |
| user-agent value is a valid product token | RFC 9309: letters, underscores and hyphens only | No crawler can match the group |
| No byte order mark, served as text/plain | RFC 9309 requires UTF-8 and text/plain; Google ignores the BOM | Usually tolerated, not guaranteed |
33 sites answered /robots.txt with a web page
This is the largest single failure in the sample and it does not look like a failure from a browser. 33 of the 1,104 responses were HTML, and the page titles say what they are: the site's own home page in most cases, returned with a 200 rather than a 404. bls.gov returned the U.S. Bureau of Labor Statistics home page. cdc.gov returned the Centers for Disease Control and Prevention home page. interpol.int, gov.sg, utah.gov, wits.ac.za, accor.com, ryanair.com, renfe.com and schwab.com all did the same thing. nus.edu.sg was at least honest about it and returned a page titled "404 - URL invalid", but returned it with a 200 status, which for a crawler is the same answer. myntra.com and aeromexico.com returned "Site Maintenance". khanacademy.org returned "Client Challenge", which is a bot defence page.
Google's documentation describes exactly what it does with this: "if the content downloaded is HTML instead of robots.txt rules, Google will try to parse the content and extract rules, and ignore everything else." So the file is not rejected. It is parsed, nothing that looks like a directive is found, and the crawler proceeds with an empty ruleset. Of the 33 HTML responses, exactly one contained the string "user-agent" anywhere in its markup. The other 32 yield no rules at all. A site in that state has not blocked anything and has not allowed anything deliberately either: it has simply told every crawler that it has no opinion, while its operators may well believe a robots.txt is in place.
The size check found the same 33 sites from another direction. Eight responses exceeded the 500 kibibyte limit, and all eight were HTML. technion.ac.il returned 1,344 kibibytes, falabella.com 990, accor.com 728, nikkei.com 668, tewhatuora.govt.nz 660, citizenm.com 556, trenitalia.com 553 and 10thplanetperry.com 534. Not one real robots.txt file in the sample came close to the limit, so on this corpus the size rule turns out not to be a rule about large rule sets at all. It is a second detector for the same misconfiguration.
Twelve more hosts answered 200 with an empty file, among them gov.scot, epfl.ch, visa.com and spelman.edu. That case is different and it is not a defect: under RFC 9309 a file with no groups means no rules apply, which is a legitimate way to say that everything is permitted. It is worth separating from the HTML case for exactly that reason. The empty file is a decision that a parser reads correctly. The HTML page is an accident that a parser also reads correctly, and the two produce the same access outcome by completely different routes. Whether a given crawler even reaches the file is a separate question, and one this blog measured on 18 September when 92 of 1,056 sites refused GPTBot the file they served to a browser. The status code a site returns when it cannot serve the file matters too, because a 404 and a 503 are opposites under the RFC: 223 hosts here answered 4xx, where a crawler may access any resource, and 50 answered 5xx, where the RFC says a crawler "MUST assume complete disallow".
| Host | Title of the page returned | Size |
|---|---|---|
| bls.gov | U.S. Bureau of Labor Statistics | 77 KiB |
| cdc.gov | Centers for Disease Control and Prevention | CDC | 60 KiB |
| technion.ac.il | The Technion home page, in Hebrew | 1,344 KiB |
| falabella.com | falabella.com | Todo lo que necesitas en un solo lugar | 990 KiB |
| tewhatuora.govt.nz | Health New Zealand | Te Whatu Ora | 660 KiB |
| nikkei.com | The Nikkei home page, in Japanese | 668 KiB |
| nus.edu.sg | 404 - URL invalid | 3 KiB |
| khanacademy.org | Client Challenge | 3 KiB |
| myntra.com | Site Maintenance | (under 500 KiB) |
| lloydsbank.com | Lloyds Bank - Internet Banking - Error | (under 500 KiB) |
| ryanair.com | Ryanair | (under 500 KiB) |
| schwab.com | Charles Schwab | (under 500 KiB) |
One missing colon, and the rule does not exist
Three files in the sample contain a line with no colon in it, which under both documents is an invalid line and is dropped. On two of the three the dropped line is harmless: amsterdam.nl carries a stray "DIP" and a Dutch sentence explaining that the entry above is Cloudflare's AI crawler, both of which read as comments that nobody prefixed with a hash. The third is the case worth reproducing, because it is the ordinary way this goes wrong.
healthline.com serves a 567 line robots.txt that names AI crawler tokens and is otherwise carefully maintained. Line 564 reads "Disallow /account/*". There is no colon after Disallow. Every parser that follows either document skips that line, so /account/ is not disallowed for anybody, and nothing in any crawler's behaviour will ever report that back to whoever wrote it. The neighbouring lines in the same file, "Disallow: /" at line 556 and "Disallow: /*?*" at line 566, are both written correctly, which is what makes this kind of error durable: it sits inside a file that visibly works.
Line 566 of that same file illustrates a second and subtler rule. It comes after two Sitemap lines, and a reader could reasonably assume the Sitemap records ended the preceding group and that "Disallow: /*?*" therefore applies to everyone. RFC 9309 says otherwise, in one sentence written for precisely this confusion: a Sitemaps record "MUST NOT terminate a group". So that rule still belongs to the group opened at line 558 by "User-agent: DDM-DCipher/1.0.7", and it constrains that one obscure crawler rather than the whole web. Whether that was the intent is not something a parser can know, and this post does not claim to know it either.
Three other files put an Allow or a Disallow before any user-agent line at all: alz.org, somersetvethospital.com and lehifreepress.com. RFC 9309 addresses that directly, saying "The crawler SHOULD ignore 'disallow' and 'allow' rules that are not in any group (for example, any rule that precedes the first user-agent line)." Those rules apply to nobody. This failure class is small here, three files out of 1,059, and it is worth reporting at that size rather than inflated: the common robots.txt mistakes on this corpus are not ordering mistakes. They are the ones in the previous section and the next two. It is the same pattern this blog found when 559 of 581 blocked pages turned out to be closed by a wildcard rather than by a rule naming the crawler, and when 89 of 145 files ruled only on the whole site. Directives that resolve differently from how they read are more common than directives that are simply absent. One more file, cancer.org, returned a gzip compressed body although the request offered no Accept-Encoding header, which a strict client reads as binary rather than as text; it decompresses cleanly and is counted here as one file, not as a parse failure.
What the file says
- 556 Disallow: /
- 558 User-agent: DDM-DCipher/1.0.7
- 559 Disallow: /
- 561 Sitemap: https://www.healthline.com/sitemap.xml
- 562 Sitemap: https://www.healthline.com/hlcms-articles-99.xml
- 564 Disallow /account/*
- 566 Disallow: /*?*
What a parser takes from it
- 556 applies to the group opened above it
- 558 opens a group; the token is not a valid product token
- 559 disallows everything, for that one crawler
- 561 a Sitemaps record, which does not end the group
- 562 a second Sitemaps record, same
- 564 no colon, so the line is invalid and dropped
- 566 still inside the DDM-DCipher group, not global
224 files carry a field Google says it does not support
This is the most common defect by a wide margin, and the least dramatic. 224 of the 1,059 files use at least one field outside the four Google names, spread across fifteen distinct field names. crawl-delay accounts for most of it, appearing in 158 files, and it is the one case where the documentation is unambiguous and widely ignored: Google's own sentence listing its supported fields names crawl-delay in parentheses as an example of one that is not supported. Other crawlers do honour it, so writing it is not pointless, but it is worth knowing that the largest crawler on the web reads that line and discards it.
Below crawl-delay the list gets more interesting, because it is a record of site owners inventing syntax to say something the standard has no field for. Content-Signal appears in 31 files, carrying values such as "search=yes,ai-input=yes,ai-train=no,use=reference" on unesco.org and svt.se, which is Cloudflare's Content Signals Policy and a convention this blog has already measured at the endpoint level. Host appears in 27 files, a Yandex convention. Noindex appears in 8, including census.gov, whoi.edu and mskcc.org, carrying values such as "/quickfacts/" and "/README.txt", and it is not among the four fields Google lists as supported.
Then there are the one-off inventions, and six of them are all trying to solve the same problem, which is that there is no standard way to point a crawler at an llms.txt from robots.txt. livechat.com, visitdubai.com and td.com each wrote a field called LLMs with the URL of their llms.txt as the value. canadiantire.ca wrote LLM-Content. australia.com wrote LLM. zapier.com wrote Agentmap, pointing at /.well-known/ai-catalog.json. wizzair.com wrote a Link field with a rel="ai-policy" parameter, borrowing HTTP header syntax. Six different spellings of the same intention, none of them in any specification, and none of them read by anything. The gap those sites are working around is real, and this blog has written about how rarely the two files agree with each other when 19 of 23 sites publishing an llms.txt said nothing about AI in robots.txt.
thefurrow.tv went furthest, declaring "Citation-Policy: allow-with-attribution" and "Training-Window: 30d". Both express a position that no field in any published standard can currently carry, and both are read by nothing. That is not a criticism of the site, which is stating something it has every right to state. It is a description of where the protocol is: the intentions people want to express have moved past the four fields the protocol defines, and the overflow is landing in invented syntax that no parser will ever act on. Two files, amsterdam.nl and messly.com, contain a bare URL on a line of its own, which a field and value parser reads as a field named "https" with a value of "//" and the rest, an accidental demonstration of the same thing.
1,394 user-agent values that no product token can match
RFC 9309 defines what may appear after user-agent, and it is narrow: "The product token MUST contain only uppercase and lowercase letters ("a-z" and "A-Z"), underscores ("_"), and hyphens ("-")." A value containing a space, a slash, a dot or an exclamation mark is not a product token, so no crawler looking for its own name can match that group, and the rules underneath it govern nothing.
107 files contain at least one such value, across 1,394 lines, with 732 distinct values between them. Most are legacy blocklists pasted from a list of unwanted bots: census.gov and arxiv.org both carry "Yahoo! Slurp", bundesregierung.de carries "MegaIndex.ru/2.0" and "Mail.RU_Bot", esa.int carries a full browser style string beginning "Mozilla/5.0 (compatible; Yahoo! Slurp". None of those will ever match anything, and the cost is only the false confidence of a long file.
The case that matters for this site's subject is congress.gov, which declares 129 user-agent lines in one group and includes "User-agent: ChatGPT Agent" at line 39. OpenAI's crawler documentation names four product tokens, OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User, and the string "ChatGPT Agent" appears nowhere on it. That line matches nothing. In this instance it costs congress.gov nothing, because the three valid OpenAI tokens are all present elsewhere in the same group and the group is disallowed either way. It is a good illustration precisely because the consequence is nil: the file looks more comprehensive than it is, and only a parser can tell you which of the 129 lines are load bearing.
A second class needs stating carefully, because reporting it as an error would be misleading. 137 files use a user-agent value that is invalid only because it contains a digit, 376 lines in total, and that includes real and current crawler tokens such as AI2Bot and MJ12bot. A literal reading of the RFC grammar excludes digits. Every parser in production accepts them, vendors ship tokens containing them, and treating them as defects would produce a number that is true against the grammar and false against the world. They are therefore excluded from the 232 figure and reported here separately. This is the same territory as the tokens this blog counted when 1,004 robots.txt files named 2,209 distinct tokens, and it is worth remembering throughout that a user agent is a claim, not an identity: a correctly spelled token binds only the crawlers that choose to honour it, and tokens change, which is how renamed crawler tokens leave stale robots rules behind them.
| Value as written | Host it was read from | Why no crawler matches it |
|---|---|---|
| ChatGPT Agent | congress.gov | Space; not one of OpenAI's four documented tokens |
| Yahoo! Slurp | census.gov, arxiv.org | Space and an exclamation mark |
| MegaIndex.ru/2.0 | bundesregierung.de | Dots and a slash |
| Mozilla/5.0 (compatible; Yahoo! Slurp | esa.int | A browser string, not a product token |
| DDM-DCipher/1.0.7 | healthline.com | Slash and dots |
| Python urllib | amsterdam.nl | Space |
| Scrapy/2.9.0 (+https://scrapy.org) | broadinstitute.org | A full identification string |
| AI2Bot | congress.gov | Digit only; accepted by every parser in practice |
What this run did not measure
The figures above are one HTTPS request per hostname on one day, and the limits follow from that. No crawler was observed. No access log was read. Nothing here says that GPTBot, ClaudeBot or PerplexityBot visited any of these sites, or that a defect changed what any of them retrieved. The claim is narrower and it is the only one the evidence supports: these are the files served, and this is what the two published parsing specifications say happens to them.
The sample is a sampling frame rather than the web. The 1,419 hostnames are the committed corpus seeds used across this blog's measurements, weighted toward large organisations across government, education, healthcare, news, software, ecommerce, travel and finance, plus a set grouped by publishing platform. Large organisations are more likely than average to have a maintained robots.txt and more likely to sit behind a CDN, so the HTML at /robots.txt rate of 33 in 1,104 should not be read as a rate across the web in either direction. 315 of the 1,419 hosts did not return a 200 at all and are absent from every percentage above.
A defect is also not a synonym for a problem. 158 sites writing crawl-delay may know perfectly well that Google ignores it and may be writing it for the crawlers that do not. Six sites inventing a field to point at an llms.txt are describing a real gap rather than making a mistake. The two failures in this sample that reliably cost a site something it intended are the HTML page served at /robots.txt and the rule with no colon in it, and those account for 33 and 3 files respectively. Everything else on the list is better read as a map of where the protocol has run out of vocabulary than as a list of errors.
Nor does a clean file mean a readable site. 236 of the 1,059 files name at least one AI crawler token in a user-agent line, and 89 of those 236 also carry a defect, but robots.txt is only the permission layer. Whether a crawler is then served the page, and whether the page carries prose it can read, are separate measurements this blog has taken separately, including the finding that 79 of 115 sites refused a crawler their own robots.txt allows. The method behind all of these runs is documented in the crawlability study, the vocabulary is in the glossary entry for generative engine optimization, and the single host version of the check in this post is the robots.txt tester, which reports per crawler rather than per file.
-
Files servedMeasured 1,419 hostnames asked, 1,104 answered HTTP 200 with a body, on 19 September 2026. -
Parsing outcomeSpecified Derived from RFC 9309 and Google's specification, both read at source on the same date. -
Crawler behaviourNot measured No access logs, no observed crawler requests, no claim that any crawler visited. -
Effect on citationsNot measured Nothing here connects a defect to whether an answer engine cited the site. -
Author intentNot knowable A parser reads lines, not intentions; no file is described here as a mistake on that basis. -
Rate across the webOut of scope The corpus is weighted toward large organisations and 315 hosts returned no 200.
Lantad
Published .
A robots.txt validator is worth less than it sounds if all it does is colour the syntax. The question a site owner is really asking is narrower: of everything written in this file, which lines will a crawler act on, and which will it silently drop. Those are different questions, because the two documents that govern the answer both say, in their own words, that a parser keeps the lines it understands and discards the rest without telling anybody. A file can be 99 percent correct and still fail at the one line that was meant to matter.
Common questions
What does a robots.txt validator check?
Whether each line will be acted on, not whether the file looks tidy. The checks that matter are that the response is robots.txt text rather than HTML, that it is under the 500 kibibyte parsing limit, that every line carries a colon, that rules sit under a user-agent line, that field names are among the four Google supports, and that each user-agent value is a valid product token. Both RFC 9309 and Google's specification say a parser skips what it cannot read and keeps going, so none of these failures produces an error anybody sees.
Why would my robots.txt return HTML?
Almost always because nothing is serving the path and a catch-all route answers instead, returning the home page or an error page with a 200 status. It happened on 33 of the 1,104 hosts that answered on 19 September 2026, including bls.gov and cdc.gov. Google's documentation says it will parse the HTML for rules and ignore everything else, so unless the markup happens to contain directives, the site is left with no rules at all. Only one of those 33 responses contained the string user-agent anywhere.
Does Google support crawl-delay?
No. Google's robots.txt specification, last updated 31 August 2026, lists the fields it supports as user-agent, allow, disallow and sitemap, and names crawl-delay in the same sentence as an example of a field that is not supported. 158 of the 1,059 files measured on 19 September 2026 use it. Other crawlers do honour it, so the line is not useless, but it does nothing at Google.
Is a defect in my robots.txt the reason AI is not citing me?
This measurement cannot say that, and no honest reading of it should. It records what 1,419 hostnames served on one day and what two published specifications say a parser does with those bytes. It observed no crawler, read no access logs, and tested no citation outcome. A defect tells you a line is not doing what it reads as doing, which is worth fixing on its own terms, and nothing more than that.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Answer Engine Optimization (AEO)
Answer Engine Optimization (AEO) is the practice of making a page readable, allowed, and structured enough that AI answer engines can use it as a source when they compose an answer.
Can GPTBot read your robots.txt? 92 of 1,056 sites refused it the file
Lantad asked 1,419 hostnames for /robots.txt three times on 18 September 2026, changing nothing between the requests but the user agent string. 1,056 returned a robots.txt to a desktop browser string. 92 of those returned a non-200 to OpenAI's example GPTBot string, and 63 of the 92 had served the same file to this scanner's own token seconds earlier.
Crawler user agent: 1,004 robots.txt files named 2,209 tokens, and 120 name one Anthropic no longer documents
A user-agent line in robots.txt is an exact product token, and a token that matches nothing is a rule that does nothing. Lantad requested /robots.txt once from each of 1,419 hostnames on 17 September 2026. 1,004 files parsed into at least one group and named 2,209 distinct tokens between them, 1,098 of which appear in a single file, 813 of which contain a character RFC 9309 does not let a crawler put in its own name, and 120 of which name anthropic-ai or Claude-Web, neither of which appears on Anthropic's crawler page.
A 404 on robots.txt allows every crawler, a 503 blocks them all
RFC 9309 gives a missing robots.txt and a broken robots.txt opposite meanings: a 4xx lets a crawler access any resource on the server, a 5xx obliges it to assume complete disallow. Google's documentation carves 429 out of that rule. Lantad's code does not.
Allow AI crawlers in robots.txt: 89 of 145 files ruled only on the whole site
Lantad requested /robots.txt once from each of 1,027 hostnames on 13 September 2026. 706 responses parsed into at least one group, and 145 of those name at least one of the 15 AI crawler tokens this scanner evaluates. On 89 of the 145, every rule in those groups governs the whole site. Fifty six write at least one path rule, and on 29 of the 145 deleting the AI rules entirely would have changed no verdict we tested.
