Robots.txt not working: 34 of 1,104 sites answered a crawler with an HTML page

Why is my robots.txt not working when the URL loads fine?

Because loading fine and carrying rules are different properties, and only one of them is visible from a browser tab. The chain from a request to a verdict has four steps, and a file can pass the first three and fail the fourth without anything announcing it.

RFC 9309, the Robots Exclusion Protocol, sets the terms. Section 2.3 states that the file MUST be UTF-8 encoded and Internet Media Type text/plain. Section 2.3.1.2 says crawlers SHOULD follow at least five consecutive redirects, even across authorities. Section 2.2.1 covers what happens once the body is parsed: a crawler finds the group matching its product token, falls back to the group with the star value if no group matches, and where no groups are present at all, no rules apply. Section 2.2.2 adds that a matching group with no rules in it leaves the URI allowed.

Read those together and the failure mode falls out. The media type requirement in section 2.3 is an obligation on the server, not an instruction to the crawler to reject a body that breaks it. Neither the RFC nor Google's robots.txt documentation, read at source on 3 November 2026, tells a crawler to discard a successful response because its content type was wrong. Google's page says the opposite about the status: success codes lead crawlers to process the robots.txt file as provided by the server. So an HTML document returned at 200 is processed, parses to zero groups, and lands on section 2.2.1, where no rules apply. The outcome is identical to a 404, which Google's page says it treats as if a valid robots.txt file did not exist, and this blog has already published that a 404 on robots.txt allows every crawler while a 503 blocks them all. The difference is that a 404 is legible. A 200 carrying a web page is not.

What 1,419 sites answered when asked for robots.txt

One request per hostname, sent on 3 November 2026 as LantadBot/1.0 with redirects followed. 1,104 hosts answered HTTP 200. 158 answered 403, 57 answered 503, 51 answered 404, and 31 did not answer at all through DNS, TLS or a twenty second timeout. The remaining 18 were spread across 202, 429, 401, 406, 418, 498 and 529, which accounts for all 1,419.

Eight of those 18 answered 202, which matters because 202 is a success code and Google's documentation groups all 2xx together as responses to process as provided. Counting the 2xx responses rather than only the 200s gives 1,112 successes, of which 84 parsed into no group, because all eight of the 202 responses were empty or were tiny HTML documents. The figures through the rest of this post use the 1,104 that answered exactly 200, which is the frame every recent corpus post on this blog uses, and the eight extra cases sit in the same category rather than a different one.

Of the 1,104, the shipped parser resolved 1,028 into at least one user-agent group, and 76 into none. That 1,028 is the usual denominator here: it is the set this blog counted when it reported that 25 of 1,024 files refused Claude's search crawler yesterday, and the set behind the finding that 1,004 files named 2,209 crawler tokens. Within those 1,028, 221 files name at least one of eleven AI crawler tokens and 128 disallow the site root to at least one of them, so roughly a fifth of the files with rules have taken a position on AI crawling.

The 76 have not taken a position, and cannot. They are not a rounding error on that fifth: they are 6.9 percent of every host that answered successfully, and they split almost evenly into two causes that look nothing alike. 34 were HTML documents. 42 were genuine plain text files with no rule in them. The next three sections take those in turn, because the fixes are different even though the verdict is the same.

The 34 responses that were not robots.txt files at all

Every one of the 34 began with an HTML doctype or an html root element, and every one of the 34 carried a text/html content type, so the body and the header agreed with each other and disagreed with the request. Together they weighed 8,292,165 bytes, against a median response size of 730 bytes across all 1,104 bodies that answered 200. The largest was technion.ac.il at 1,445,846 bytes. The smallest was myntra.com at 483 bytes.

The titles say what the pages are. technion.ac.il returned the Technion home page, tewhatuora.govt.nz returned Health New Zealand, nikkei.com returned the Nikkei front page, interpol.int returned the INTERPOL home page, and bls.gov returned the home page of the US Bureau of Labor Statistics. 26 of the 34 were an ordinary page of the site in this way. Five carried a title naming a failure: kaust.edu.sa was titled 404, nus.edu.sg was titled 404 and URL invalid, lloydsbank.com was titled Internet Banking and Error, and aeromexico.com and myntra.com were both titled Site Maintenance. All five answered 200 while saying in their own markup that something had gone wrong, which is the same mismatch between status and content this blog measured when it found 125 of 1,094 sites answering a nonexistent URL with HTTP 200.

The remaining three are the most interesting, because they were identical. khanacademy.org, eurostar.com and myswitzerland.com each returned a byte identical 3,038 byte document titled Client Challenge, carrying a Content-Security-Policy in a meta element. That is a bot protection interstitial, served at HTTP 200 to a request for a plain text file, which means the defence that is supposed to sit in front of a site answered on behalf of the one file whose job is to tell a crawler what the site permits. Lantad has measured the front door version of this before, reporting that 103 of 1,089 sites served an unknown bot something while refusing GPTBot, and that 92 of 1,056 sites refused GPTBot the file itself. A challenge page at 200 is worse than a refusal, because a refusal is a status a crawler is told how to handle.

Eight of the 34 crossed the size limit. Google's documentation states that Google enforces a robots.txt file size limit of 500 kibibytes and that content after that limit is ignored. polimi.it, technion.ac.il, wiley.com, tewhatuora.govt.nz, nikkei.com, falabella.com, accor.com and trenitalia.com were all larger than that. It is worth being precise about what the limit did here, which is nothing: there were no rules at any offset in any of the eight, so there was nothing past the cut to lose. The two documents also differ on what 500 kibibytes is. RFC 9309 section 2.5 says a crawler's parsing limit MUST be at least 500 kibibytes, which is a floor on how much a crawler has to read. Google's page describes the same number as a ceiling on how much it will read. Both statements can be true at once and a site near the boundary should not assume they say the same thing.

Three ways a successful response carries no rules

The 34 reached the same verdict by three different routes, and knowing which route a site took is the whole of the fix. None of the three is visible in a status code, and two of the three are invisible in a browser as well, because a browser renders the HTML it was handed and shows a page rather than an error.

The first route is a redirect that drops the path. 23 of the 34 finished at a URL whose path was no longer /robots.txt. bls.gov answered the request with a 302 whose Location header was https://www.bls.gov, with the filename removed, and the second request returned the home page. accor.com answered with a 301 to https://group.accor.com/ and then a 307 to /en. In both cases the redirect was followed correctly, as section 2.3.1.2 requires, and arrived somewhere that was never the file. This is a cousin of the behaviour behind 122 of 391 robots.txt redirects crossing to a different host, except that a cross host redirect at least lands on a robots.txt file, which RFC 9309 says to apply to the original authority.

The second route is a page served at the right URL. 11 of the 34 were delivered at /robots.txt with no path change at all, including lloydsbank.com, hamad.qa, patagonia.com, schwab.com and the three Client Challenge documents. Here the request reached the intended resource and the server answered with markup, which is a configuration or a defence rather than a routing mistake.

The third route is a failure dressed as a success, which is the five soft error pages. kaust.edu.sa took both routes at once: it redirected to a path containing error and 404 and then returned 200 with a document titled 404.

All three end in the same place. Running evaluateRobots from core/src/robots.ts against the path / for GPTBot returns allowed for all 34, and for all 42 in the next section, so 76 of 76. That verdict is not a judgement about the site owner's intention. It is the only verdict available, because section 2.2.1 has nothing else to offer when there are no groups. A reader can check a single file against a named token with the robots.txt tester, which evaluates a path per crawler rather than reporting the syntax, and can see what a crawler receives for a page with what GPTBot sees.

One request, followed to the page it actually returned

The abstraction is worth replacing with a single trace, because the numbers above describe something that takes two requests and four seconds to see. bls.gov was re-requested by hand on 3 November 2026, with the same user agent and the redirect followed one hop at a time.

The first request went to https://bls.gov/robots.txt. The answer was HTTP/1.0 302 Moved Temporarily with a Location header reading https://www.bls.gov. The filename is not in that Location value. A redirect that adds a www prefix would normally preserve the path, and this one does not, so the second request is for the site root rather than for the file. That second request returned HTTP/2 200 with a content type of text/html and a body of 79,128 bytes, which is the home page of the US Bureau of Labor Statistics. Passing that body to parseRobotsTxt returns zero groups, zero rules and zero sitemaps.

Nothing in the exchange is an error by the terms of either document. The redirect is a legitimate 302. Following it is what section 2.3.1.2 asks for. The 200 is a genuine success. The body is valid HTML. The only thing wrong is that a crawler asked a question and was handed an unrelated answer at full confidence, and the protocol has no way to say so. A site owner checking this by opening the URL sees a working page and concludes the file is fine, which is why this failure survives in production on sites with an operations team.

The same trace on the other 22 path dropping cases differs only in the hostnames. What it means for AI visibility is specific rather than general: these sites have not blocked GPTBot, ClaudeBot or PerplexityBot, so a reader who assumes the worst is wrong. They have also not allowed them deliberately, and a per crawler rule that anyone writes into the file later will not take effect, because the file the crawler reaches is not the file being edited. The registry of tokens the scanner evaluates is listed on the AI crawlers page, and the gap between a rule and what a crawler then receives is the subject of the finding that 79 of 115 sites refused a crawler their own robots.txt allows.

The 42 real robots.txt files that still governed nothing

The other 42 of the 76 were plain text, correctly served, and carried no rule. These are the honest version of the same outcome, and they are much more common on managed platforms than on hand built sites.

10 were empty or whitespace only, 9 of them exactly zero bytes, including gov.scot, visa.com, med.or.jp and mediclinic.co.za. 27 held one or more Sitemap lines and nothing else, which is a perfectly valid file: a Sitemap directive is site wide and sits outside any group, so a file can declare a sitemap while containing no user-agent line at all. This blog has counted the directive itself, reporting that 839 of 1,016 files declare a sitemap, and the 27 here are the subset where the declaration is the entire file.

Those 27 concentrate sharply. 14 are in the Webflow stratum and 6 in the SaaS marketing stratum, and the Webflow figure is the platform default rather than 14 independent decisions. Across that stratum, 42 hosts were asked, 38 answered 200, and 18 of the 38 carried no rule, 14 as Sitemap-only files and 4 as empty ones. That matches what Lantad found on a different sample on 11 September 2026, when 20 of 40 Webflow robots.txt files held no rule at all. Two independent samples putting the figure near half is a stronger statement than either alone, and it is a statement about a platform default rather than about the people using it.

The last 5 held nothing but comments. One, lamoncloa.gob.es, was a single hash character. The other four were byte identical at 1,248 bytes each: treepoints.green, rejot.dev, replicad.xyz and highlandscurrent.com each served a block of prose beginning with a statement that, as a condition of accessing the website, you agree to abide by the following content signals. It is a licence condition written where a crawler is specified not to read it, because RFC 9309 treats everything after a hash as a comment. Four sites have therefore published terms in the one place guaranteed to be discarded, while writing no Disallow rule to accompany them. That is the inconvenient reading, and it is the right one: a comment is not a control, and this blog made the same point about the content signal line that asks while the Disallow lines decide.

What this measurement does not show

This is one request per hostname on one date from one network location, so most of what a reader wants to know is outside it.

It does not show what any crawler did. Every figure here is what LantadBot received and what this repository's parser made of it, which is a statement about the responses and the standard rather than about GPTBot, ClaudeBot or Googlebot. A crawler that checks the content type before parsing would reach a different conclusion on all 34 of the HTML bodies, and none of the three documents this post cites requires it to, so whether any crawler does is unmeasured here and is not inferable from a site side scan. The honest claim is the narrow one: by the rules as written, these responses carry no rules.

It does not show that the sites are unprotected in practice. A site can refuse a crawler at the edge while serving it a useless robots.txt, and the three Client Challenge responses are evidence of exactly that posture. Nothing in this measurement reads a page body, so a site in this list may be blocking AI crawlers effectively by another route, or may be serving everything to everyone, and this run cannot separate those two. The scanner's method, including the sampling frame, is on the methodology page and the corpus itself is described in the research section.

It does not show intent, and the distinction matters more here than in most posts on this blog. A Webflow site with a Sitemap-only file has accepted a default. A government home page served at /robots.txt is a misconfiguration. Both resolve to allowed and only one of them is a decision. The measurement reports the verdict and cannot report which of the two produced it.

And it does not show your site. These 1,419 hostnames are two sampling frames stratified by platform and sector, not a random draw from the web, and the smaller strata hold about thirty sites each. The move from here is cheap: request your own /robots.txt with redirects followed, check that the final URL still ends in robots.txt, check that the content type is text/plain, and check that the body contains a user-agent line. If any of the three fails, the rules you believe you have are not being applied, whatever you have written. Lantad's own crawler identifies itself and publishes what it does at /bot, the vocabulary is defined in the generative engine optimization entry, and the platform guides for getting cited by ChatGPT and getting cited by Claude cover what each crawler needs after the file lets it in. For a site whose file turns out to be a different file entirely, the earlier finding that three of six sites served a different robots.txt from the one governing the captured page is the nearest thing in this corpus to a second opinion.