A 404 on robots.txt allows every crawler, a 503 blocks them all

Why a missing robots.txt and a broken robots.txt mean opposite things

Section 2.3.1 of the protocol splits the outcome of a robots.txt fetch into four named cases: successful access, redirects, unavailable status, and unreachable status. The two that decide this question are the last two, because they are the two that leave the crawler holding no rules at all.

Unavailable is section 2.3.1.3. It defines the case as the crawler trying to fetch the file and the server responding with status codes indicating that the resource in question is unavailable, and it gives the 400 to 499 range as the HTTP example. The consequence is one sentence long: if a server status code indicates that the robots.txt file is unavailable to the crawler, then the crawler may access any resources on the server. A 404 is therefore not the absence of an instruction. It is an instruction, and the instruction is yes.

Unreachable is section 2.3.1.4, and it runs the other way. If the robots.txt file is unreachable due to server or network errors, the document says, this means the robots.txt file is undefined and the crawler must assume complete disallow, with server errors identified as the 500 to 599 range. That is the strongest requirement level the document uses anywhere. Not should, not may. A crawler that carries on fetching your pages while your robots.txt returns 503 is not following the specification it says it follows.

There is a release valve, and knowing about it changes the shape of the problem from permanent to temporal. The same section continues: if the robots.txt file is undefined for a reasonably long period of time, for example 30 days, crawlers may assume that the robots.txt file is unavailable as defined in section 2.3.1.3, or continue to use a cached copy. So a 5xx that lasts an hour is a lockout, and a 5xx that lasts two months converges back on the 4xx behaviour. The dangerous window is the middle of that range: long enough to cost you every conforming crawler on the web, short enough that nothing has yet timed out into permissiveness.

Two consequences follow for anyone thinking about AI visibility rather than about classical search ranking. The first is that this is one of the very few controls in the stack that fails closed by design. Almost everything else that governs whether a machine can read your pages fails open: a missing structured data block, an absent llms.txt, a stylesheet that did not load. A 503 on one small text file is the exception, and it takes the whole origin with it. The second is that robots.txt sits at a different layer from the thing that actually returns status codes to crawlers, which is the argument in two layers decide whether AI can read your site. Your file can be flawless and your edge can still refuse the request. Your file can also be flawless and 503, at which point its flawlessness is irrelevant, because nothing parsed it. How the score works keeps those two as separate findings for exactly that reason.

Google documents a different 5xx path, and pulls 429 out of the 4xx rule

The specification says what a conforming crawler must do. It does not say what any particular crawler does, and the operator of the largest fleet publishes a description that departs from it in two places.

Google's robots.txt documentation sets out handling by status class. For 5xx it does not say complete disallow and stop. It says that for the first 12 hours Google stops crawling the site but keeps trying to fetch the robots.txt file, that if it cannot fetch a new version then for the next 30 days it will use the last good version while still trying to fetch a new one, and that if the errors are still not fixed after 30 days, and the site is generally available to Google, it will behave as if there is no robots.txt file.

Set that beside the RFC and it reads less like a contradiction than like a gap filled in. The specification permits a cached copy after a reasonably long period; Google names the period, names the copy, and names an intermediate state the RFC does not describe at all, which is half a day of not crawling followed by a fallback to your last good rules rather than to nothing. If your robots.txt broke this morning and its last working version disallowed a training crawler, that disallow is documented as still in force tomorrow. Complete disallow and last good version produce the same answer only when the last good version was itself restrictive, and for most sites it was not.

The second departure is a single status code, and it is the one worth writing on a card. Google's documentation states that its crawlers treat all 4xx errors, except 429, as if a valid robots.txt file did not exist. 429 is Too Many Requests: exactly the status a rate limiter or a bot defence returns when it decides a client is being greedy. Under the plain text of the RFC, 429 falls inside the 400 to 499 range, so it is unavailable, so the crawler may access any resource on the server. Under Google's documentation it is pulled out of that rule by name. The same three digits open the gates in one text and are explicitly withheld in the other.

That is not a hypothetical on this site. When we captured six real storefronts on 15 July 2026, allbirds.com answered our honestly identified crawler with HTTP 429 and produced no score at all, which is reported in what a crawler actually meets on a real storefront. That 429 was on the page request rather than on the robots.txt request, and the distinction is the whole point: a defence tuned to throw 429 at automated traffic has no particular reason to exempt one text file, and if it does not exempt it, the file that governs every crawler on the site starts returning a status that two primary documents read in opposite directions. The other vendors do not settle it. Read OpenAI's bots documentation and Anthropic's crawler article as they stood on 29 July 2026 and neither carries any statement about HTTP status codes or robots.txt fetch failures. Both tell you the tokens and how to block them. What happens when the file itself errors is undocumented in both directions, and if you find a page on either host that covers it, we will correct this post.

The four numbers RFC 9309 attaches to a robots.txt fetch

Most of the protocol is prose about grammar and matching. Four numbers in it constrain implementations, and all four are worth knowing because each one is a place where a crawler can legitimately behave differently from the one next to it while both remain conformant.

Five is the redirect budget. Section 2.3.1.2 says crawlers should follow at least five consecutive redirects, even across authorities, which in the HTTP case means across hosts. If the file is reached inside those five hops it must be fetched, parsed, and its rules followed in the context of the initial authority, so a robots.txt served from a CDN hostname still governs the original site rather than the CDN. Past five hops, crawlers may assume the file is unavailable, which routes straight back to the permissive branch. Google's documentation puts the same number differently: it follows at least five redirect hops and then stops and treats it as a 404 for the robots.txt file. A redirect loop on that path therefore ends in an open site, not a closed one.

Twenty four is the cache ceiling. Section 2.4 says crawlers may cache the file's contents, may use standard cache control as defined in RFC 9111, and should not use the cached version for more than 24 hours unless the file is unreachable. That last clause is the one people miss. It is the hook that makes Google's 30 day fallback conformant: once the file is unreachable, the 24 hour ceiling stops applying, and a crawler may hold your old rules for as long as it likes. It also means a robots.txt change you made this morning is not guaranteed to be in effect this afternoon.

Five hundred kibibytes is the parsing floor. Section 2.5 says crawlers should impose a parsing limit and that the limit must be at least 500 kibibytes. It is a floor on the crawler, not a cap on you, but the practical reading is the same either way: rules past roughly half a megabyte are outside what any crawler is obliged to read. Very few robots.txt files come close, and the ones that do usually got there by generating a Disallow line per URL, which is a different problem.

Thirty days is the amnesty, and it is the only one of the four expressed as an example rather than a requirement. The text reads for example 30 days, which leaves the reasonably long period undefined and lets each crawler pick. Google picked 30. Nothing obliges anyone else to. Put the four together and the honest summary is that the specification pins down direction far more tightly than timing: everyone agrees a 5xx closes the site and a 4xx opens it, and nobody has to agree on how long the closure lasts. That is a very different guarantee from the one people assume they have, and it is the same shape as the gap between what llms.txt promises and what crawlers do with it.

What Lantad's scanner does with each status, read from its own source

All of the status handling in this product lives in one exported function, robotsPolicyFromStatus in core/src/robots.ts, and it is short enough to describe completely. It takes a status and a body and returns one of three policies. A status from 200 to 299 returns kind parsed, with the file handed to the parser. A status of 500 or above returns kind assume_disallow, carrying a note that says robots.txt returned that status and that crawlers commonly treat a failing robots.txt as disallow-all until it recovers. Everything else returns kind allow_all, carrying a note that says per RFC 9309 an unavailable robots.txt imposes no restrictions.

Three branches, and the third one is a default rather than a case. That is where 429 lands. Google's documentation pulls 429 out of its 4xx handling; this code does not, because the branch is written as a range test and a fallthrough. Whether that is right is arguable in both directions. The RFC's plain text is on the side of the fallthrough. Google's documented behaviour is on the other side, and Google is the crawler most likely to be looking at your site. Naming it is the part that is not arguable, which is why it is in this post and in the takeaways rather than in a comment nobody reads.

The second divergence is larger and it is upstream of that function. When the robots.txt fetch fails outright, from a network error, a timeout, or the scan's own request budget running out, core/src/pipeline.ts never calls robotsPolicyFromStatus at all. It constructs an allow_all policy directly, with a note recording that the file could not be fetched and was treated as unrestricted. Section 2.3.1.4 names network errors alongside server errors as unreachable, so a conforming crawler must assume complete disallow there. Lantad assumes the opposite. The argument for it is that a scanner is not a crawler harvesting content, and that reporting a site as fully blocked because our own request timed out would be the confidently wrong grade this product exists not to emit, an instinct set out in why we withhold a grade. The argument against it is that the specification does not carve out scanners. Both are true. We have not changed the behaviour, and this post does not promise that we will.

What the code does get right is refusing to dress the assumption up as a rule. When the policy is assume_disallow, core/src/verdict.ts still applies the access deduction of 0.4 for each penalised purpose class, and the penalised classes are search and user_agent, so a failing robots.txt costs 0.8 of a sub-score that starts at 1.0. But the wording changes: instead of stating that robots.txt disallows AI search crawlers, the deduction reads that robots.txt is erroring, so they are treated as disallowed. The per-class block defects are suppressed entirely in that state, because every crawler then reads as disallowed with no rule attached, and raising them would invent a Disallow line that does not exist. A single defect called robots_unavailable, severity major, titled robots.txt is failing, carries the finding instead. The bot access matrix marks the same rows as blocked with robots.txt failing rather than as blocked by a rule. That distinction between a measurement and an assumption is the same one behind a user agent is a claim, not an identity, and the same reason our scanner publishes what it is.

What to check on your own site, and what a scan cannot tell you

The check itself is one request and it is not the interesting part. Fetch your own robots.txt and read the status line. If it is 200, read the body and confirm it is the file you think it is rather than a soft 404 or an HTML error page served with a 200, which is common and which the parser will happily read as a file with no rules. If it is anything else, you have found the thing this post is about, and the direction of the problem depends on the first digit.

The interesting part is that a single check is a snapshot and this failure is usually intermittent. A robots.txt that returns 503 under load is fine every time you look at it and broken during the traffic spike, which is precisely when crawlers are also most likely to arrive. There is no way to see that from outside with one fetch, and there is no way to see it from inside either unless your logging separates requests for that one path. Two things are worth doing about it and neither is a purchase. Add /robots.txt to whatever already pings your homepage, so the file has its own uptime line rather than inheriting the site's. And look for it in your access logs by path, counting non-200 responses, because the file is small enough that nobody notices it failing.

There are also limits worth stating plainly. An external scan can tell you the status your server returned to one request from one address at one moment, which is what the robots.txt tester reports per crawler token. It cannot tell you what your server returns to a real crawler from a real crawler network, because that is a different client with a different reputation hitting possibly different edge rules, and it cannot tell you what any crawler already has cached from before your file broke. Under section 2.4 that cached copy can outlive the 24 hour ceiling for as long as the file stays unreachable, so a correct answer today does not describe what crawlers are currently acting on.

Which is a specific instance of a general point about generative engine optimisation that is worth ending on. The measurable and the decisive are not the same set. A status code is measurable and decisive, which makes it worth checking first. Whether a crawler honoured what it read is decisive and not measurable from outside, which is why the tokens that never appear in your logs stay invisible however carefully you look. Between those sits the layer most sites actually lose on, which is whether the page a crawler receives contains the text a browser shows: that is prose parity, and what GPTBot sees returns the received text so you can read it yourself. Get the status code right first, because a 503 on one file makes every other question moot, and the per crawler access view will show you which tokens are affected once it is.