robots.txt redirect: 122 of 391 sites sent one, and all 122 crossed hosts

What 391 hostnames returned for one request to /robots.txt

239 of the 391 answered HTTP 200 at the hostname we asked, which is the case everybody assumes is universal. 122 answered a redirect. The remaining 30 answered something that carries no rules at all: 20 of HTTP 404, 5 of HTTP 202, 4 of HTTP 429 and 1 of HTTP 403. One hostname, marmitedefontes.com, did not answer inside twenty seconds and is excluded from every figure here.

Those five HTTP 202 responses are the same hosting layer captcha shell we recorded when five of 391 home pages sent an X-Robots-Tag noindex on 9 September 2026, against this identical seed frame. They matter to this measurement for a different reason. A 202 is a 2xx, and under the specification a successful fetch means the crawler must parse and follow whatever it received, so a captcha page delivered at 202 is a robots.txt file with no groups in it rather than an error.

The 404s and the 403 land in a category that is well defined and widely misread, since an unavailable status permits a crawler to access any resource on the server. That is why a 404 and a 503 on robots.txt are opposites rather than two flavours of the same failure. Our own scanner records the same outcome in the same direction: when the robots fetch fails for any reason, the policy becomes allow_all with a note saying so, and FETCH_LIMITS.maxRedirects in core/src/config.ts caps the chain it will follow at five. That is a setting somebody chose to match the specification, not a measurement of anything. How a capture is taken and what it is allowed to conclude are recorded on the methodology page.

The split that follows is only about the 122. Everything below counts inside that group unless it says otherwise.

Does a robots.txt redirect still govern the original host?

Yes, and the specification is unusually direct about it. RFC 9309, the standards track document published in September 2022 that defines robots.txt, gives redirects their own section. It says that crawlers SHOULD follow at least five consecutive redirects, even across authorities, which it glosses as hosts in the case of HTTP. It then says that if a robots.txt file is reached within five consecutive redirects, the file MUST be fetched, parsed, and its rules followed in the context of the initial authority. If there are more than five, crawlers MAY assume the file is unavailable, which routes straight back to the permissive branch.

Read that second sentence against our numbers. The rules that govern a bare domain whose robots.txt redirects to www are the rules written in the www file, and they bind the bare domain. Nothing is lost by the redirect provided the crawler follows it, and 120 of our 122 chains resolved in a single hop with 2 taking two hops, so none of them came close to the five hop ceiling where the permissive branch opens.

Google's robots.txt specification page, last updated 31 August 2026, states its own version: Google follows at least five redirect hops as defined by RFC 1945 and then stops and treats it as a 404 for the robots.txt file, and Google does not follow logical redirects in robots.txt files, naming frames, JavaScript and meta refresh as the kinds it means. Treating it as a 404 matters, because the same page says Google's crawlers treat all 4xx errors except 429 as if a valid robots.txt file did not exist, which means no crawl restrictions.

That is one vendor. We searched the raw bytes of OpenAI's crawler documentation and of Anthropic's crawler page on 10 September 2026 for the string redirect, case insensitively, and found zero occurrences in each. Both pages document their tokens carefully and neither says what happens when the file moves. Anthropic's page carries a date of 7 April 2026 and OpenAI's shows none. This is not an accusation of bad behaviour: undocumented is not the same as unhandled, and the parser almost certainly follows. It is a gap in what a site owner can verify, and it sits alongside the other places where the specification says one thing and the documentation says nothing, such as the behaviour of a trailing star in a path pattern.

Why every redirect in the sample crossed to a different host

Not one of the 122 redirects stayed on the hostname we asked. That sounds like a finding about redirects and is really a finding about what robots.txt redirects are for. A site does not usually redirect one file. It redirects a whole hostname to its canonical form, and /robots.txt is carried along by the same rule that carries every other path.

115 of the 122 were exactly that move, from the bare registrable domain to its www hostname. Zero went from www to the bare domain, in a sample where both directions are common enough in the wild that a single direction is worth stating rather than assuming. The 7 remaining crossed to a different registrable domain and are the subject of the next section.

The canonical direction is not a detail if you have ever edited the wrong file. RFC 9309 defines the robots.txt URI as scheme, authority and path, so example.com and www.example.com are two authorities with two files, and a change made to one does not reach the other. We measured that separately when three of six sites served a different robots.txt on their second hostname on 2 September 2026. The redirect is the thing that stops those two files disagreeing, because it removes one of them. A site that serves two different real files at the two hostnames has a genuine ambiguity. A site that redirects has one file and a rule about where it lives.

There is a platform pattern in the direction as well. In the Webflow category of the frame, 35 of 42 hostnames redirected and 34 of those 35 went from the bare domain to www. In the Wix and Squarespace category, none of the 54 redirected: every one served HTTP 200 at whatever hostname it was asked. Those are two defensible designs and a site owner inherits one of them without choosing it, which is worth knowing before you conclude anything about your own file from a single request. You can see the group a given file produces for a given token with the robots.txt tester, and the gap between editing a file and a crawler acting on it is its own subject, covered in when a robots.txt edit reaches a crawler.

Seven redirects that left the registrable domain entirely

The interesting 7 are the ones where following the redirect hands a crawler a file that a different business wrote. Under the clause quoted above those rules still apply to the host that was asked, so the arrangement is legal, deliberate in most cases, and quietly consequential.

societies.io redirects to societies.ai, and the file it lands on is the most carefully written AI robots.txt in the whole run: 754 bytes naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Amazonbot, Bytespider, YouBot and more, each with Allow: / under it. Whoever wrote that thought about the answering crawlers as well as the training ones, which is the distinction we counted when 16 of 140 files named OAI-SearchBot against 25 naming GPTBot. Every one of those permissions reaches societies.io only through the redirect.

nowhabersham.com redirects to nowgeorgia.com, whose 117 byte file disallows /wp-admin/ and two query string patterns and sets Crawl-Delay: 5. Crawl-Delay is not a rule RFC 9309 defines; it falls under the section on other records, which crawlers may interpret and may ignore. workweave.dev takes its rules from weaveos.com, spellbook.legal from spellbook.com, getalembic.com from alembic.com by way of an intermediate http hop, and prodshort.com from a path inside a CloudFront distribution hostname, where the file is 24 bytes reading User-agent: * and an empty Disallow, which permits everything.

Four of those target files carry a Sitemap line, and in each case the sitemap it names is on the target domain rather than the domain that was asked. A crawler that arrives at workweave.dev and follows the chain correctly is told about weaveos.com's sitemap. That is not a violation of anything. It is what happens when a rebrand moves the canonical hostname and the old domain keeps answering, and it means the old domain's crawl instructions are now maintained by whoever maintains the new one. The crawler directory lists which token belongs to which vendor if you want to check what a given file is actually addressing, and the failure mode where a token itself moves is covered in renamed crawler tokens leave stale robots rules.

Fifteen chains that ended with no rule for the host that was asked

Following a redirect correctly does not guarantee you arrive at a file. 114 of the 122 chains ended on HTTP 200, 6 ended on HTTP 404 and 2 on HTTP 403. The 8 that ended on a 4xx are the clean case of the permissive branch: equala.org, consciencecart.com, caseflood.ai, dollyglot.com, spottedinprod.com and verdaccio.org each redirect their robots.txt to their www hostname, where the file is not there, and optainhealth.com and t-townroofing.com redirect to a 403.

Among the 114 that reached HTTP 200, seven more carry nothing a crawler can act on. Five returned a body of zero bytes, which is a valid robots.txt containing no groups, so it permits everything by saying nothing. Two returned an HTML document with a 200 status. www.rocketable.com answers /robots.txt with a full HTML shell served as text/html, and permitify.com is the odd one in the entire run: its robots.txt redirects to www.permitify.com/robots.txt, which redirects again to the bare home page of pangeanotes.com, a different company, which answers 200 with HTML. RFC 9309 is clear about what that produces, since if no groups are present at all, no rules apply.

So 15 of the 122 redirect chains end with no applicable rule and 107 end at a non-empty body that is not an HTML document. Serving HTML at /robots.txt is not rare and not confined to redirects: we found two of 66 Squarespace sites doing it directly on the same day. The status code is what makes it quiet. A 404 tells a crawler nothing is there. A 200 tells it that what came back is the file, and the parsing rule says to use whatever lines are parseable, which for an HTML document is none of them.

None of this describes crawler behaviour. Whether a crawler follows the chain, whether it caches the result, and whether it asks for the file at all are separate questions, and the last of them has an unflattering answer: in a controlled study of ten consumer assistants, six never requested robots.txt in any trial. The permission a file grants and the request a server actually serves are also different things, which is what made sites that ban GPTBot serving it a 200 anyway worth counting.

What to check when your own robots.txt redirects

Four checks, in the order that decides the answer, and none of them needs a tool.

First, ask for /robots.txt at every hostname that resolves to your site, not just the one you think of as canonical, and do it without following redirects so you can see the status line rather than the destination. A 3xx here is not a fault. It is information about which file is the real one.

Second, look at where the Location points. If it is your own www hostname, the file there is the file that governs both, and it is the only one worth editing. If it points at a domain you do not control, or at a bucket or distribution hostname, then whoever controls that file controls your crawl instructions, and a change you did not make will reach crawlers as though you had made it.

Third, follow the chain to the end and read the status and the first bytes of the body. Eight of our 122 ended on a 4xx, which reads to a crawler as no restrictions at all, and seven more ended on a 200 that carried nothing parseable. All fifteen would have looked healthy to anyone who checked only that the first request returned a redirect.

Fourth, remember which layer you are actually testing. robots.txt is a request written in a file, and the response your origin and edge send to a crawler is a separate mechanism that can disagree with it, which is why it is the weaker of the two layers that decide whether AI can read your site. If you want the other layer, what GPTBot sees renders a page the way a crawler receives it, and AI visibility is the wider measurement all of this sits inside.

The finding underneath all four is small and worth stating plainly. A redirected robots.txt is normal, it is provided for in the specification, and on this sample it worked: 107 of 122 chains delivered a real file to a crawler that follows the rules. The cost of the arrangement is that the file governing your hostname is not always at your hostname, and on 7 of 391 sites it was not even on your domain.