BlogFindings
Do AI crawlers follow redirects? 878 of 1,322 home pages answered with one
Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for robots.txt on 11 October 2026, then requested the home page of the 1,324 that permitted LantadBot with automatic redirect following switched off, and walked every Location header by hand. 878 of the 1,322 that answered were reached only through at least one HTTP redirect. 1,031 redirect responses carried five different status codes, 62 chains mixed a permanent code with a temporary one, and 14 of the readable home pages moved themselves again in the markup, four of those from inside a noscript element.
This run tested the first half of that assumption rather than the second. On 11 October 2026 Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for /robots.txt as LantadBot, kept the 1,324 that permitted the root, and then requested each home page with automatic redirect following switched off, reading every Location header and following it by hand up to twelve hops with no JavaScript executed. The point of turning the automatic following off is that it hides exactly what this post is about. 878 of the 1,322 hostnames that answered never served anything at the URL they were asked for, and the shape of the detour varies far more than the word redirect suggests.
In short
- The bare domain is usually not where the home page is served: 878 of the 1,322 corpus home pages that answered Lantad on 11 October 2026 were reached only through at least one HTTP redirect, and 444 were served at the URL asked for.
- The question do AI crawlers follow redirects has no answer in the crawler documentation itself: the OpenAI, Anthropic and Perplexity crawler pages, all three read on 11 October 2026, do not contain the word redirect once between them.
- 840 of the 1,031 redirect responses were 301, with 136 302s, 35 308s, 18 307s and 2 303s, and 62 chains mixed a permanent code with a temporary one, so those chains state two different things about the same move.
- 14 of the 1,074 readable home pages carried a meta refresh and 4 of those sat inside a noscript element, which is the one element that fires for a reader with no JavaScript: southwest.com pointed that reader at /html/errors/javascript_disabled.html.
- A chain can end worse than it started. 102 of the 878 redirected chains ended on a non-200 status, among them fda.gov, which sent LantadBot through two hops to an abuse-detection apology page that returns 404.
How many home pages are actually a redirect?
Two in three. 878 of the 1,322 hostnames that produced a terminal response did so somewhere other than the address requested, and 444 answered where they were asked. The depth is shallow almost everywhere: 744 chains were a single hop, 117 were two, 16 were three, and exactly one ran to five, which was nature.com passing through an identity provider at idp.nature.com and coming back to www.nature.com.
Most of that single hop is the oldest convention on the web. 686 of the 744 one-hop chains moved the bare domain to the same path on the www host and changed nothing else. That is not interesting on its own, and this post is not an argument against it. The other 58 are more interesting, because they move the reader somewhere that is not the same page: europa.eu sends a client to european-union.europa.eu/select-language, belgium.be to a language_selection_page carrying a destination query string, portugal.gov.pt to /pt/gc25, and nato.int to /en. In each case the thing the site calls its home page lives at a path, and the bare domain is a router.
That matters more for a scanner than for a browser, and it is the reason this was worth counting. 151 of the 878 redirected chains terminated at a path other than the site root. Every downstream signal a measurement tool reads is attached to that final URL rather than to the one in the sitemap, the canonical tag or the link somebody typed. An audit that reports on the bare domain and an audit that reports on the terminal URL are describing two different documents, and nothing in the output of either says which one happened. Lantad records the full chain for this reason and prints it in the report, which is documented in the methodology. The cost in time is real too, and a previous run measured it: redirected hostnames delivered their first byte in roughly double the time of hostnames that answered directly, which is set out in the post on page speed and AI search.
| Hops | Hostnames | What it usually is |
|---|---|---|
| 0 | 444 | Served at the URL requested |
| 1 | 744 | 686 of them bare domain to the same path on www |
| 2 | 117 | Host move plus a locale or trailing slash correction |
| 3 | 16 | Locale routers and case corrections stacked |
| 5 | 1 | nature.com, out to an identity provider and back |
Do AI crawlers follow redirects, and does any vendor document it?
The honest answer to the first half is that redirects plainly work well enough for crawlers in general, and the honest answer to the second half is that nobody who runs an AI crawler has written it down. Lantad read three crawler documentation pages on 11 October 2026: OpenAI's crawler page, Anthropic's help centre article on its crawler, and Perplexity's crawler guide. All three describe their tokens, their purposes and how to block them. Not one of them contains the word redirect. There is no stated hop limit, no statement on whether a 302 and a 301 are treated differently, and no statement on what happens to a chain that leaves the origin. This is the same documentation gap recorded for the Retry-After header in an earlier run, where no crawler vendor documented it either.
What is documented is the protocol and one search engine's reading of it. RFC 9110 defines the 3xx class and the difference between the codes, including the distinction that 307 and 308 preserve the request method where 301 and 302 historically did not. Google's own page on redirects and Google Search, last updated 2026-04-14 UTC, goes further and sorts the mechanisms into two groups by what they mean for the canonical: permanent, where the indexing pipeline uses the redirect as a signal that the target should be canonical, and temporary, where the source page stays in results. It also ranks them, in its own words, by how likely Google is able to interpret them correctly, with server-side redirects at the top.
That ranking is the useful part for anybody writing about AI crawlers, because the ordering is not arbitrary. A server-side redirect arrives in a response header and needs nothing from the client. Everything below it in that list needs the client to parse a document, and in one case to run a script. A crawler that reads raw bytes sits at the point in the list where the mechanisms stop being reliable, which is where the rest of this post goes.
| Mechanism | Google's class | Needs the client to |
|---|---|---|
| HTTP 301, HTTP 308 | Permanent | Read a response header |
| HTTP 302, 303, 307 | Temporary | Read a response header |
| meta refresh at 0 seconds | Permanent | Parse the document |
| meta refresh above 0 seconds | Temporary | Parse the document and wait |
| JavaScript location | Permanent | Render the page and run a script |
Which redirect status codes sites actually send
The 878 chains contained 1,031 redirect responses between them, and the distribution is lopsided. 840 were 301, 136 were 302, 35 were 308, 18 were 307 and 2 were 303. So 875 of the 1,031 carry the permanent meaning in Google's table above and 156 carry the temporary one, which is roughly what anybody would hope to see for a home page that has no intention of moving back.
The defect is not in the counts, it is in the chains. 62 of the 878 chains contain both a permanent code and a temporary one, which means the chain as a whole asserts two incompatible things about where the page belongs. fda.gov answers 302 and then 301. bundesregierung.de, admin.ch, nps.gov, worldbank.org, insee.fr, cnrs.fr and epfl.ch all do a version of the same. bundesbank.de runs 301, 301, 302, ending on the temporary code, and utah.gov runs 302, 302, 301, ending on the permanent one. Nothing in either document says which code in a mixed chain wins, and there is no reason to assume two engines resolve it the same way.
A smaller group is unambiguously wrong rather than merely ambiguous. 16 chains step down from HTTPS to HTTP partway through, including parliament.uk, hsbc.com, healthdirect.gov.au, cedars-sinai.org, absa.co.za and cas.cn, each sending a client that arrived on a secure connection to the insecure scheme on the www host. Most of those recover on the next hop because the HTTP host redirects back to HTTPS, so a person never notices. A client with a strict transport policy may simply stop. 25 chains leave the registrable domain altogether, and the great majority of those are legitimate brand moves worth knowing about: hdfcbank.com now lands on hdfc.bank.in, icicibank.com on icici.bank.in, spectator.co.uk on spectator.com, and citizenm.com on marriott.com. One is not a brand move. 10thplanetperry.com returns 301 to wargapoker1.xyz, which serves 1,130 words of Indonesian gambling copy under the title WARGAPOKER. For anyone measuring generative engine optimization, that is what an expired domain looks like from the outside: a working 301 into content with no relationship to the site that issued it.
The redirects a reader with no JavaScript cannot follow
1,075 of the 1,322 chains ended on HTTP 200 and 1,074 of those delivered parseable HTML, carrying a median 948 visible words. 14 of those 1,074 documents contained a meta refresh, and 11 of the 14 set a delay of zero seconds, which is the case Google's documentation calls an instant meta refresh and reads as a permanent redirect. Lantad opened all 14 by hand rather than trusting the match, and they split into three groups that mean completely different things.
Seven are plain stubs with the refresh in the head and nothing else in the document. irishrail.ie is 92 bytes of markup whose entire content is a refresh to /en-ie. amsterdamumc.nl is 151 bytes with the title redirect and a refresh to /web/show. rbc.com is 148 bytes carrying a noindex and a refresh to /about-rbc.html. kdca.go.kr, contpaqi.com, bb.com.br and sciencespo.fr are the same pattern at 194 to 320 bytes, and sciencespo.fr is the only one that also supplies a working link in the body for a client that acts on neither. Six of these documents carry zero visible words. A reader that takes the delivered bytes as the page, which is what a crawler reading raw HTML does, gets a site with nothing on it, which is the same failure recorded in the run on noscript elements that held no text arriving by a different route.
Four are the group worth the post. slack.com, mercadolivre.com.br, op.fi and southwest.com put the instant meta refresh inside a noscript element, and noscript is the one element in HTML that fires for exactly the client this site measures. slack.com sends it to /?nojsmode=1 and mercadolivre.com.br to a device configuration endpoint carrying noscript=true, both of which are reasonable fallbacks. southwest.com sends it to /html/errors/javascript_disabled.html and op.fi to /c/portal/no-javascript. Those are error pages. The southwest.com document is 7,604 bytes with a full set of meta tags, a title and a description, and zero visible words in the body, so a crawler that obeys the instruction lands on a JavaScript-disabled notice and a crawler that ignores it reads an empty page. Neither path produces prose parity, and the GPTBot view of that URL is the fastest way to see it on any site. The remaining three are oddities: sbi.co.in sets a five second delay to sbi.bank.in, which Google reads as temporary rather than permanent for a move that is clearly permanent; thehill.com sets 900 seconds, which is a page refresh and not a redirect at all; and patagonia.com delivers an unprocessed Akamai edge template, so the bytes contain an esi:choose block and a refresh whose URL is the literal string $(original_url). Separately, a script-only move is common enough to matter: kaist.ac.kr at 457 bytes and snuh.org at 197 bytes carry nothing but a location assignment in a script tag, which is the mechanism Google ranks last and warns may never be seen if rendering fails. Only two of nine engines rendered JavaScript at all in an earlier run.
| Host | Bytes | Visible words | Delay | Target | Placement |
|---|---|---|---|---|---|
| irishrail.ie | 92 | 0 | 0s | /en-ie | head |
| rbc.com | 148 | 0 | 0s | /about-rbc.html | head |
| amsterdamumc.nl | 151 | 0 | 0s | /web/show | head |
| kdca.go.kr | 194 | 0 | 0s | /kdca/index.do | head |
| contpaqi.com | 210 | 9 | 0s | www.contpaqi.com | head |
| sciencespo.fr | 320 | 5 | 0s | ./fr | head |
| bb.com.br | 1,580 | 0 | 0s | /site | head |
| sbi.co.in | 4,296 | 35 | 5s | sbi.bank.in | head |
| southwest.com | 7,604 | 0 | 0s | /html/errors/javascript_disabled.html | noscript |
| patagonia.com | 14,031 | 100 | 30000000s | $(original_url) | head |
| op.fi | 163,516 | 75 | 0s | /c/portal/no-javascript | noscript |
| slack.com | 241,376 | 1,857 | 0s | /?nojsmode=1 | noscript |
| mercadolivre.com.br | 369,818 | 973 | 0s | device config, noscript=true | noscript |
| thehill.com | 518,577 | 3,152 | 900s | none, self refresh | head |
Where a redirect chain ends badly
Following a chain is not the same as arriving. 102 of the 878 redirected chains ended on a status other than 200, against 129 of the 444 that were served directly, so a redirect is not what decides whether a crawler gets a page. Refusal is, and refusal was measured separately in the run on CDN blocking. What the redirect layer adds is a set of endings that no single response code would have shown.
fda.gov is the clearest. The bare domain answers 302 to /apology_objects/abuse-detection-apology.html, that path answers 301 to the www host, and the www host answers 404. So the site has a bot-detection system, the system routes a refused client to a page that explains the refusal, and the page does not exist. Lantad re-ran it twice, two hours apart, with the same result both times. imf.org is a shorter version: 302 to www.imf.org, then 404 for LantadBot, while the same request with a desktop browser user agent string is refused with a 403 at the bare domain. Both agents fail, by different routes, and neither failure is a redirect problem exactly, which is the point of walking the chain rather than reading one status.
The clearest case of a chain that behaves differently by user agent is flutterwave.com. Asked as LantadBot, it answers 302 to https://flutterwave.com/undefined/, and that path answers 404. Asked with the browser string, the same request answers 302 to /us/. It is a locale router that resolves to the JavaScript value undefined when it cannot place the client, and it then serves a 404 for the URL it just invented. 47 of the 1,323 hostnames compared answered the two agents with a different first status at all, and the honest reading of most of those 47 is that they are about refusal rather than routing: 23 redirected LantadBot where the browser string got a 403 or a 503, and 18 did the reverse. Only two both redirected and sent the two agents to different targets, flutterwave.com being one. A user agent is a claim and not an identity, which an earlier post sets out, and a router that branches on it will branch on a crawler too. The related question of whether a site declares that it varies by agent was counted in the run on the Vary header, and 404s that arrive dressed as a page were counted in the soft 404 run.
GET https://fda.gov/ as LantadBot
- GET https://fda.gov/ 302 to /apology_objects/abuse-detection-apology.html
- GET https://fda.gov/apology_objects/abuse-detection-apology.html 301 to the www host
- GET https://www.fda.gov/apology_objects/abuse-detection-apology.html 404 not found
- Bytes of readable page content delivered none
What to check on your own site
None of this needs a tool to reproduce. Request your own bare domain with automatic redirect following switched off and read the Location header, then do it again for each hop until something answers 200. Four questions are worth asking of the result, and they are in order of how much they cost when the answer is wrong.
First, does the chain end on 200 for a non-browser client. That is the only question whose wrong answer is fatal, and it is the one a browser will never surface, because a browser is not the client being refused. Second, does the terminal document carry the prose. If the bytes at the end of the chain hold a refresh instruction and nothing else, the chain has not finished and a reader that stops at the bytes has nothing. Third, does any hop change scheme, host or path in a way the rest of the site does not know about, because the canonical tag, the sitemap and the structured data all name a URL and a redirect can move the page away from it. Fourth, do the codes agree with each other: a permanent move stated with a temporary code, or a chain that mixes both, is a signal the site did not mean to send.
The one addition worth making to that list is to check it per crawler rather than once, because the robots.txt answer and the redirect answer are independent. A path can be allowed and still unreachable, and both have to hold. The robots.txt tester evaluates a path against each token separately, the AI crawler reference lists the tokens currently worth testing, and the per-platform notes in how to get cited by ChatGPT cover the fetching behaviour for that engine specifically. If the redirect is being generated by a framework rather than a server config, the locale and trailing-slash behaviour is usually the cause, and the Next.js guide covers where that is configured.
Sample Illustrative, not a measurement of any real site.
Flow: Request the bare domain (3xx) to 3xx with a Location; Request the bare domain (2xx or 4xx) to Terminal response; 3xx with a Location to Follow and log the hop; Follow and log the hop (up to 12 hops) to Terminal response; Terminal response (200) to 200 with HTML; 200 with HTML to Prose a reader can use; 200 with HTML to Refresh or script only.
What this run did not measure
It did not measure what any AI crawler does with a redirect. Nothing here observed GPTBot, ClaudeBot or PerplexityBot following or refusing anything. Every request came from LantadBot, from one network location, and the crawler behaviour in this post is quoted from vendor and specification documentation or, where the documentation is silent, reported as silent. The hop limits in the post are Lantad's own walking limit of twelve, not a published limit of anybody's crawler, because no crawler vendor publishes one.
It is one URL per hostname, the home page only, asked once on one day, so a transient failure looks permanent and a site whose home page redirects oddly but whose article pages do not is recorded on the strength of the home page. The 95 hostnames excluded before the walk are excluded rather than counted as anything: 12 disallowed LantadBot at the root, 53 served a 5xx on robots.txt, which the scanner's own parser treats as disallow until it recovers, and 30 failed to return robots.txt at all. The JavaScript redirect figures are the narrowest claim in the post on purpose. A regular expression found a location assignment in the delivered script text of 78 documents, which is not the same as a redirect, so only the documents Lantad opened and read are named, and the count of 78 is reported as a pattern match rather than as a finding.
The corpus is a fixed list chosen to span industries and platforms, not a random sample of the web, so the proportions describe these 1,419 hostnames and nothing wider. Redirects on robots.txt itself, which follow different rules and have their own hop allowance in the protocol, were counted in a separate run and are not in these figures.
-
What sites sendMeasured 1,031 redirect responses across 878 chains, every hop recorded -
What the markup does nextMeasured 14 meta refreshes in 1,074 readable documents, all 14 opened and read -
What a named AI crawler doesNot measured Quoted from vendor documentation, or reported as undocumented -
Whether a redirect changes a citationNot measured No engine was asked anything in this run
Lantad
Published .
A redirect is the most ordinary thing on the web and the least examined. Somebody types a bare domain, the server says the real address is somewhere else, and a browser follows without showing anyone. The assumption underneath every piece of advice about AI crawler access is that this is free: that a 301 costs nothing, that the crawler ends up where a person would, and that the only thing worth auditing is what the final page contains.
Common questions
Do AI crawlers follow redirects?
In practice yes, and no vendor says so in writing. Lantad read the OpenAI, Anthropic and Perplexity crawler documentation pages on 11 October 2026 and none of the three contains the word redirect, so there is no published hop limit, no statement on whether a permanent and a temporary code are treated differently, and no statement about chains that leave the origin. The documented behaviour that does exist belongs to Google, whose redirects page, last updated 2026-04-14 UTC, ranks the mechanisms by how likely it is able to interpret them correctly and puts server-side redirects first.
Is a bare domain to www redirect a problem for AI visibility?
Not on the evidence here. It was the single most common thing in the run, 686 of the 744 one-hop chains, and it ended on HTTP 200 in the overwhelming majority of those. The cases that cost something are different in kind: a chain that ends on a non-200 status, which happened to 102 of 878 redirected chains, a chain that steps from HTTPS to HTTP, which happened 16 times, and a terminal document that carries a refresh instruction instead of the page, which happened 14 times.
Why does a meta refresh inside a noscript element matter more than one in the head?
Because noscript fires for precisely the client that reads raw HTML without running scripts, which is what every documented AI crawler is when it fetches a page. Four of the 14 meta refreshes Lantad found on 11 October 2026 were inside a noscript element, and two of the four pointed at an error page rather than at content: southwest.com at /html/errors/javascript_disabled.html and op.fi at /c/portal/no-javascript. A refresh in the head applies to every client equally. A refresh in noscript is a rule written specifically for the reader this site measures.
How do I tell whether my home page is being served where I think it is?
Request it with automatic redirect following switched off and read the Location header yourself, then repeat for each hop. A browser hides the chain by design, so the only way to see it is to stop following automatically. The four things worth checking at the end are the terminal status code, whether the terminal document holds the actual prose, whether any hop changed the scheme or the host, and whether the codes in the chain agree with each other. 151 of the 878 redirected chains in this run ended at a path other than the site root, which is a different URL from the one most sites name in their own canonical tag and sitemap.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
Prose Parity
Prose Parity is Lantad's metric for the share of a page's rendered, human-visible text that is present in the raw HTML an AI crawler receives, where 100% means a crawler sees everything a human visitor sees.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
Answer Engine Optimization (AEO)
Answer Engine Optimization (AEO) is the practice of making a page readable, allowed, and structured enough that AI answer engines can use it as a source when they compose an answer.
