Does page speed affect AI search: on 184 of 379 home pages a redirect took half the wait

Does page speed affect AI search, and what Google documents about it

Start with the one operator that writes the answer down. Google's crawl budget documentation splits a site's budget into a crawl capacity limit and crawl demand. The capacity limit, which the page also calls hostload, is described as a limit on the total amount of time your server spends holding connections open for Google, factoring in both the number of parallel connections and their duration. Then it says what moves it. Under the heading crawl health the page states that if the site responds consistently and its response times, including latency and Time to First Byte, remain stable or improve, the limit goes up and more connections can be used to crawl; and that if the site slows down, or responds with 5xx server errors or rate limiting signals such as HTTP 429, the limit goes down and Google crawls less.

Three things about that are worth holding onto before any measurement. The first is that it is a statement about crawl volume, not about placement. A slow server means fewer pages fetched per unit of time, which for a large site means slower discovery of new and changed pages. It does not say a slow page ranks lower, and reading it that way is how a documented mechanism becomes an undocumented promise. The second is that no number appears anywhere on the page. There is no threshold, no target, no millisecond figure a site owner could check themselves against. The third is that the page has moved: the older address under the Search docs, at developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget, answered our request on 14 September 2026 with a permanent redirect to developers.google.com/crawling/docs/crawl-budget, so the crawl documentation now sits in a section of its own.

That last detail is small and it sets up the rest of this post, because a permanent redirect is exactly what most of the sites we measured serve to anything that asks for their home page. Google's own documentation on how much crawling a server earns is itself behind one.

None of this is written about AI crawlers. It is written about Googlebot, and the crawl budget page makes no mention of any AI surface, so anything it implies for AI answers is inherited rather than stated. What the other operators say, which is close to nothing, is a section of its own below. The pattern of one vendor documenting a limit and the rest publishing silence is one we have measured before, most directly when Googlebot's 2MB fetch limit turned out to have no counterpart in three other crawler documentation sets.

What 379 home pages did when a crawler asked once

The sample is the platform sampling frame committed to this repository at worker/seeds/corpus-seeds-platform.json, generated on 3 August 2026: 392 hostnames stratified by the platform a site is built on, from Framer and Webflow through Shopify storefronts and static documentation sites. It is deliberately not a random sample of the web, and it is weighted toward small and mid sized sites rather than the large organisations in our industry frame, so read every figure here as a description of those 392 and nothing wider.

Each hostname was asked for https://<host>/ as LantadBot/1.0, the user agent documented on our bot page, with a 25 second timeout and no JavaScript executed. Two passes followed redirects and recorded time to first byte of the final response; a third pass did not follow redirects and recorded the first status code. One hostname, whatsupnewp.com, did not complete within the run and is excluded from every count, leaving 391.

Of those 391, 379 ended at an HTTP 200 with an HTML content type. Five answered HTTP 202 with a body of exactly 169 bytes, which is the same signature we traced to a captcha redirect when five of 391 home pages sent an X-Robots-Tag noindex on this corpus five days earlier. Two answered 403, two answered 404, and three failed at the transport layer.

Across the 379, the median page delivered its first byte 0.467 seconds after the request left our client. A quarter were under 0.278 seconds, three quarters were under 0.601, the ninetieth percentile sat at 0.758 and the ninety fifth at 1.006. Nineteen pages took longer than a second, three took longer than a second and a half, and the slowest in the whole sample, lehifreepress.com, took 1.805 seconds. Nothing timed out. Nothing came close to timing out.

That is the first honest answer to the question in the title, and it is a deflationary one. If there is a response time at which an AI crawler gives up, no site in this sample was anywhere near it, and no operator publishes the number that would let anybody check. A post that told these 379 site owners to go and optimise their servers for AI would be selling a fix for a problem none of them has.

The measurement's own limits belong here rather than in a footnote. Every request left from one client on one network on one morning, so the absolute figures carry our own network path and are not a benchmark anybody should compare their own server against. The two timed passes disagreed by a median of 12.8 percent across the 376 hostnames where both succeeded, which is the resolution of this instrument: differences smaller than that are noise. The comparison the next section rests on is a factor of two, which is comfortably outside it. How a capture is scored here is a separate question from how fast it arrived, and speed is not an input to the score at all.

Half the wait on 184 pages was spent on a response with no page in it

Now the part of the wait that is not the page. With redirect following switched off, 186 of the 391 hostnames answered the first request with a 3xx: 163 sent a 301, 17 sent a 308, five sent a 307 and one sent a 302. RFC 9110, the HTTP Semantics specification of June 2022, defines 301 Moved Permanently in section 15.4.2 and 308 Permanent Redirect in section 15.4.9, and the practical difference between them is method preservation rather than anything a crawler fetching a home page would notice. Following the redirects, 184 of the 379 pages that ended at HTML had passed through at least one hop, 178 through exactly one and six through two.

Where do they go? Overwhelmingly one place. 173 of the 184 were the bare domain handing back the www hostname. Ten landed on a different registered domain entirely, which is what a rebrand, a domain sale or a lapsed registration looks like from the outside, and one, bindwell.ai, appended a locale path. Not one went the other way, from www to the bare domain.

The cost is measurable and it is not small. The 184 redirected hostnames delivered their first byte of HTML in a median 0.562 seconds. The 195 that reached the page with no redirect at all did it in a median 0.282 seconds, almost exactly half. Measured directly rather than by subtraction, the redirect hop itself took a median 0.299 seconds and accounted for a median 50.8 percent of the total wait on the first pass. On a typical site in this sample, the second round trip is not a rounding error on the response time Google says it watches. It is half of it.

There is an obvious objection and it is a fair one, so it goes in the body rather than being buried. The corpus lists apex hostnames, and we asked for the apex, so the apex to www redirect is partly an artefact of which URL we chose. It is also the URL a crawler most often meets, because a bare domain is how a domain gets written in prose, in a citation and in a directory entry, and it is what a model guesses when it constructs a URL from a brand name. A site that answers its canonical hostname instantly and its bare domain with a 301 has not made a mistake. It has made a choice whose price is one round trip, paid by every client that arrives at the name people actually write down.

This is also not the first redirect finding on this corpus, and the two are about different files. When we measured robots.txt redirects on 391 of these hostnames, the question was authority: RFC 9309 says to apply the fetched file to the original host, and 122 of 122 redirects crossed hostnames. Here the question is only time, and the answer is that half of it buys nothing. If you want to see what a single fetch of your own home page returns before any of this, what GPTBot sees runs the same request. The wider picture of what we are willing to publish about AI visibility across this corpus lives on the research page.

The fastest quarter of pages carried no more words than the slowest

The reason a speed post on this site has to carry a second measurement is that speed is easy to measure and easy to mistake for the thing that matters. So we counted, for every one of the 379 pages, how many words of extractable prose arrived in the bytes the server sent, with scripts, styles, comments, templates and noscript blocks removed and no JavaScript executed. That is the same shape of question our score calls prose parity, asked here without any scoring attached.

The two measurements do not line up. Sort the 379 by time to first byte and cut the list into quarters. The fastest quarter, everything from 0.141 to 0.278 seconds, carries a median 796 words. The second quarter carries 754, the third 734, and the slowest quarter, from 0.601 to 1.805 seconds, carries 945. That sequence is not an ordering. It runs down and then up, the whole sample's median is 796, and the only reading it supports is that knowing how fast a page answered tells you nothing useful about how much of it a crawler could read.

The individual cases make it concrete. The fastest page in the sample, oviedocommunitynews.org, delivered its first byte in 0.141 seconds and carried 796 words, which is a good outcome on both counts. kube-green.dev answered in 0.163 seconds and carried 63. At the other end listr.pro took 1.529 seconds to send 329,124 bytes of HTML containing four words of prose. Twenty four of the 379 pages carried fewer than 50 words, which is the same population we counted from a different angle when 17 of 380 home pages sent a crawler zero words three days earlier on this corpus.

The stratum figures say the same thing more bluntly. The static documentation sites were the fastest group in the sample at a median 0.237 seconds and had the fewest words at a median 333, because a documentation home page is mostly navigation. The Shopify storefronts were among the slowest at a median 0.531 seconds and carried the most at a median 1,134 words. A ranking by speed and a ranking by readable content are close to unrelated in this data, and only one of them is the thing an answer engine needs.

None of that makes speed worthless. It means speed is a crawl volume lever, exactly as Google's documentation frames it, and not a readability lever. The readability lever is whether the words are in the first response at all, which is a rendering question rather than a timing one: we have measured text in hydration JSON scoring the same as no text, and 28 Next.js home pages sending fifteen times more payload than prose. For a client rendered stack the fix is server rendering the prose, which is what the React fix guide is for, and it will usually make the page slower to first byte and far more readable. Optimising the first number would take you the wrong way on the second.

What the AI crawler operators publish about response time

We read the crawler documentation of three of the AI operators whose tokens this scanner evaluates, on 14 September 2026, and searched the text that arrived for every phrase a site owner would need: timeout, latency, response time, crawl rate, crawl delay.

OpenAI's bots page documents four agents, OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User, with an example user agent string and a published IP address file for each. It says nothing about how long the crawler waits, how often it returns, or what it does when a server is slow. Perplexity's crawler guide documents PerplexityBot and Perplexity-User and is silent on the same points.

Anthropic is the exception and it is worth quoting precisely, because it is the only AI operator in this set that gives a site owner a control. Its crawler help article says that Anthropic aims for minimal disruption by being thoughtful about how quickly it crawls the same domains and respecting Crawl-delay where appropriate, and then states plainly: to limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt. It prints the example, a ClaudeBot group with Crawl-delay: 1.

Read carefully, that is a rate control and not a speed threshold. It lets a server under load tell ClaudeBot to space its requests out. It says nothing about how long the crawler waits for any single response, and Crawl-delay is not in RFC 9309, which is why Anthropic calls it non standard in its own sentence. We have written about the same directive causing trouble in the other direction, where a crawler can be unverified over a directive the standard never defined.

So the state of the published record on 14 September 2026 is one operator documenting that response times govern crawl volume, one documenting a rate limit a site can request, and two documenting neither. That is the same distribution we found when no crawler vendor documented Retry-After, and it is why the honest answer to the question in the title has to be split by vendor rather than given once.

What to check on your own site, in the order that pays

The measurement suggests an order, and it is not the order most speed advice uses. Start with whether the crawler has to ask twice, because that is the one item here with a measured cost of about half the wait and a fix that changes nothing a reader sees.

Ask your own bare domain over HTTPS with redirects switched off and look at the first status line. If it is a 301 or a 308 to the www hostname, you are paying a round trip on every arrival at the name people write down, and the choice is either to serve the page on the bare domain or to accept the cost knowingly. Either is defensible. Not knowing which one you have is the part worth fixing, and it takes one request to find out. If the first response is a 202 with a tiny body, or a 403, you have a bot defence answering rather than a server, and the timing question is moot until that is settled: check what your file actually permits with the robots.txt tester and compare it against the tokens each vendor really publishes in our AI crawler reference.

Second, count the words in the first response rather than in the browser. This is the measurement that moved by a factor of hundreds across our sample while the timings moved by a factor of ten, and it is the one that decides whether there is anything for an answer engine to quote. A page that arrives in 0.16 seconds carrying 63 words has an excellent score on the metric nobody grades it on. The practical steps for getting cited by ChatGPT all assume the words reached the fetcher in the first place.

Third, and only third, look at the server itself. Google's documentation is clear that consistently longer response times reduce how much it crawls, so this is real, and for a large site with frequent changes it is worth engineering effort. For a marketing site of a few dozen pages whose home page already answers in half a second, it is not where the next unit of AI visibility comes from. The aggregate picture we are prepared to publish across this corpus sits on the research page, and it consistently puts readability ahead of speed.

The last item is the one this post cannot settle. No AI crawler operator publishes a response time threshold, so nobody outside those companies can tell you how slow is too slow for GPTBot or ClaudeBot. We measured 379 real home pages and the slowest took 1.805 seconds, which is almost certainly fine everywhere, but that is an inference from the absence of any published limit rather than a finding. If an operator publishes one, it will change this post's third item and none of the first two.