BlogFindings
Reduce crawl rate: Google now documents Retry-After, and 0 of 63 sites that pushed back sent it
Google added Retry-After to its reduce crawl rate documentation on 6 October 2026, and when Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for their home page once on 22 October 2026, 63 of the 1,388 that answered returned one of the three status codes that page names and not one of them carried the header.
So the question this post asks is what happens on the other side of the instruction. A response header is only worth documenting if servers send it, and that is measurable without asking any vendor anything. Lantad requested the home page of all 1,419 hostnames in this repository's two committed corpus seed files once each on 22 October 2026, as an identified AI crawler with no JavaScript and no browser, and recorded the status line and the response headers exactly as they arrived. Sixty three of the hosts that answered pushed back with one of the three codes Google's page names. None of the sixty three said when to come back. Four responses in the entire crawl carried Retry-After, every one of them on a status code the new guidance does not cover, and two of those four carried a value the grammar in the specification does not permit.
In short
- Google's crawling documentation changelog, read on 22 October 2026, carries an entry dated 6 October 2026 reading Added Retry-After HTTP header details to the reduce crawl rate documentation, and that page now carries Last updated 2026-10-06 UTC.
- The reduce crawl rate page tells site owners to return 500, 503 or 429 to shed crawler load, and offers Retry-After on two of those three codes: it is described for 503 and 429 and not for 500.
- Lantad requested the home page of each of the 1,419 hostnames in this repository's two committed corpus seed files once on 22 October 2026 as LantadBot with no JavaScript; 1,388 answered, and 63 of them returned 503, 429 or 500.
- Not one of those 63 responses carried a Retry-After header, and a second request to all 63 returned the same status code on 61 of them and still no header.
- Four of the 1,388 responses carried Retry-After at all, none of them on a 500, 503 or 429, and two of the four carried the value 28800, 28800 where RFC 9110 allows a single HTTP-date or a single count of seconds.
| What was counted | Sites |
|---|---|
| Hostnames asked | 1,419 |
| Returned an HTTP status | 1,388 |
| No response inside 20 seconds | 31 |
| Returned 503, 429 or 500, the three codes Google names | 63 |
| Of those 63, carried a Retry-After header | 0 |
| Carried Retry-After on any status code | 4 |
What Google added to its crawl rate page on 6 October 2026
Google's crawling documentation changelog, read on 22 October 2026, carries one entry for October. It is dated 6 October 2026 and reads: Added Retry-After HTTP header details to the reduce crawl rate documentation. The page it refers to carries Last updated 2026-10-06 UTC, which agrees with the changelog, and the two dates agreeing is worth saying out loud on a site that has had to report crawler files whose own dates disagree.
The new text sits inside the section on urgently reducing crawler traffic. That section already told site owners to return a 500, 503 or 429 status code instead of a 200 to shed load for a short period, which the page describes as a couple of hours, or 1 to 2 days. What is new is the sentence after it: when returning a 503 or 429 status code, the page says, you can also include a Retry-After HTTP header, as defined in RFC 9110 HTTP Semantics, to indicate when Google's crawlers can retry the request, using either a delay in seconds or an absolute UTC date and time. Two examples follow, one reading Retry-After: 120 on a 503 and one reading Retry-After: Wed, 21 Oct 2026 07:28:00 GMT.
Two details in that sentence are worth holding onto, because both are visible in the page itself rather than inferred. The first is that the header is offered for two of the three codes the same page recommends: 503 and 429 are named, and 500 is not. The second is the verb. The page says the header indicates when Google's crawlers can retry, which states what the site is telling Google rather than what Google then does. Nothing on the page promises the value is honoured, and this is the same gap that showed up when general crawl budget advice was read against AI crawlers. Google's other relevant page, the one on HTTP status codes and network errors, still carries Last updated 2026-02-04 UTC and contains no occurrence of the string Retry-After at all.
| Page | Last updated | Names Retry-After | Read on |
|---|---|---|---|
| Reduce the Google crawl rate | 2026-10-06 UTC | Yes, for 503 and 429 | 22 Oct 2026 |
| HTTP status codes and network errors | 2026-02-04 UTC | No | 22 Oct 2026 |
| Reduce the Google crawl rate, as read in August | 2025-12-18 UTC | No | 14 Aug 2026 |
How do you reduce crawl rate for an AI crawler?
The documented answer differs sharply by vendor, and only one vendor documents a response header at all. Google's page is the detailed one, and everything in it is about Google's own crawlers: shed load with a 500, 503 or 429, optionally name a retry time on the 503 or the 429, expect the rate to fall across the whole hostname rather than on the failing URLs alone, and expect it to climb back once the errors stop. The page also warns in its own voice that the effect is broad, that Googlebot will discover fewer new pages and refresh existing ones less often, and that if the same URL answers with those codes for multiple days it may be dropped from the index.
For the other three vendors in the August set, the documented levers are narrower. Anthropic's crawler help article, read on 22 October 2026, offers the non-standard Crawl-delay extension to robots.txt and says the bots respect Crawl-delay where appropriate, which is a request-rate instruction in a file rather than a response header. OpenAI's crawler overview and Perplexity's crawler guide, both read the same day, name no HTTP status code and no Retry-After in their crawler guidance at all. Perplexity's page does document firewall configuration, with named sections on Cloudflare and AWS rules, but that is guidance on letting the crawler through rather than on asking it to slow down. So for two of the four vendors the published choice is to allow a crawler or to block it, with no documented way to ask for a lower rate.
That asymmetry is the practical answer to the question in this heading, and it is not a comfortable one. A site that wants to slow an AI crawler without disappearing from it has one documented route with Google, a robots.txt extension with Anthropic, and no documented route with the other two. The blunt instrument, a Disallow rule, is the one every crawler reads, and it does not reduce a rate, it removes the host. Checking which rule a given token actually matches before relying on either is what the robots.txt tester is for, because the difference between a throttle and a removal is the difference between being read less often and not being read.
Flow: Crawler traffic too high to Return 500, 503 or 429; Return 500, 503 or 429 (optional) to Retry-After on 503 or 429; Retry-After on 503 or 429 to Rate falls across the host; Rate falls across the host (errors stop) to Rate recovers as errors stop; Rate falls across the host (if sustained) to Beyond 1 to 2 days URLs may be dropped; Crawler traffic too high (if errors are not feasible) to File a request with Google.
Retry-After is defined for 503 and 3xx, and 429 came from a different document
Google's new sentence cites RFC 9110 for the header, and reading that document directly is worth the two minutes because it names a slightly different set of responses than the Google page does. Section 10.2.3 of RFC 9110 defines Retry-After as the field servers send to indicate how long the user agent ought to wait before making a follow-up request. It then names two cases: sent with a 503, it indicates how long the service is expected to be unavailable, and sent with any 3xx response, it indicates the minimum time to wait before issuing the redirected request. The value is either an HTTP-date or a count of delay seconds, and the grammar is explicit that delay seconds is a non-negative decimal integer. Read on 22 October 2026, section 10.2.3 does not mention 429 anywhere.
That pairing lives in the older document. RFC 6585, published in April 2012, is the one that defines 429 Too Many Requests, and it states in the same section that the response may include a Retry-After header indicating how long to wait before making a new request, with a worked example reading Retry-After: 3600. So the combination Google now recommends for a 429 is specified, and it is specified somewhere other than the document Google cites for it. That is a bibliographic point rather than an error: both documents are current, 6585 updates the older HTTP specification rather than being replaced by 9110, and a site owner following the Google page would write a header that both documents support.
The reason to be precise about it is that status codes carry different meanings to different readers, and a site that picks one for a crawler is making a claim about itself. A 503 asserts a temporary condition. A 429 asserts the client asked too often. Neither asserts that the client is unwelcome, which is a distinction this blog has had to make before when a billing mechanism turned out to rest on a status code with no definition behind it. The specification gives a server one field for saying how long the condition lasts, and that field is the whole subject of this post.
| Document | Read section | Pairs a retry time with |
|---|---|---|
| RFC 9110, HTTP Semantics | 10.2.3, Retry-After | 503 and any 3xx, and 429 is not mentioned |
| RFC 6585, April 2012 | 4, 429 Too Many Requests | 429, which may include Retry-After |
| Google, reduce the crawl rate | Urgently reduce crawler traffic | 503 and 429, and not the 500 it also names |
What 1,419 sites sent when an AI crawler asked once
The measurement is deliberately narrow. Each of the 1,419 hostnames in the two committed corpus seed files, 392 across ten platform strata and 1,027 across eight industry strata, was asked for its home page exactly once on 22 October 2026. Every request went out from one network location as LantadBot/1.0, the token described on the bot page, with redirects followed, a twenty second timeout, no JavaScript executed and no browser involved, which is the same posture the rest of the corpus work in the methodology uses. The only fields recorded were the final status code and the response headers.
Of the 1,419, some 1,388 returned an HTTP status and 31 returned nothing: 22 connection failures and 9 timeouts. Among the 1,388, the bulk answered normally, with 1,082 returning a 200. The next largest group was refusal rather than overload: 224 returned 403, a code that says the request is forbidden and that neither Google's page nor section 10.2.3 of RFC 9110 associates with a retry time. Then came the three codes Google's new text is about. Fifty six returned 503, six returned 429 and one returned 500, which is 63 hosts in total, or about one in twenty two of those that answered.
Not one of those 63 responses carried a Retry-After header. That is the finding, and it is a clean zero rather than a small number. The distribution of the 63 is lopsided in a way worth recording: 62 of them sit in the industry file and exactly one in the platform file, so pushback of this kind is a behaviour of large institutions with infrastructure in front of them rather than of small sites on hosted builders. Travel led with 12 of its 124 answering hosts, then education with 11 of 130, finance with 9 of 123, government with 8 of 118, healthcare with 8 of 120, ecommerce with 8 of 125, news with 4 of 127 and SaaS with 2 of 130. Nine of the ten platform strata produced none at all, which is a different shape from the one that appears when industry blocking is read out of robots.txt and a reminder that a status code and a robots rule are opposite kinds of signal.
Was the 503 about load, or about who was asking?
A 503 is a claim about the server, not about the client, so the obvious question is whether these hosts were genuinely shedding load at the moment they were asked. Two follow-up requests to each of the 63 answer it partly. Asked a second time with the same crawler token, 61 of the 63 returned the same status code, so these are stable responses rather than a transient blip caught once. Two recovered and returned a 200 on the second attempt. On the second pass, as on the first, none of the 63 carried a Retry-After header.
Asked with a current desktop Chrome user agent string instead, the same 63 hosts split. Fifty one returned the same code they had given the crawler, three returned a 403 and nine returned a 200. Those nine are the interesting ones: cbc.ca, npr.org, cathaypacific.com, premierinn.com, morganstanley.com, scandichotels.com, usaa.com, turkiye.gov.tr and polimi.it all served a page to a browser string while answering the identified crawler with a code that means the service is unavailable. For those nine the response was about the client rather than about capacity, which makes the missing Retry-After less of an oversight and more of a category question: a server that is not actually overloaded has no retry time to give.
Two cautions belong on that reading. The first is that a user agent string is a claim and nothing more, which is the whole argument of an earlier post on why a user agent is a claim rather than an identity, so this comparison measures what two strings were served and not what any real browser or crawler would receive. The second is that the majority went the other way: 51 of the 63 refused both strings equally, and a blanket 503 from an edge network looks the same whoever asks. That pattern is close to what appeared when CDN behaviour was measured against an unknown bot, and seeing which of the two a given host does to a specific token is what the GPTBot view is for. Either way the consequence for AI visibility is the same: a crawler that receives a 503 with no retry time has nothing to schedule against.
Asked again as LantadBot
- 61 of 63 returned the same status code
- 2 of 63 returned 200 instead
- 0 of 63 carried a Retry-After header
- 54 answered 503, 6 answered 429, 1 answered 500
Asked with a Chrome user agent string
- 51 of 63 returned the same code as before
- 9 of 63 returned 200 and served a page
- 3 of 63 returned 403 instead
- 1 answered 500, as it had to the crawler
The four Retry-After headers in 1,388 responses
Four responses in the whole crawl carried the header, and none of them was a 500, a 503 or a 429. One was a 406 from aucklandcouncil.govt.nz carrying the value 0. One was a 403 from bloomberg.com, also carrying 0. The remaining two carried the value 28800, 28800: a 403 from marriott.com and, oddly, a 200 from citizenm.com, which served a page and attached a retry time to it. All four were identical on a second request and identical again when asked with a browser user agent string, so none of them is a momentary artefact.
None of the four is usable in the way Google's new text describes, and the reasons differ. A Retry-After of 0 is a well formed value under the grammar in RFC 9110, since zero is a non-negative integer, but on a 403 or a 406 it is attached to a response whose meaning is not temporary, so there is nothing for the retry time to be about. A value arriving as 28800, 28800 is two values in a field the specification allows a single HTTP-date or a single count of seconds, so a client reading it strictly has nothing well formed to parse. And Retry-After on a 200 is advice attached to a success, which no part of either document gives a meaning to. The honest summary is that the header was present four times and informative zero times, which is a pattern this blog has recorded before with status codes that carry the wrong meaning for the body they arrive with.
What this measurement does not show is worth stating as plainly as what it does. No host was asked repeatedly or quickly enough to trigger rate limiting, so these are the responses a first contact crawler receives and not what any of these sites does to a crawler that hammers it. A site that correctly sends Retry-After under sustained load would appear in this data as a plain 200, and nothing here would reveal it. The measurement also says nothing about whether any crawler honours the header: that would need a server under observation and a crawler returning to it, which is a different study from the one page per host crawl this corpus supports. The documentation gap this post opened with has narrowed by one page at one vendor. The deployment gap, on this sample and on this single request, has not started to close.
-
aucklandcouncil.govt.nz406, Retry-After: 0 Well formed value on a status code whose meaning is not temporary -
bloomberg.com403, Retry-After: 0 Refusal rather than overload, so the retry time has no subject -
marriott.com403, Retry-After: 28800, 28800 Two values where the grammar allows one date or one count of seconds -
citizenm.com200, Retry-After: 28800, 28800 A retry time attached to a successful response, which no specification defines
Lantad
Published .
Google changed its advice on how to reduce crawl rate on 6 October 2026. The page now tells a site owner that a 503 or a 429 response may carry a Retry-After header naming either a delay in seconds or an absolute UTC date and time, and it prints two worked examples of the syntax. That is worth recording here because this blog published the opposite finding on 14 August 2026: six crawler documentation pages from four vendors named Retry-After nowhere, and one of the six was this exact Google page, which carried a Last updated date of 18 December 2025 when it was read. One vendor has now closed that gap on one of its two relevant pages.
Common questions
How do you reduce crawl rate for Google's crawlers?
Google's reduce crawl rate page, carrying Last updated 2026-10-06 UTC, says to return a 500, 503 or 429 status code instead of a 200, and that on a 503 or a 429 you can also include a Retry-After header giving either a delay in seconds or an absolute UTC date and time. The same page warns that the effect applies across the whole hostname, that fewer new pages will be discovered, and that URLs answering with those codes for multiple days may be dropped from the index.
Does Retry-After work on a 429 as well as a 503?
Both documents support it, though not in the same place. Section 10.2.3 of RFC 9110 defines Retry-After for 503 and for any 3xx response and does not mention 429. RFC 6585, published in April 2012, is the document that defines 429 Too Many Requests, and it states that such a response may include a Retry-After header, with the example Retry-After: 3600.
Do OpenAI, Anthropic and Perplexity document a way to slow their crawlers?
Read on 22 October 2026, Anthropic's crawler help article offers the non-standard Crawl-delay extension to robots.txt and says its bots respect Crawl-delay where appropriate. OpenAI's crawler overview and Perplexity's crawler guide name no HTTP status code and no Retry-After in their crawler guidance. None of the three documents a response header for the purpose.
How many sites actually send Retry-After to an AI crawler?
In this measurement, four of 1,388 responses carried the header and none of those four was a 500, 503 or 429. Of the 63 hosts that did return one of those three codes to a single LantadBot request on 22 October 2026, zero carried Retry-After, and a second request to all 63 produced the same result.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
Retry-After names when to come back, and four crawler vendors never document it
HTTP defines a response header that lets an overloaded server tell a client exactly when to retry. Read on 14 August 2026, six crawler documentation pages published by Google, OpenAI, Anthropic and Perplexity describe what a server error does to crawling and none of the six names Retry-After anywhere.
A 404 on robots.txt allows every crawler, a 503 blocks them all
RFC 9309 gives a missing robots.txt and a broken robots.txt opposite meanings: a 4xx lets a crawler access any resource on the server, a 5xx obliges it to assume complete disallow. Google's documentation carves 429 out of that rule. Lantad's code does not.
CDN blocking AI crawlers: 103 of 1,089 sites served an unknown bot and refused GPTBot
Lantad requested the home page of 1,419 hostnames three times each on 26 September 2026, changing nothing between the requests but the user agent string. 1,089 answered a browser with HTTP 200. 103 of those served an unknown bot user agent as well and then refused the GPTBot string, and on 76 of the 103 the site's own robots.txt permits GPTBot at the root. Sorting the refusals by the infrastructure fingerprint on the response puts 32 of 87 Akamai fronted sites in the refused column against 37 of 344 on Cloudflare.
