BlogFindings

Retry-After names when to come back, and four crawler vendors never document it

HTTP defines a response header that lets an overloaded server tell a client exactly when to retry. Read on 14 August 2026, six crawler documentation pages published by Google, OpenAI, Anthropic and Perplexity describe what a server error does to crawling and none of the six names Retry-After anywhere.

14 August 2026 16 min read Lantad

This post is a reading of published documents, not a crawl. Six pages were fetched and searched directly on 14 August 2026, one to three from each of four vendors, and the counts below are counts of what those pages say rather than observations of how any crawler behaved on any server. Lantad ran no experiment for it and holds no measurement of retry timing by any bot. The reason the question belongs here is that it decides something a site owner has to plan around: when a server sheds load, the operator wants to know how long the shed costs, and the only two places that answer could come from are a documented header or a documented recovery interval. One of those turns out to be missing everywhere.

In short

RFC 9110 section 10.2.3 defines the Retry-After response header as the way a server indicates how long a user agent ought to wait before making a follow-up request, with a value that is either an HTTP-date or a count of seconds, and it names 503 and 3xx as the responses it belongs on.
RFC 6585, published in April 2012, defines 429 Too Many Requests and states in the same section that the response MAY include a Retry-After header indicating how long to wait before making a new request, illustrated with the example Retry-After: 3600.
Google's HTTP status codes page, carrying Last updated 2026-02-04 UTC, and its reduce crawl rate page, carrying Last updated 2025-12-18 UTC, both describe crawler behaviour on 429 and 5xx in detail, and read on 14 August 2026 neither page contains the string Retry-After.
The crawler documentation published by OpenAI and Perplexity, read on 14 August 2026, names no HTTP status code at all in its crawler guidance, and Anthropic's crawler article, carrying a dateModified of 2026-04-07, offers the non-standard robots.txt Crawl-delay extension rather than any response header.
Lantad measured none of the vendor behaviour above. Its one relevant stored capture is allbirds.com on 15 July 2026, which answered the identified crawler with HTTP 429 and produced a null composite and a null grade.

Publisher	Page	Describes server errors	Names Retry-After
Google	HTTP status codes, updated 2026-02-04	Yes, in detail	No
Google	Reduce the crawl rate, updated 2025-12-18	Yes, recommends them	No
Google	robots.txt specification	Yes, for robots.txt only	No
OpenAI	Crawler overview	No status code named	No
Anthropic	Crawler support article, modified 2026-04-07	No, offers Crawl-delay	No
Perplexity	Perplexity Crawlers	No status code named	No

Six crawler documentation pages fetched and string-searched on 14 August 2026. The right-hand column is a count of occurrences in the page text, not a claim about crawler behaviour.

What Retry-After specifies, and which responses it belongs on

The definition is short enough to quote in full and worth quoting because the wording is more modest than most people remember. Section 10.2.3 of the HTTP semantics specification states that servers send the Retry-After header field to indicate how long the user agent ought to wait before making a follow-up request. It then splits by status. Sent with a 503 Service Unavailable response, it indicates how long the service is expected to be unavailable to the client. Sent with any 3xx redirection response, it indicates the minimum time the user agent is asked to wait before issuing the redirected request. The value is either an HTTP-date or a count of delay seconds, so both "Retry-After: Fri, 31 Dec 1999 23:59:59 GMT" and "Retry-After: 120" are legal, and the specification notes that in the second example the delay is two minutes.

Two things in that definition matter for anyone deciding whether to set the header. The first is that it is advisory in the strict sense: the client ought to wait, and nothing in the specification obliges it to. The second is what the section does not cover. It names 503 and 3xx, and it does not name 429, because 429 is not defined in that document at all. That status comes from RFC 6585, published in April 2012, which added a handful of status codes to HTTP and defined 429 Too Many Requests as indicating that the user has sent too many requests in a given amount of time. Its own text says the response representations should include details explaining the condition and may include a Retry-After header indicating how long to wait before making a new request, and it prints a worked example whose header line reads Retry-After: 3600.

So the pairing site owners assume is real, and both halves were standardised long before any of the crawlers in question existed. That is what makes the documentation result in the next two sections a finding rather than a curiosity: the mechanism is not new, not contested and not vendor specific.

One separation is worth making, because the two questions get merged. Whether a crawler can read your page at all is a matter of robots.txt permission and of what the server returns to a non-browser client, which is the thing a what GPTBot sees check exercises. Retry-After does not touch permission. It assumes the crawler is allowed and the server is simply unable right now, and it addresses only the timing of the next attempt. A site can be perfectly configured for AI visibility and still lose a day of crawling because nobody agreed on how long a pause should last.

Sample Illustrative, not a measurement of any real site.

GET /product/example, client over the limit

Server decides the client has exceeded its allowance 429 Too Many Requests
Header the specification says the response MAY include Retry-After: 3600
What that value means wait 3600 seconds
Alternative legal form of the same field an HTTP-date
Obligation the specification places on the client none, it ought to wait
Effect on whether the page is allowed at all no effect, this is timing only

The shape of a rate limited response as RFC 6585 section 4 defines it, including the header line printed in the specification's own example. Constructed from the specification text, not captured from any real site.

Google documents a crawl rate, not a retry interval

Google is the only one of the four vendors that documents error handling in any depth, so it is the strongest test of the question. Its HTTP status codes reference, carrying Last updated 2026-02-04 UTC, states that Google's crawlers treat the 429 status code as a signal that the server is overloaded, and that it is considered a server error. On the 5xx family it states that 5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling, that for Google Search already indexed URLs are preserved in the index but eventually dropped, and that any content Google receives from URLs returning a 5xx status code is ignored. It closes the loop with recovery: once the server starts responding with a 2xx status code, Google gradually increases the crawl rate for the site. For 500, 502 and 503 specifically it adds that the decrease in crawl rate is proportionate to the number of individual URLs that are returning a server error.

That is a genuinely detailed model, and every quantity in it is a rate rather than an interval. The page describes a crawl rate going down and later coming back up as a function of how many URLs are erroring, and at no point does it hand the server a way to name the duration. Searched on 14 August 2026, the page does not contain the string Retry-After.

The companion page is more pointed still, because it is the one that actively recommends the technique. Google's guidance on how to reduce the crawl rate, carrying Last updated 2025-12-18 UTC, tells a site owner that if they need to urgently reduce the crawl rate for a short period, for example a couple of hours or one to two days, they should return a 500, 503 or 429 HTTP response status code instead of 200 to the crawl requests. It explains that the reduced crawl rate affects the whole hostname, both the URLs that return errors and the URLs that return content, and that once the number of errors is reduced the crawl rate will automatically start increasing again. It carries a warning against doing this for longer than one to two days, on the grounds that if Googlebot observes these status codes on the same URL for multiple days the URL may be dropped from the index. This page does not contain the string Retry-After either.

Read together, the pair is coherent. Google expresses duration through how long you keep serving the error rather than through a number in a header, and it publishes the consequences of overshooting. That is defensible, and it is also strictly less expressive than the header: a server can stop erroring the moment it recovers, but it cannot say in advance that recovery is twenty minutes away, so the crawl rate climbs back on Google's schedule rather than the operator's. The same bluntness runs through the advice covered in crawl budget advice ends in a Disallow line. A genuinely different case is what a status code on robots.txt itself means, where a 4xx and a 5xx have opposite effects on permission rather than on rate, set out in a 404 on robots.txt allows every crawler, a 503 blocks them all.

Response	Documented effect on crawling	Documented effect on the index	Duration named
429	Treated as server overload, a server error	Preserved, then eventually dropped	No
500, 502, 503	Crawl rate decreases in proportion to erroring URLs	Persistent errors removed from the index	No
Other 4xx	No effect on crawl rate	Removed from the index	Not applicable
2xx after errors	Crawl rate gradually increases again	Eligible for processing	No

What Google's HTTP status codes page states for each response, read on 14 August 2026 from the page carrying Last updated 2026-02-04 UTC. Quotations of documentation, not measurements of crawler traffic.

What the AI crawler documentation says about a server under load

The three AI vendors whose tokens turn up most often in the logs of sites this scanner is pointed at publish shorter pages, and the shortness is the result. OpenAI's crawler overview documents its tokens, their purposes and its published IP ranges, and tells webmasters to manage crawling through robots.txt. Searched on 14 August 2026 its crawler guidance names no HTTP status code at all: not 429, not 503, not 500. There is no sentence about what happens to crawling when a server errors, no sentence about recovery, and no mention of Retry-After.

Perplexity's crawler page is the same on this axis and interesting for what it does instead. It names PerplexityBot and Perplexity-User, publishes their IP endpoints, and devotes a section to web application firewall configuration, walking a site owner through writing a rule that allows the bots through by matching user agent and source address together. Its concern is a site accidentally blocking the crawler, which is a real and common failure. It says nothing about a site deliberately shedding load, names no status code, and does not mention Retry-After.

Anthropic's crawler article, carrying a dateModified of 2026-04-07, is the only one of the three that addresses crawl pacing at all, and it answers in a different file entirely. It states that the bots aim for minimal disruption by being thoughtful about how quickly they crawl the same domains and respecting Crawl-delay where appropriate, and that to limit crawling activity Anthropic supports the non-standard Crawl-delay extension to robots.txt, with a worked example setting a delay of 1 for ClaudeBot. That is a real answer to the pacing question and it is worth knowing about. It is also a robots.txt directive rather than a response header, which means it is a standing policy set in advance rather than something a server can say in the moment it is struggling, and the specification that standardised robots.txt never defined it. The consequences of leaning on that particular directive were covered here in Cloudflare can unverify a crawler over a directive RFC 9309 never defined.

A negative result like this needs the same reading as any documentation evidence. Absence from a published page is not proof that no system at these companies honours the header, and none of these pages claims to be an exhaustive list of what its crawler respects. What it establishes is the part a site owner can act on, because an operator can only follow instructions that exist. Across four vendors and six pages read on one day, the published instruction for slowing a crawler is a robots.txt directive in one case, a sustained error response in another, and nothing at all in the remaining two, while the header HTTP defines for this exact conversation is named by none. That thinness is a pattern rather than a one-off, matching the earlier count showing six of the nine vendors in the registry publish exactly one crawler token.

Google Serve errors Return 500, 503 or 429, for no longer than one to two days
Anthropic Crawl-delay Non-standard robots.txt extension, worked example sets 1
OpenAI Nothing stated Crawler guidance names no status code and no pacing control
Perplexity Nothing stated Documents allowing the bot through a firewall, not slowing it
Retry-After Named by none Zero occurrences across all six pages searched

What each vendor's crawler documentation offers a site owner who needs to slow a crawler down, read on 14 August 2026. A record of what the pages say, not a test of what the crawlers do.

What a 429 costs when the client is an AI crawler

The reason any of this is worth a site owner's attention is that an error served to an AI crawler is not a deferred fetch, it is an absent citation. A search crawler that comes back tomorrow loses a day of freshness on a page it already has. An answer engine asked a question today either has your page or it does not, and if it does not, the answer is assembled from whatever else it holds. The gap between those two failure modes is the whole reason this site treats access as a first-class thing to measure rather than as a precondition worth assuming, which is set out in the scan methodology.

Lantad has exactly one stored capture that speaks to this directly, and its limits should be stated before its content. It is a single page, on a single day, and it is a measurement of one site's response rather than of any crawler's reaction. On 15 July 2026 the scanner requested the homepage of allbirds.com while identifying itself honestly as LantadBot, and received HTTP 429. The stored fixture records a baseline status of 429, a content type that is not HTML, a rendered document of zero bytes and a raw body of eighteen characters. Every one of the fifteen crawler tokens in the matrix carries a probe verdict of not_probed, because there was no page to compare anything against.

What the report does with that is the part worth borrowing. The verdict is a state called access_fatal, and the composite score and the letter grade are both null. Structure, schema and parity are nulled too, on the reasoning that they would otherwise be computed against an error page and would describe the error rather than the site, which in one earlier case rendered green. The stored headline reads that crawlers cannot fetch this page and that everything else is moot until the URL serves content. That refusal to grade is the rule described in why we withhold a grade, and the same storefront sat alongside a competitor scoring 94.3 on the same platform on the same day, written up in what a crawler meets on a real storefront.

The connection back to the header is direct. That response carried a status the specifications pair with Retry-After, and whatever the server said about when to return, no published crawler documentation from the four vendors above commits to reading it. A site in that state is relying entirely on the crawler's own backoff schedule, which Google describes qualitatively and the others do not describe at all. It is also worth being clear about what a 429 does not damage: nothing about the page's markup, its structured data or its prose is wrong. The content is fine and unreachable, which is the least diagnosable kind of problem because every check that reads the page passes.

Page served, ordinary scan

Baseline status: 200, content type HTML
Composite and grade: computed
Crawler matrix: each token probed
Structure, schema, parity: measured
Report says: what to fix, ranked

HTTP 429, measured 15 July 2026

Baseline status: 429, content type not HTML
Composite and grade: both null
Crawler matrix: 15 tokens, all not_probed
Structure, schema, parity: all nulled
Report says: everything else is moot

The stored allbirds-rate-limited fixture, measured 15 July 2026, against what the same scan reports when a page is served. Fields read from the capture, not typed by hand.

What to check before you rate limit a crawler

Nothing above argues that a site should stop rate limiting. A server falling over serves nobody, and shedding load is the correct response to an emergency. The argument is narrower: the interval you think you are communicating is probably not being received, so the decision should be made on duration rather than on the header, and it should be made per crawler rather than in one blanket rule.

Start with what your edge is already doing without being asked, because that is where the surprises are. Rate limiting rules, bot management products and firewall defaults frequently return 429 or 403 to clients that are not browsers, and the operator who would have chosen differently often does not know the rule exists. That is the mechanism behind the finding that headless browsers draw refusals at a meaningful rate on ordinary sites, reported here in headless Chromium was blocked on 15.2 percent of the top 10,000 sites. The check is to request your own page as a plain client with a crawler user agent and read the status, then repeat it from an address outside your own network.

Then separate the two decisions that a blanket rule merges. Permission is a robots.txt question, evaluated per token, and it is the thing a robots.txt tester answers. Pacing is a rate question, and the vendors answer it in different files: Anthropic in robots.txt, Google through sustained error responses with a warning attached about how long is too long, OpenAI and Perplexity not at all. A site that wants a specific crawler to slow down rather than go away should set the mechanism that crawler's own vendor documents, and should not assume a Retry-After value is doing work.

Set the header anyway. It costs nothing, it is correct behaviour toward every other HTTP client including the ordinary libraries and monitoring systems that do honour it, and no evidence here says any crawler ignores it. The claim is only that none of the four vendors documents reading it, which is a statement about what you have been told rather than about what happens. Publishing what a bot is and what it does is the same standard this scanner holds itself to on its own bot page, and the dated write-ups behind claims like the ones in this post are collected on the research page. If the answer to how long your last outage cost you in AI answers is that nobody can say, that is the honest answer, and it is the one this post ends on.

Flow: Crawler is causing load to Permission or pacing?; Permission or pacing? (permission) to Disallow in robots.txt; Permission or pacing? (pacing) to Crawl-delay, Anthropic only; Permission or pacing? (pacing) to Serve 429 or 503, Google path; Serve 429 or 503, Google path (past 1 to 2 days) to Dropped if sustained.

The decision a site owner is actually making when a crawler is causing load, given what the four vendors document. A description of the documented options, not a measurement of outcomes.

Written by

Lantad

Published 14 August 2026.

Every piece of operational advice about a site buckling under crawler load ends in the same instruction: return an error status and the crawlers will back off. That instruction is sound, and it leaves one question unanswered that HTTP answered decades ago. If the server knows the outage will last twenty minutes, or that the rate limit resets in an hour, it can say so in the response itself. The header for that is Retry-After, and the question this post asks is narrow and checkable: does any AI crawler vendor tell site owners it reads one. Retry-After is not an obscure extension. It is defined in RFC 9110, the core HTTP semantics specification, alongside Content-Type and Location.

Common questions

Do AI crawlers respect the Retry-After header?

Their published documentation does not say. Six crawler documentation pages from Google, OpenAI, Anthropic and Perplexity were fetched and string-searched on 14 August 2026, and none of them contains the string Retry-After. That is a statement about what site owners have been told, not proof that no crawler reads the header, and no test of actual crawler behaviour is reported here.

What status code should I return to slow down a crawler?

Google's guidance on reducing its crawl rate, carrying Last updated 2025-12-18 UTC, says to return 500, 503 or 429 instead of 200, and warns against doing so for longer than one to two days because a URL returning those codes on multiple days may be dropped from the index. Anthropic instead documents the non-standard Crawl-delay extension in robots.txt. OpenAI and Perplexity name no status code in their crawler documentation.

Is Retry-After part of the HTTP standard?

Yes. Section 10.2.3 of RFC 9110 defines it as the field a server sends to indicate how long a user agent ought to wait before making a follow-up request, with a value that is either an HTTP-date or a number of seconds. RFC 9110 names 503 and 3xx responses, and RFC 6585, which defines 429 Too Many Requests, states separately that a 429 response may include it.

Does returning 429 to an AI crawler hurt my AI visibility?

It prevents the fetch, so nothing on the page can be read while the error is being served. In Lantad's stored capture of allbirds.com on 15 July 2026 an HTTP 429 produced a null composite score and a null grade, with every crawler token unprobed, because no page arrived. How long the effect persists after recovery is not something this post measures for any AI engine.

See what AI can read on your site

Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.

Scan your site free How we measure

In short

What Retry-After specifies, and which responses it belongs on

Google documents a crawl rate, not a retry interval

What the AI crawler documentation says about a server under load

What a 429 costs when the client is an AI crawler

What to check before you rate limit a crawler

Common questions

See what AI can read on your site

Related

A 404 on robots.txt allows every crawler, a 503 blocks them all

Cloudflare can unverify a crawler over a directive RFC 9309 never defined

Why we will not grade a page we could not measure

AI crawler

Crawl budget advice ends in a Disallow line, and AI crawlers read it too

AI visibility