Crawl delay robots txt: 157 of 1,014 files set one, and 11 named an AI crawler

Crawl-delay is not in the standard it is written into

Three documents can reasonably be called the specification for this file, and the directive is in none of them.

RFC 9309, Robots Exclusion Protocol, Standards Track, September 2022, is the one that made the file a standard rather than a convention. Lantad downloaded its 25,265 byte text on 20 September 2026 and searched it case insensitively for crawl-delay, crawl delay and Crawl-Delay. The string appears zero times. The formal grammar in section 2.2 defines a user-agent line and allow and disallow rules and nothing else, and the note beside it tells implementers to define any further lines they need, naming sitemaps as the example. Crawl-delay is exactly such a further line, defined by nobody in particular.

The Web Robots Pages description of /robots.txt is the document the RFC lists as its normative reference for the protocol it extends, dated 2007 in that reference list, and the page itself points back to the original 1994 Standard for Robot Exclusion. Fetched on 20 September 2026, it discusses User-agent and Disallow and contains no occurrence of crawl-delay either.

Google's robots.txt documentation, carrying Last updated 2026-08-31 UTC when read on 20 September 2026, is the most direct of the three. In a parenthesis attached to the list of fields it accepts, it says Google supports user-agent, allow, disallow and sitemap, and that other fields such as crawl-delay are not supported. That is the largest crawler on the web saying in one clause that the line changes nothing when it arrives.

None of this makes writing the directive wrong. A de facto extension is still a signal, and the robots.txt tester on this site parses the file the same way the scanner does, so it will show you the same thing: a Crawl-delay line changes no access verdict, because the only verdicts robots.txt produces are allowed and disallowed. What it does mean is that a site owner writing the line has no specification to appeal to when a crawler ignores it, and that the useful question is per vendor rather than general.

How many robots.txt files set a Crawl-delay?

The request was one GET of https://host/robots.txt per hostname, sent as LantadBot/1.0 with redirects followed, a twenty second timeout, no JavaScript executed and from a single network location. The corpus is the 1,419 hostnames in this repository's committed seed files, an editorial sampling frame of large organisations and platform grouped sites rather than a random draw from the web.

Of the 1,419, 58 were never reached, because the network this scan ran from refused the connection at its own egress boundary before it left. That is a limit of the environment rather than a property of those sites, and it is not evenly spread: 55 of the 58 sit in the news category, so news lost 55 of its 128 hostnames and is under-represented in everything below. A further 30 failed at the transport layer. Of the 1,331 that returned a status, 1,095 answered HTTP 200, and 33 of those 1,095 sent an HTML page rather than a robots file, a failure mode counted in its own right when we checked 1,059 files against the parsing rules. That leaves 1,062 textual responses and 1,014 that parsed into at least one user-agent group, which is the denominator for every figure here.

157 of those 1,014 files carry at least one Crawl-delay line, and between them they carry 328 such lines. 792 of the 1,014 use only the four fields Google accepts and nothing else.

The values cluster hard. 157 of the 328 lines say exactly 10, which is nearly four times the next most common value. 41 lines say 1, another 41 say 5, 24 say 3 and 16 say 2. At the other end, 18 lines ask for 60 seconds or more and five of those ask for 300 or more. The largest single value in the corpus is 1,000. Two lines say 0, which asks for no delay and is a way of writing nothing.

Prevalence tracks how the file was written rather than how much traffic the site takes. The directive appears on 34.3 percent of the WordPress grouped hosts and 33.3 percent of the Wix and Squarespace grouped hosts, against 2.9 percent of the single page app startups, 3.2 percent of the SaaS marketing sites and none at all of the 29 Framer hosts or the 16 static documentation sites. Those are the numbers of an inherited template rather than of a rate somebody chose, and platform defaults deciding the contents of this file is a pattern this blog keeps meeting: it is the same mechanism behind the 89 of 145 files that ruled only on the whole site.

What does crawl delay robots txt do to an AI crawler?

Since no specification answers this, the only honest answer is to read what each operator publishes about its own crawler. Lantad opened the nine vendor documentation pages named in this scanner's crawler registry on 20 September 2026 and searched each for the string crawl-delay. Eight of the nine returned a page. Meta's crawler page answered HTTP 400 to this client and was not read, so Meta is unread here rather than silent, and no claim is made about it.

Three of the eight pages name the directive, and only one of the three honours it.

Anthropic's help centre article on its crawlers, updated 7 April 2026, states that to limit crawling activity it supports the non-standard Crawl-delay extension to robots.txt, and prints a worked example of a group naming ClaudeBot with Crawl-delay set to 1. Elsewhere on the same page it says its crawling aims for minimal disruption by being thoughtful about how quickly it crawls the same domains and respecting Crawl-delay where appropriate. Those two sentences are not equally strong and the second is the weaker, but the directive is documented and named. The same page's habits around identity are the subject of the one IP list Anthropic publishes for three separate bots.

Apple's About Applebot page, published 4 September 2026, says the opposite in one sentence: Applebot does not follow crawl-delay. That page is at support.apple.com/en-us/119829, written here as plain text because the host is not on this site's registered outbound list. Apple's habit of deferring to another vendor's group instead is measured in the 64 files that hand Applebot to their Googlebot rules.

Amazon's About AmazonBot page is equally plain. After listing what the crawler does respect, including nofollow and the page level noarchive and noindex meta tags, it states that they do not support the crawl-delay directive.

The remaining five say nothing at all. OpenAI's crawler documentation, Perplexity's bots guide, Common Crawl's CCBot page, Google's crawler overview page and ByteDance's Bytespider page each contain the string zero times. Silence is not refusal, and it is not consent either. It is the same gap this blog found when six crawler documentation pages never named Retry-After, the header HTTP actually defines for telling a client when to come back. The current token list, with each vendor's documentation page beside it, is in the crawler directory.

The tokens carrying a Crawl-delay are not AI crawler tokens

A Crawl-delay line applies to the group it sits in, and nothing else. So the question that decides whether any of this reaches an AI crawler is which user-agent line the delay was written under.

Attributing each line needed the same grouping rule the scanner's parser applies: consecutive user-agent lines extend one group, a user-agent line after a rule starts a new one, and an unrecognised field does not break a run of tokens. The parser itself records Crawl-delay nowhere, because it cannot change an access verdict, so this pass read the raw lines and resolved each one's group by that rule.

Counted per file, 11 of the 157 put a Crawl-delay inside a group naming one of the 15 AI crawler tokens this scanner evaluates. 57 put it only in the wildcard group. 71 put it only in groups naming some other crawler entirely. Eight files wrote a Crawl-delay before any user-agent line at all, where it belongs to no group and has nothing to apply to.

Counted per line, and counting a merged group once for each token it names, the wildcard carries 73 of the 328 lines and the next five are AhrefsBot at 51, dotbot at 46, bingbot at 30, MJ12bot at 27 and Slurp at 21. Not one of those five is among the 15 AI crawler tokens. GPTBot appears in 14 of the 328 lines and ClaudeBot in 11, and ClaudeBot is the only AI token in the list whose vendor documents supporting the directive. Claude-User, the token Anthropic documents for a page fetched at a person's direct request, appears in none of the 328.

The wildcard figure is the one to be careful with, because it is the group every crawler with no group of its own falls back to, and that is most of them. 206 of the 1,014 files name at least one of the 15 AI tokens. 45 of those 206 also set a Crawl-delay somewhere in the file, but only 11 put it in the AI group, which means 34 sites wrote both a rule for an AI crawler and a rate limit, and left the rate limit where the named crawler will never read it. Under RFC 9309 a crawler that matches a named group reads only that group, a mechanism this blog has now measured twice from the access side: once when 559 of the 581 pages GPTBot lost were closed by a rule that never named it, and once when a file named the wildcard rather than the AI crawler. The consequence runs both ways. A named group inherits nothing from the wildcard, and the delay is part of what it does not inherit.

One file asked ClaudeBot and GPTBot to wait 1,000 seconds

The largest value in the corpus is worth reading in full, because it shows what the directive is being asked to carry.

dtu.dk, the Technical University of Denmark, serves an 1,855 byte robots.txt with four groups. The first names the wildcard, the second ClaudeBot, the third GPTBot and the fourth GoogleOther. All four carry the identical list of sixteen Disallow paths and an identical Sitemap line, copied verbatim, so the file declares four sitemaps that are the same sitemap. The only difference between the groups is the last line of each: the wildcard gets Crawl-delay 10, and ClaudeBot, GPTBot and GoogleOther each get Crawl-delay 1,000.

A thousand seconds between requests is 86 requests a day. Read as an intention it is clear and reasonable: the site wants those crawlers present but slow. Read against the documentation it splits in half. Anthropic's page says it supports the directive, so the ClaudeBot group has a documented chance of doing something. OpenAI's page does not mention it, so the GPTBot group is a request with nothing published behind it, sitting in a group that is otherwise an exact copy of the wildcard rules. The site has written two lines that look identical and are not.

github.com puts the same directive the other way round, and that ordering is the point. Its 6,397 byte file opens with a single group naming GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot together, then writes 72 Disallow rules and 12 Allow rules inside it, and sets Crawl-delay 1. The access decision is carried entirely by the Allow and Disallow lines, which every one of those five crawlers is documented to read, and the delay is a hint layered on top rather than the mechanism. Six further groups follow, and the wildcard is the last of them, merged with a group naming baidu.

If the goal is genuinely to reduce load rather than to express a preference, the specified lever is elsewhere. HTTP defines status codes and headers for an overloaded server, and how much of a site actually gets fetched is decided by machinery this blog looked at in how crawl budget advice lands on AI crawlers. A robots.txt line is the wrong layer for a rate, which is why a CDN can treat ignoring it as a question of conduct rather than of protocol, the situation described in how a crawler can be unverified over a directive RFC 9309 never defined.

Thirteen other field names, and seven appeared in one file each

Crawl-delay is the established extension, so the other thirteen names are the more revealing half of the tally. 222 of the 1,014 files carry at least one field outside the four Google accepts, across 458 lines, and the tail of that distribution is people inventing an AI policy vocabulary inside a file that has no way to carry one.

Three of the thirteen are not AI related at all. Host appears in 28 files holding a hostname and Clean-param in two holding query parameter names, and neither is in the grammar RFC 9309 defines. Noindex appears in 36 lines across eight files, holding paths, and it is absent from RFC 9309, absent from The Web Robots Pages and absent from the four fields Google's documentation lists. Content-Signal appears in 30 files, which is the syntax Cloudflare published for declaring training, search and input intent, and the gap between what those lines declare and what the same file actually blocks is measured in the seven of 22 sites declaring ai-train=no that blocked no crawler.

The rest are one-offs, and every one of them is an attempt to point at a file rather than to state a rule. Three files declare llms with a URL pointing at their own llms.txt, canadiantire.ca writes llm-content for the same purpose, australia.com writes eighteen llm lines, one per locale, and wizzair.com writes three Link lines borrowing the HTTP Link header's syntax, complete with rel values naming an ai-policy and two machine-readable documents. zapier.com writes Agentmap pointing at an agent catalogue under its own well-known directory. Nothing reads any of these fields, and the file they are pointing at already has an agreed location: the llms.txt proposal puts it at the site root, so a pointer to it tells a client something it either checks already or does not.

The clearest case is thefurrow.tv, whose entire robots.txt is 129 bytes. It reads User-agent: $llm, then Allow: /, then Citation-Policy: allow-with-attribution and Training-Window: 30d. The two invented fields are plainly the point of the file, and the file cannot deliver them, because the only group it declares is addressed to a token no crawler has. RFC 9309's grammar builds a product token from hyphens, underscores and the letters A to Z in either case, and a dollar sign is none of those, which puts this file in the same class as the 813 tokens carrying a character the grammar disallows that we found when 1,004 files named 2,209 distinct tokens. There is no wildcard group either, so nothing in the file applies to anybody. A site has written down a citation policy and an attribution requirement in the one place where they are addressed to no one.

The malformed end of the tail is as instructive. Two files contain a bare URL on its own line, three such lines between them, which any parser splitting fields at the first colon reads as a field named https. On messly.com that line is a sitemap URL that was never given its Sitemap key. On amsterdam.nl the stray line sits four lines above a Crawl-delay 3 that does work.

What to check on your own file, and what this did not measure

The check is three questions, in order, and only the last is about the number.

First, is there a group that names the crawler you care about? If there is, that group is the only one it reads, so anything in the wildcard group, delay included, is invisible to it. Second, does that crawler's operator document reading the directive at all? On the evidence above that is one vendor of the nine, so for eight of them the line is a preference you are expressing rather than an instruction you are giving. Third, is the value one you would accept if it were honoured exactly? A file asking for 1,000 seconds is asking for 86 requests a day, and if the site has more than 86 pages it wants read, the request and the goal are in conflict. The vocabulary underneath all three questions is set out under AI crawler.

Nothing here says a Crawl-delay line is harmful, and two things argue for keeping one. It costs a line, and the single vendor that documents it runs a crawler many sites want present. What the measurement says is narrower: the directive is being written as though it were part of the standard and as though it were addressed to AI crawlers, and on 20 September 2026 it was neither. 157 files set it, 11 aimed it at an AI crawler, and the tokens actually carrying it belong to search and backlink crawlers rather than to any of the 15 this scanner evaluates. What this scanner does read, and what it declines to score, is set out in the methodology.

Four limits bound the figures above. 58 hostnames were never reached from this environment and 55 of them are news publishers, so the category most likely to be rate limiting an AI crawler deliberately is the category most missing from the sample. No access log was read on any site, so nothing here evidences what any crawler did with any of these lines, only what the file says and what the operator publishes; whether a request claiming to be a given crawler even is one is a separate problem, covered in why a user agent is a claim rather than an identity. This scanner's own parser discards Crawl-delay outright, so its presence is a fact we can report about your file and not an input to any score, which is a gap in this product rather than a finding about the web. And the corpus is one request per site from an editorial frame of large organisations, so every rate here supports a statement about these 1,419 hostnames and nothing wider, exactly as the sector figures did in the most blocked AI crawler count.