# Cloudflare can unverify a crawler over a directive RFC 9309 never defined

> Cloudflare's verified bots policy names an AI crawler ignoring the robots.txt crawl-delay directive as an example of a policy breach, and its post of 7 August 2026 makes verified status revocable for conduct. RFC 9309 does not mention crawl-delay, and Google's own robots.txt specification says it is not supported.

- Canonical page: https://lantad.co/blog/crawl-delay-can-unverify-a-crawler
- This file: https://lantad.co/blog/crawl-delay-can-unverify-a-crawler.md
- Last substantive update: 2026-08-07

## Key facts

- **Published:** 2026-08-07
- **Category:** Findings
- **Author:** Lantad
- **Length:** 3733 words
- **Takeaway 1:** Cloudflare published Unveiling good and bad behaviors on the Agentic Internet on 7 August 2026, introducing BotBase as a directory of all known bots and agents which, in its own comparison with the previous Bots Directory that only included known good bots, is also capable of tracking less-than-good bots and agents.
- **Takeaway 2:** Cloudflare's verified bots policy page, last updated 1 July 2026, sets two bars for verification, honest self-identification and non-abusive behavior, and lists among its examples of a policy breach an AI Crawler that does not respect the crawl-delay directive in robots.txt.
- **Takeaway 3:** RFC 9309, published on the Standards Track in September 2022, defines only user-agent, allow and disallow in its grammar and does not mention crawl-delay anywhere, while Google's robots.txt specification, last updated 2026-07-08, states that other fields such as crawl-delay are not supported.
- **Takeaway 4:** Cloudflare's post of 7 August 2026 reports 206 million Precursor evaluation events across 73,438 zones on its network in a 24-hour period, from a client-side script that its announcement of 13 July 2026 says captures pointer movement, keyboard activity, focus changes and visibility.
- **Takeaway 5:** Lantad measured none of this and cannot measure it from outside a site: verification state is held on Cloudflare's network and the behavioural signals are collected inside a visitor's browser, so neither appears in any scan of a hostname.

## Summary

There are two ways for an AI crawler to lose access to a page. The site can ask it not to come, in a file the crawler is free to ignore, or the network in front of the site can refuse the request before the origin ever sees it. Cloudflare published a post about the second one on 7 August 2026, and the part worth reading closely is how much of it now turns on the first.

Lantad measured none of what follows. We hold no verification status, we run no traffic through anyone's bot pipeline, and we have no figure for how any crawler is classified on any network. What this post does is read three documents against each other: Cloudflare's announcement, Cloudflare's own policy page describing what a verified bot must do, and the specification that both of them treat a well behaved crawler as obeying. Those three do not agree about what a robots.txt file contains, and the disagreement lands on one line that is now attached to a consequence.

## What Cloudflare published on 7 August 2026

The post is titled Unveiling good and bad behaviors on the Agentic Internet, and its first announcement is BotBase, which Cloudflare describes as a directory of all known bots and agents. The second is a wider rollout of a client-side detection system called Precursor, which the fourth section of this post comes back to.

BotBase replaces something narrower, and Cloudflare draws the comparison itself: compared to our previous Bots Directory, which only included known good bots, BotBase is also capable of tracking less-than-good bots and agents. A list of approved crawlers has become a list of every crawler, with a judgement attached to each entry rather than membership standing in for the judgement.

That judgement has two conditions, and [the post](https://blog.cloudflare.com/good-and-bad-agentic-behaviors/) states them as a numbered pair: you declare yourself honestly, and you don't abuse the trust you've earned. Then it states the consequence in one sentence, which is the subject of everything below. If you abuse trust on the Cloudflare network, you should not be easily allowed, so you will be unverified.

Read that as a change of kind rather than of degree. A directory of good bots is a claim about identity: this name belongs to this operator, and here is the proof. A trust judgement is a claim about conduct, conduct continues after the paperwork is done, and a status that rests on conduct is therefore revocable. A crawler that is verified in September is verified because of what it did in August.

For an [AI crawler](https://lantad.co/glossary/ai-crawler) operator that changes what the credential is worth carrying. For a site owner it changes what the allowlist means, and the point was already legible in a controlled study we read earlier this month. In that experiment the single agent stopped by Cloudflare's free AI bot controls was the one [listed in Cloudflare's bot directory](https://lantad.co/blog/cloudflare-blocked-one-of-seven-browsing-agents), which the paper's authors read as self-identification rather than detection. Appearing on the list was what made that agent addressable at all. Under the model described on 7 August, appearing on the list is also what makes the listing removable.

## What a crawler has to do to stay verified

The announcement is a blog post, and blog posts summarise. The operative document is [Cloudflare's verified bots policy page](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/policy/), which carries a last updated date of 1 July 2026 and states the requirement in two bars rather than two slogans.

The first bar is honest self-identification, which the page defines as declaring who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS. Every one of those three is a mechanism this blog has looked at directly. The signature scheme is the one where [a crawler's public keys live on the crawler's own host rather than on yours](https://lantad.co/blog/what-web-bot-auth-actually-specifies), which is the reverse of every access file a site owner writes. The published IP list is the pattern behind [the four JSON files OpenAI maintains for its four crawlers](https://lantad.co/blog/openai-crawler-ip-ranges-counted). And the reason any of the three is required at all is that [a user agent string is a claim rather than an identity](https://lantad.co/blog/a-user-agent-is-a-claim-not-an-identity), so a name on its own proves nothing.

The second bar is the new one, and it is where conduct enters. Cloudflare defines non-abusive behavior as a bot that obeys robots.txt and crawl directives, maintains reasonable request rates, and has not been observed evading website owner preferences or attacking sites.

That sentence puts a file written on your server into the conditions of a credential held on Cloudflare's. Nothing about your robots.txt gains any power to stop a request. What it gains is a role as evidence in somebody else's decision about whether a crawler keeps the status that makes its requests welcome by default. Those are different things, and conflating them is the most common error in this whole subject.

The policy page then lists examples of a breach, and they are worth reading as a set because most of them are facts about infrastructure rather than about crawling. Adding a set of IPs that are not solely used by the verified service. The service IPs being breached by an attacker. Unpatched vulnerabilities. A block of IPs not briefed on onboarding being added to the list. The disclosed purpose of the service not reflecting the traffic. And then one that is not like the others at all: an AI crawler that does not respect the crawl-delay directive in robots.txt.

## Crawl-delay is not in the standard, and Google does not support it

[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) is the Robots Exclusion Protocol, published on the Standards Track in September 2022. It is the document that fixes the file at /robots.txt, defines how a group is matched to a crawler name, and specifies the grammar a parser is expected to accept. That grammar contains three rules: user-agent, allow and disallow. Sitemap appears only as an optional other record, explicitly outside the core protocol. Crawl-delay is not in the grammar, is not in the other records, and is not mentioned in the document at all.

It is not a Cloudflare invention either. Crawl-delay is an old convention, widely written and widely honoured, and several crawlers do read it. What it does not have is a single definition. There is no specification text stating its units, its permitted range, how it interacts with a group inherited from a wildcard, or what a parser should do with a value it cannot make sense of. Every consumer decides for itself.

The largest consumer decided not to read it. [Google's robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec), last updated 2026-07-08, introduces its list of supported lines with a parenthesis that names the exception directly: Google supports the following fields, other fields such as crawl-delay are not supported. Four fields follow, user-agent, disallow, allow and sitemap. So the one directive Cloudflare names in a removal example is the one Google's own documentation singles out as ignored.

This is not a contradiction and nobody is wrong. Cloudflare's policy governs bots seeking verification on Cloudflare's network, and it may hold them to whatever it chooses, including conventions no standard covers. Google's specification describes what Googlebot parses. Both statements are true at once. But the two together mean something specific for anyone writing the file, and it is not what a site owner would guess: a robots.txt is now read by at least two audiences with different grammars, and the line with the least specification behind it is the one carrying a stated consequence.

The pattern is familiar from the rest of this file's history. The same document already produces opposite outcomes depending on the status code that serves it, which is why [a 404 on robots.txt allows every crawler while a 503 blocks them all](https://lantad.co/blog/robots-txt-404-and-503-are-opposites). A file whose meaning depends on the reader is a file that needs testing against each reader, not proofreading once.

## The half of the detection stack that a crawler cannot reach

The other system in the announcement works nothing like a policy page, and the contrast is the reason the crawl-delay line matters as much as it does.

[Precursor](https://blog.cloudflare.com/introducing-precursor/), announced on 13 July 2026 by Benedikt Wolters, is a client-side session-based verification system. Cloudflare states that when it is enabled, Cloudflare automatically injects a lightweight script into HTML responses from your site as they pass through our network, with no additional configuration, network connections, or third-party embedding required. That script attaches lightweight event listeners to capture interaction signals such as pointer movement, keyboard activity, focus changes, and visibility.

The scale is in the August post: 206 million Precursor evaluation events across 73,438 zones on the Cloudflare network in a 24-hour period at the time of writing. The reason for continuous evaluation is given plainly too, that behavior often shifts from human to agentic and back over a session, which is not something a single check at the door can catch.

Now hold that against what a fetch-only crawler is. GPTBot, ClaudeBot, CCBot and the rest of the tokens in [our crawler registry](https://lantad.co/tools/ai-crawlers) issue an HTTP request and read a response body. They execute no JavaScript, so the injected script never runs. They have no pointer, so there is no movement to characterise. They have no keyboard and no focus changes and no visibility state. A well behaved training crawler and a badly behaved one are equally invisible to a system that measures how a hand moves a mouse, because neither of them has a hand.

Cloudflare's two posts do not say what the system returns for a client that executes no JavaScript, and we have not tested it, so nothing here is a claim about how such a client is scored. The structural point stands without that claim: behavioural detection is aimed at automation wearing a browser, which is the population a separate study we covered was able to classify at [0.993 accuracy from network, TLS and browser fingerprints combined](https://lantad.co/blog/web-agents-identifiable-robots-txt-stopped-one). For a client that arrives without a browser at all, the declared identity and the conduct record are what remain. Which is to say that for the crawlers most site owners are actually asking about, the list is close to the whole of it, and the list has just become revocable.

None of this touches whether your text is readable once a request is allowed through. That question is separate, it is answered in the response body rather than at the edge, and [prose parity](https://lantad.co/glossary/prose-parity) is the name we give to it.

## Why none of this shows up in a scan of your site

RFC 9309 is unusually direct about the limits of the file it defines. These rules are not a form of access authorization, it says, and it follows that with advice: listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable, so anyone controlling access to a path should employ a valid security measure relevant to the application layer.

That sentence has always meant a robots.txt cannot stop anybody. What the 7 August announcement adds is the other direction. The file cannot grant access and it never could, but a crawler's observed compliance with it is now an input to whether the network in front of you extends that crawler the benefit of the doubt. The weaker layer does not become stronger. It becomes evidence, which is a different job, and one it was not designed for either.

We have written before that [robots.txt was never the layer that answers](https://lantad.co/blog/two-layers-decide-if-ai-can-read-your-site), and that remains the right way round. The refinement is that the layer that does answer has started reading the layer that does not.

Now the part that constrains what any scanner, ours included, can honestly tell you. Verification state lives in BotBase, on Cloudflare's network. Conduct is observed across requests to other people's sites, aggregated somewhere you have no access to. Precursor's signals are collected inside a visitor's browser and evaluated on the network. Not one of those three is a property of your hostname, so not one of them can be fetched from outside it. A scan can show that a named crawler received a 403, and cannot show whether that came from a firewall rule, a bot score, a rate limit or an origin that was simply down.

We say this on our own product rather than about somebody else's. [What our method does and does not do](https://lantad.co/methodology) states its boundary before it states any result, and [the documentation for our own crawler](https://lantad.co/bot) says what our scanner sends and how to recognise it, because a tool that asks you to trust its identity should publish that identity first. The honest summary of this entire development, from a scanner's point of view, is that it happens somewhere a scanner cannot look.

## What to check on your own site this week

Two of the three documents in this post describe things you cannot touch. The first one describes a file you can, and it is worth an hour because the failure modes are quiet ones.

Start with whether your groups match anything. RFC 9309 matches a group to a crawler by name, and a name that no crawler answers to inherits nothing and forbids nothing. A group with a typo, an obsolete token or a product name that was never a user agent is not a weak rule, it is an absent one, and it looks identical to a correct rule when you read the file yourself. This is not hypothetical: [a renamed crawler token leaves a robots.txt group matching nobody](https://lantad.co/blog/renamed-crawler-tokens-leave-stale-robots-rules), and nothing on your site reports the change. Reading your file the way a specific named crawler reads it is what [a robots.txt tester](https://lantad.co/tools/robots-txt-tester) is for, and it takes a couple of minutes per token.

Then look at your crawl-delay line, if you have one, and decide what it is for. After this week it is doing two jobs that pull in different directions. It is a request to crawlers that read it, ignored by at least one major search crawler whose documentation says so in as many words. And it is now something an AI crawler operator may be held to on a network you do not control. Neither job is a reason to delete it. Both are reasons to know which crawlers you expect to honour it, rather than assuming the line applies to everyone because it is in the file.

Then stop worrying about the file and check the response. Every access question above is upstream of the only question that decides whether you are quotable, which is whether your sentences are in the HTML a client receives. Fetching a page as a named crawler and reading what comes back is the whole of that test, and [seeing what a crawler receives](https://lantad.co/tools/what-gptbot-sees) answers it in one request. If the answer is that your text arrives only after a script runs, the access layer was never your constraint.

One last thing, on order of operations. Cloudflare's own [agent readiness diagnostics groups its checks by effort](https://lantad.co/blog/agent-readiness-score-counts-three-of-four-groups), and permission sits in the first group with the sitemap and the machine-readable copy for a reason: the sophisticated items further down are a second storey on a first floor that has to exist. That order has not changed this week. What changed is that one line on the ground floor now has a consequence attached to it that is not written anywhere in the standard that defines the file.

## Questions and answers

**Can an AI crawler lose its verified status with Cloudflare?**

Yes, according to Cloudflare's post of 7 August 2026, which states that if you abuse trust on the Cloudflare network, you should not be easily allowed, so you will be unverified. Its verified bots policy page, last updated 1 July 2026, sets two bars for verification, honest self-identification and non-abusive behavior, and lists examples of a breach that include adding IPs not solely used by the verified service, a disclosed purpose that does not reflect the traffic, and an AI crawler that does not respect the crawl-delay directive in robots.txt.

**Is crawl-delay part of the robots.txt standard?**

No. RFC 9309, published on the Standards Track in September 2022, defines user-agent, allow and disallow in its grammar, mentions sitemap only as an optional record outside the core protocol, and does not mention crawl-delay anywhere. Google's robots.txt specification, last updated 2026-07-08, is explicit that it supports user-agent, disallow, allow and sitemap, and that other fields such as crawl-delay are not supported. Crawl-delay is a widely used convention with no single specification behind it.

**Does Cloudflare's Precursor detection apply to crawlers that do not run JavaScript?**

Cloudflare's announcements do not say. Precursor is described in its post of 13 July 2026 as a script injected into HTML responses that captures pointer movement, keyboard activity, focus changes and visibility, none of which a client that executes no JavaScript produces. Neither that post nor the one of 7 August 2026 states what the system returns for such a client, and Lantad has not tested it, so no claim is made here about how a fetch-only crawler is scored.

**Can a scan of my site tell me whether a crawler is verified?**

No, and neither can any other external tool. Verification state is held in BotBase on Cloudflare's network, a crawler's conduct record is aggregated from its traffic to sites other than yours, and Precursor's signals are collected inside a visitor's browser. None of the three is a property of your hostname. A scan can show that a named crawler received a particular status code from your site, and cannot show which layer produced it.

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot