Verified bots: Cloudflare will grade robots.txt compliance, and 232 of 1,059 files carried a defect

What does Cloudflare now measure about verified bots?

Cloudflare's Verified bots list has until now answered a question about identity. An operator applies, demonstrates that the requests it sends come from the ranges and the agent it claims, and the network treats traffic from that bot as belonging to a known party. The post published on 16 October 2026 keeps that step and adds a second one that runs continuously. Its own summary of the change is four words: verify, but continuously re-verify.

Two behaviours are named as the initial signals. The first is whether a bot respects instructions, which the post locates in robots.txt, described there as the primary way to communicate site owner preferences, with the expectation that responsible bots consult and honour what websites publish in line with RFC 9309. The second is whether a bot responds to direct feedback, which the post defines narrowly as an HTTP 429 response, and the question it asks of that case is whether the bot slows down, pauses, or continues sending requests at the same pace.

Neither observation yields a pass or a fail. The post says the observations will translate to an explainable trust band, a range showing how consistently a bot respects site preferences, and it is explicit that this is a range rather than a permanent judgment about a bot or its operator. It also sets a bar for what registers at all: isolated violations and other types of one-offs may not be significant, and what Cloudflare says it looks for instead are repeated patterns of behaviour that disregards site preferences.

The consequence is not hypothetical, and the post names its own precedent. Cloudflare de-listed Perplexity's bot from its Verified bot list in 2025, after it sought to access resources on websites that opted out of scraping and subsequently attempted to circumvent controls when those requests were blocked. The system is running internally today. Scores will later be published on Radar, and Cloudflare plans to make the system available to all its customers, who will be able to choose the trust threshold used to decide how Verified bot traffic is handled. That final sentence is the one that reaches a site owner, because it turns a vendor's assessment into an access decision on their own domain. This blog has written before about the moment Cloudflare made verified status revocable for conduct. What is new is that conduct is now measured on a schedule rather than noticed after a complaint.

The first signal is robots.txt, and 232 of 1,059 corpus files carried a defect

Naming RFC 9309 sets a bar that is narrower than most robots.txt files assume. The Robots Exclusion Protocol, published on the Standards Track in September 2022, defines a grammar with three meaningful keys: a user-agent line that opens a group, and the allow and disallow rules that sit inside it. It does not mention crawl-delay anywhere. It tells a crawler it must try to parse each line, must match the product token case-insensitively, and must assume complete disallow when the file is unreachable because of a server or network error. It permits caching, and says a crawler should not use a cached copy for more than 24 hours.

That is the specification a bot is now to be graded against. The question this scanner can answer from outside is what the files on the other side of that grade actually contain. A robots.txt validator run against the 1,419 hostnames in this repository's two committed corpus seed files on 19 September 2026 found 1,104 answering HTTP 200, and of the 1,059 responses that held real directives, 232 carried at least one defect measured against RFC 9309 or Google's published parsing rules.

Those defects are not cosmetic. 107 of the files named a user-agent value containing a space or punctuation that RFC 9309's product token grammar does not permit, across 1,394 such lines, which means no crawler's product token can match them and every rule in those groups binds nothing. 33 of the 1,104 responses were HTML pages rather than robots.txt text, and only one of the 33 contained the string user-agent anywhere, so the other 32 hand a compliant crawler a file with no rules in it at all. 224 files used a field outside the four that Google documents supporting, with crawl-delay among them in 158 files.

Set those two findings beside each other and a specific problem appears. A bot that reads a file naming a token it cannot match, and then crawls the paths that file was written to close, has honoured RFC 9309 exactly. A bot that treats the same file as blanket permission has also honoured it. The behaviour a trust band would record is identical in both cases, and the difference between them lies in a file the bot did not write. Cloudflare's earlier policy made this sharper still by naming crawl-delay, a directive RFC 9309 never defined, among its examples of a breach. The 16 October post names the specification instead, which is both the stricter choice and the more defensible one, and it also narrows the grade to what that specification actually covers.

There is a prior condition underneath all of this, and it is worth stating because a grade cannot see it. A study submitted to arXiv on 16 July 2026 and revised on 19 July 2026, Do Generative AI Assistants Respect robots.txt?, presents a controlled empirical study of ten widely used AI assistants with advertised web-search capabilities across 200 trials, and reports that some of those systems accessed restricted resources without requesting robots.txt. A bot that never fetched the file is not a bot that disagreed with its contents, and from the outside the two are hard to tell apart.

The second signal is HTTP 429, and 63 of 1,388 sites returned one

The second signal is where this corpus holds the sharper figure, and it points in an awkward direction.

A 429 is the response Cloudflare's post names as the clearest way a website can push back, and it describes that code as a directive for the visitor to reduce its request rate, sometimes with guidance about how long to wait before trying again. That guidance has a name and a specification behind it. Google's reduce crawl rate documentation, carrying Last updated 2026-10-06 UTC, tells site owners to return a 500, 503 or 429 instead of a 200 in order to shed crawler load, and says that when returning a 503 or a 429 you can also include a Retry-After header, giving either a delay in seconds or an absolute UTC date and time. It is offered on two of those three codes and not on the 500.

Lantad requested the home page of each of the 1,419 corpus hostnames once on 22 October 2026 as LantadBot with no JavaScript executed. 1,388 answered. 63 of those returned 503, 429 or 500, which is the entire population of hosts in this corpus that pushed back at all on that request. Not one of the 63 carried a Retry-After header. A second request to all 63 returned the same status code on 61 of them, and still no header. Across the full 1,388 responses, four carried Retry-After at all, and none of those four was a 500, 503 or 429.

Be precise about what that does and does not undercut, because the honest reading is narrower than the figure invites. Cloudflare does not need Retry-After in order to observe the signal it described: it can watch whether a bot's request rate falls after a 429 and grade it on that alone, which is what the page says it does. What the measurement adds is a property of the sites rather than of the scheme. On the hosts in this corpus the push back arrives with no stated interval, so a bot that slows down is guessing by how much and for how long, and a bot that waits the wrong amount of time is indistinguishable, to anything counting requests, from a bot that ignored the response. The measurement also shows how thin this signal is to begin with. 63 of 1,388 hosts emitted it once, which means the 429 half of a trust band is computed from the small minority of the web that rate limits anything at all.

A trust band grades a relationship, and the site holds half of it

Both named signals originate on the site. That is not a flaw in the design, because the only observable definition of respecting a preference is respecting a preference that somebody expressed. It does mean a trust band measures a relationship rather than a property of a bot, and it means the quality of the assessment depends on the quality of the expression it is assessed against.

This corpus holds a measurement of how often the two halves of a site's own expression disagree with each other. Of 700 hostnames that answered a baseline bot with a 2xx on 13 September 2026, 115 refused at least one of four AI crawler user agents, and on 79 of those 115 the site's own robots.txt allows at the site root a crawler that the server turned away. Across 2,800 host and crawler pairs, 312 requests were refused. Splitting those 312 against each file re-read the same day, 194 were refusals of a crawler the file permits, 96 matched a disallow already written in the file, and 22 sat behind a robots.txt that would not parse.

One caveat belongs with that figure, and it is written into this scanner rather than discovered afterwards. None of those refusals proves a deliberate block. A server that verifies crawlers by address will refuse an unverified user agent from any scanner, and every request in that run came from an address in no vendor's published range. That is the same problem Cloudflare's list exists to solve, which is rather the point: from outside a site, a deliberate refusal and a failed verification look identical, and our methodology says so instead of grading around it.

What a site owner can take from this is not that Cloudflare's scheme is wrong. It is that half of what will be graded is theirs, that their half is measurable today from outside their own network, and that it is currently wrong on a meaningful fraction of real sites. A bot's trust band is not something a site can set. A well formed file and an edge that agrees with it are.

Verification still rests on identity a site cannot check from outside

Verification is the half of this that a site cannot run for itself, and the asymmetry is worth stating because it explains who gets to compute a trust band at all.

Cloudflare can observe a bot's behaviour because it sits in the request path for a large number of sites and can compare what one agent did across all of them. A single site sees its own logs. From outside a site, which is where this scanner works, even the identity question is mostly unanswerable. Lantad asked 2,344 addresses sampled from the 293 prefixes OpenAI publishes for a reverse DNS record on 12 October 2026, and every single query returned NXDOMAIN on two independent public resolvers. Across six vendors that publish a machine readable address list, 11,362 distinct published addresses were asked, 8,510 answered, and 2,852 had no PTR record at all. All 12 addresses Perplexity publishes and 29 of the 32 exact addresses in Anthropic's list resolved to generic cloud names, so a forward confirmed lookup on those proves which cloud sent a request and not which company did.

The cryptographic alternative is specified and barely deployed. Web Bot Auth signs a request rather than asserting a user agent, and the only half of it a site can inspect from outside is the operator's published key directory. Of the nine vendors behind the 15 crawler tokens this scanner evaluates, two published such a directory on 15 September 2026 and seven did not.

So the practical division of labour as of 25 October 2026 reads like this. A network operator with position in the request path can verify identity, and now proposes to score conduct on top of it. A site owner can verify neither, can state which crawlers it means to address, and will in time be offered somebody else's score as a dial on their own traffic. Knowing which half of that is yours is the difference between acting on this announcement and waiting for it to arrive.

What a site owner can check on their own site today

Nothing in Cloudflare's post asks a site owner to do anything, and that is the gap worth filling, because the two signals being graded are the two that a site can get wrong on its own.

Four checks follow directly from what the page names. Parse your robots.txt rather than reading it, since the defects in this corpus were overwhelmingly invisible to a human skim: a token with a space in it looks like a rule on the screen and binds nothing in a parser. Our robots.txt tester evaluates a named crawler token against a live file and reports which rule decided the verdict, which is the same resolution a compliant bot performs. Confirm that the file is text and not an HTML error page, which 33 of 1,104 corpus hosts served. If you shed load deliberately, send Retry-After with the 503 or the 429, because an interval you state is one a bot can be held to and an interval you omit is one it has to guess at. Then check that your edge and your file agree, since 79 of the 115 refusing hosts in this corpus turned away a crawler their own file allows at the root.

Two of those four are observable from outside with no access to your logs, which is what this scanner does: what a crawler actually receives from your pages, and what LantadBot is told it may fetch. The other two need your own configuration in front of you. None of the four depends on a trust band, a verification state, or any figure held on somebody else's network, which is the argument for doing them now rather than when the dial arrives.

What Lantad did not measure

This post reports a document and reuses measurements already published here. The distinction matters enough to list plainly.

Lantad holds no verification status with Cloudflare, runs no traffic through any bot pipeline, and holds no trust band for any crawler. No figure above describes how any bot is classified on any network, because that state lives on Cloudflare's side of the request and does not appear in a scan of a hostname. The behaviours Cloudflare says it observes, being a bot's request rate after a 429 and its path selection after reading a file, are visible in a site's own server logs or at its edge, and this scanner reads neither of those.

The corpus figures carry their own limits, stated where each was published. The 1,419 hostnames in this repository's two committed corpus seed files are an editorial and platform sample rather than a random draw from the web, so a proportion here describes this corpus and not the internet. Each run is a single request per host on a single date unless a second check is reported, which is why every figure above carries its date. The refusal figures cannot separate a deliberate block from an address based verification failure. Our methodology and the crawlability study hold the sampling and the scoring behind these numbers, and the reasons a grade is withheld rather than guessed have not changed.