What Web Bot Auth actually specifies, and what it does not

What Web Bot Auth actually specifies

Cloudflare's Web Bot Auth documentation defines it in one sentence: an authentication method that leverages cryptographic signatures in HTTP messages to verify that a request comes from an automated bot. The cryptography underneath is not new work. It is RFC 9421, HTTP Message Signatures, a Standards Track document published in February 2024 by Annabelle Backman of Amazon, Justin Richer of Bespoke Engineering and Manu Sporny of Digital Bazaar. Web Bot Auth is a profile of that, narrowed to the case of a program identifying itself to a website.

The current core specification is draft-meunier-webbotauth-httpsig-protocol-00, dated 26 June 2026, written by Thibault Meunier of Cloudflare and Sandor Major of Google. Its abstract states the goal plainly: to allow automated HTTP clients to cryptographically sign outbound requests, allowing HTTP servers to verify their identity with confidence.

Three headers carry the scheme. Signature holds the signature value. Signature-Input names which parts of the message the signature covers, along with the parameters created, expires, keyid and tag, where tag is the fixed string web-bot-auth. Signature-Agent is optional and points at the location where a verifying server can fetch the key.

The draft constrains what a signature is allowed to cover, and the constraints are the interesting part. It must include at least one of @authority or @target-uri, which are the RFC 9421 names for the request host and the full request target, so a captured signature cannot be replayed against a different site or a different URL. It requires the four parameters above, and it recommends a validity window of no more than 24 hours. Where Signature-Agent is present it must itself be a signed component, which stops an intermediary swapping in a pointer to a key directory it controls.

The predecessor draft, draft-meunier-web-bot-auth-architecture-05 of 2 March 2026, is explicit that shared secrets are out: implementations must not use shared HMAC, and keyid must be a base64url JWK SHA-256 thumbprint. Cloudflare's announcement of 1 July 2025, which folded HTTP Message Signatures into its Verified Bots Program, names ed25519 as the verification algorithm and says Cloudflare released a Rust library and a TypeScript npm package for producing the signatures.

None of this is a finished standard. The protocol draft carries intended status Informational, and no document in the working group's set has an RFC number.

Where a crawler's public key lives, and why that is the reverse of robots.txt

A signature is only useful if the verifier can get the public key. Web Bot Auth publishes it as a JSON Web Key Set at a well known path, and Cloudflare's documentation gives the exact one: /.well-known/http-message-signatures-directory, served over HTTPS with the Content-Type application/http-message-signatures-directory+json, and itself signed with HTTP message signatures.

The part that matters for a site owner is whose host that path sits on. It is the crawler's. The operator of the bot publishes the directory on a domain it controls, the Signature-Agent header in the request points at it, and the site being crawled fetches it in order to check the signature.

That is the reverse of every access control mechanism this site normally measures. robots.txt is a file you write, on your host, that crawlers read. The same is true of llms.txt and of the meta robots tag. Web Bot Auth moves the file to the other end of the connection: the crawler writes it, on the crawler's host, and your server is the thing doing the reading.

The consequence decides what any scanner can report, so it is worth stating carefully. A tool can fetch your robots.txt and tell you what it says, which is what our robots.txt tester is for, because that file sits at a known path on your domain and is public. There is no equivalent fetch for Web Bot Auth. The signature is not on your site, it is in a request header arriving at your site, and the key is not on your site either. Nothing in how Lantad measures a page touches either one.

This also explains why the scheme needs a registry at all. Knowing that a request carries a valid signature over a key published at some directory tells you the request was signed by whoever controls that directory. Mapping that directory back to a named operator, so that a rule about GPTBot means something, is a separate problem, and it is what draft-meunier-webbotauth-registry-03 of 26 June 2026 exists to solve. Compare that with the crawler tokens we currently test, where the name in the user agent is the whole of the identity and anybody may type it.

There is a second consequence that cuts the other way, and it is the real improvement over an address list. An IP range is a property of where a crawler happens to run, so it changes when an operator changes hosting, and it is coarse, because a shared cloud range covers a great deal that is not the crawler. A signature is a property of the request.

What the working group has filed, read on 29 July 2026

The working group is called webbotauth, it sits in the Web and Internet Transport area, and its charter is approved as charter-ietf-webbotauth-01. Read on 29 July 2026, its document page listed ten active Internet-Drafts and two superseded ones.

Two of the ten were ten days old. draft-rescorla-anonymous-webbotauth-01 and draft-singh-webbotauth-hosted-directories-00 were both dated 19 July 2026. The first covers authorisation and rate limiting for agents that do not identify themselves individually, and the second covers hosting key directories on someone else's infrastructure. A scheme that requires every bot operator to run a signed well known endpoint on its own domain has an obvious problem for small operators, and the second draft is a response to it.

The two superseded documents are the ones most third party write ups still cite, which is a reason to check dates before repeating anything about this subject. draft-meunier-web-bot-auth-architecture-05 and draft-meunier-http-message-signatures-directory-05, both dated 2 March 2026, were replaced on 26 June 2026 by draft-meunier-webbotauth-httpsig-protocol and draft-meunier-webbotauth-httpsig-directory. A summary written before that date describes documents that no longer carry the specification, even where the technical content survived the rename.

The draft connected most directly to what this site measures is a JSON format for publishing crawler IP ranges, filed on 21 April 2026 by Gary Illyes. It defines a required creationTime and prefixes array, optional synctoken and notes, and per prefix an ipv4Prefix or ipv6Prefix in CIDR notation with optional services tags. Its abstract puts the problem as decreasing maintenance load on website operators while reducing the risk of inadvertently blocking beneficial clients.

That problem is not hypothetical, and we have run into it directly. We opened OpenAI's four crawler IP files on 29 July 2026 and counted the prefixes in each, which was possible only because the four URLs and the shape of the JSON were already known to us. Nothing tells a site owner where a given operator publishes such a file, and nothing obliges two operators to publish it the same way.

The remaining drafts cover operational ground rather than mechanism: crawler best practices, responsible web data collection, use cases for authenticating bots, and an identity anchor proposal for domain root discovery.

How you verify a crawler today, and what a signature would change

Nothing in the previous section is something a site owner can deploy alone. The verification that works today is the one described when we wrote that a user agent is a claim rather than an identity: take the request IP, check it against the ranges the vendor publishes for that crawler, and where the vendor supports it confirm that reverse DNS resolves to the vendor domain and forward resolves back to the same address.

Web Bot Auth does not remove that check so much as compete with it. The two answer the same question from different evidence. An address check asks whether the request came from a machine the vendor admits to running. A signature check asks whether the request was signed by a key the vendor publishes. The second is the stronger claim, because a signature travels with the request while an address is an accident of hosting, and because a signature covering @authority or @target-uri cannot be lifted from one request and reused against another site.

For a site behind Cloudflare the second is available rather than theoretical. Cloudflare's documentation, which carries a last updated date of 1 July 2026, describes registering through the dashboard and then sending signed requests, and points at a test endpoint at crawltest.com so an operator can see how its signatures are read before it deploys them.

What we have not confirmed, and are therefore not going to assert, is which crawler operators sign their production requests today. Several third party write ups name specific companies. We did not find that claim in the first party documentation we opened for this post, and an adopter list is exactly the kind of detail that is easy to repeat and hard to check, so it is not here.

If you run a site and want the answer for your own traffic rather than in general, it is already in your logs, and it does not depend on anyone's list. Look for an inbound Signature-Input header carrying tag="web-bot-auth". That is a fact about requests you actually received, which is a different and better class of evidence than a vendor announcement, and it is the same reason our own scanner publishes what it identifies itself as.

What no external scan of your site can tell you about this

Lantad scans a URL. It fetches the page the way an AI crawler would, checks what robots.txt says to each crawler token, and reports what is readable. Every one of those inputs is public and lives on the site being scanned, which is what makes an AI visibility measurement repeatable by anybody who wants to check our working.

Web Bot Auth lives nowhere near there. The signature is a header on an inbound request, so it exists only in the traffic arriving at your origin or your edge. The key directory is on the crawler's domain. An external scan of your site can observe neither, and there is no page on your site whose contents would change if every AI crawler in the world started signing tomorrow.

So this post reports a development and grades nothing, which is the position we took on the Search Console control for AI Overviews and on the IETF AI preferences drafts for the same reason. When the thing that decides an outcome is not on the site, a scanner that printed a number for it would be printing a guess, and we would rather withhold the grade than dress one up.

The practical division is worth keeping straight, because the layers now decide independently whether an AI crawler reads your page and they fail independently too. The first is what your site says: robots.txt, meta robots, and the markup a crawler actually receives. That is the layer a scan measures. The second is what your edge does regardless of what your files say, which is the argument that two layers decide whether AI can read your site, and a scan sees that one only as an outcome. Identity is a third, and it is the one nobody can see from outside.

If you want to act on this now, two things are worth doing and neither is urgent. Check your logs for signed requests, so that you know whether traffic you are already receiving carries the header, and you are reading your own evidence rather than waiting for an announcement. And make sure the layer you do control is not the thing standing in the way, because a crawler that proves its identity perfectly still gets nothing useful out of a page it is disallowed from fetching, or a page that renders empty without JavaScript.