BlogFindings
Web bot authentication: 2 of 9 AI crawler vendors publish a key directory
Lantad requested /.well-known/http-message-signatures-directory once from each of 51 AI crawler operator hostnames on 15 September 2026. Four served a key directory, 40 answered 404, and three answered HTTP 200 with an HTML page. Of the nine vendors behind the 15 crawler tokens this scanner evaluates, two publish one.
The specification question was covered here in July. This post asks a different one: who has actually turned it on. There is exactly one half of the scheme a site owner can inspect without receiving a signed request, which is the key directory an operator publishes at a fixed URL, and that makes deployment measurable from outside. On 15 September 2026 we requested /.well-known/http-message-signatures-directory over HTTPS, once, from each of 51 hostnames belonging to operators of AI crawlers, assistants and agents, as LantadBot/1.0 with redirects followed and a twenty second timeout, from one network location. Four of the 51 served a directory. The current working group draft is what every claim below about required behaviour is checked against.
In short
- Web bot authentication replaces a crawler's user agent claim with a signature a server can check, and the only half of it a site can inspect from outside is the operator's published key directory. Lantad requested that directory from 51 AI crawler operator hostnames on 15 September 2026.
- Four hostnames served one: chatgpt.com, meta.com, www.meta.com and you.com. Those four are three distinct operators, because meta.com and www.meta.com returned the identical 250 byte document. 40 hostnames answered 404.
- Of the nine vendors behind the 15 AI crawler tokens this scanner evaluates, two publish a directory and seven do not. Anthropic, Perplexity, Google, Apple, ByteDance, Common Crawl and Amazon returned nothing at the well-known path on 15 September 2026.
- All five published keys are Ed25519, and only OpenAI's carries a kid. Computed independently on 15 September 2026, that kid matches the RFC 8037 thumbprint of its own key exactly, so the one directory that labels its key labels it correctly.
- OpenAI's own crawler documentation at developers.openai.com, read on 15 September 2026, contains the string signature zero times. The vendor running the most complete directory in the sample does not mention the mechanism on the page it points site owners at.
What is web bot authentication, and what can a site check from outside?
The cryptography is not new. RFC 9421, HTTP Message Signatures, published on the Standards Track in February 2024, defines how to sign selected parts of an HTTP message. Web bot authentication is a profile of that document narrowed to one case: an automated client identifying itself to a website. The signature travels in a Signature header, the components it covers and the parameters created, expires, keyid and tag travel in Signature-Input, and the tag is the fixed string web-bot-auth. An optional Signature-Agent header names where the verifying server can fetch the key. The header layer was set out here in what Web Bot Auth actually specifies, and that post is about the document rather than about anyone's deployment.
The document itself has moved since. The individual submissions have been replaced by draft-ietf-webbotauth-httpsig-protocol-00, dated 1 September 2026, written by Thibault Meunier of Cloudflare and Sandor Major of Google, carried by an IETF working group called Web Bot Auth, with an intended status of Standards Track and an expiry of 5 March 2027. A draft acquiring a working group is a real change in standing and not a change in what any crawler does, so it is worth recording and worth not overreading.
The asymmetry that makes this post possible is worth stating plainly. A signature arrives on an inbound request, so nobody can see one without receiving traffic from the crawler in question. A key directory is a document served at a URL, and anyone can ask for it. Section 5.5.1 of the draft requires that a directory served at the well-known URI be served over HTTPS with the media type application/http-message-signatures-directory+json, and it registers the path suffix, which means the path is not an operator's choice. So the directory is the observable half, and it answers a narrower question than the one people usually ask: not whether an operator signs its requests, but whether it has published the keys that would let anyone check.
That is a different instrument from the address lists this scanner already reads. Verifying a crawler by IP means fetching the operator's published ranges and doing a lookup, which is the method behind four OpenAI crawler IP lists opened and counted and behind the finding that crawler IP range files carry two dates that disagree. Signatures move the proof onto the request. Our own crawler's identity and conduct policy is published at the bot page for the same reason: an operator that wants to be checkable has to publish something first.
Flow: Crawler sends request to Signature headers; Signature headers to Signature-Agent URL; Signature-Agent URL to Key directory at well-known URI; Key directory at well-known URI to JWKS of public keys; JWKS of public keys to Server verifies; Signature headers (not fetchable) to Needs inbound traffic; Key directory at well-known URI (this study) to Observable to anyone.
What 51 crawler operator hostnames served on 15 September 2026
The frame was built from this repository's own crawler registry rather than from a news roundup. core/src/bots.ts holds the 15 AI crawler tokens the scanner evaluates, and those 15 tokens belong to nine vendors: OpenAI, Anthropic, Perplexity, Google, Apple, Meta, ByteDance, Common Crawl and Amazon. For each vendor we probed the hostnames its own crawler documentation names, plus the company and product domains a Signature-Agent header would plausibly point at, which came to 30 hostnames. The same registry is what the AI crawler reference renders, and the vendors in it were counted before in six of the nine AI vendors publishing one crawler token. A further 21 hostnames cover operators of AI crawlers, assistants or agents outside those nine, including Cloudflare, Microsoft, Mistral, Brave, DuckDuckGo, Kagi, You.com, xAI, Cohere, DeepSeek, Diffbot and Hugging Face. Fifty one hostnames in total, one GET each.
Forty of the 51 answered 404. Four more answered something else that is not a directory: huggingface.co returned 401, phind.com returned 403, zhanzhang.toutiao.com returned 500 and duckduckgo.com returned 503. Seven answered HTTP 200, and only four of those seven carried a directory. platform.openai.com returned 3,810 bytes of text/html, bytedance.com returned 53,388 bytes of text/html and copilot.microsoft.com returned 67,283 bytes of text/html. All three are ordinary pages answering under a 200 at a path that has no directory on it, which matters to anyone writing a verifier: a status code is not a result here, and three of seven 200s in this sample would have been false positives for a check that read the code and stopped.
Split by vendor, two of the nine publish and seven do not. OpenAI publishes at chatgpt.com and Meta publishes at meta.com and www.meta.com. Anthropic returned 404 at all four hostnames we asked, including claude.com and support.claude.com. Perplexity returned 404 at all three, Google at all four including gemini.google.com, Apple at all three, Amazon at all three, Common Crawl at both, and ByteDance returned 404 at toutiao.com, 500 at zhanzhang.toutiao.com and a 200 carrying an HTML page at bytedance.com. Seven of nine is the number to carry away, and it sits alongside earlier counts of what these same vendors do and do not publish, including 6 of 15 crawler tokens publishing a User-Agent string to match and the correction recording that Meta documents five crawler tokens.
One response behaved differently depending on the method, and it is reproducible. chatgpt.com answers a HEAD request for the directory with 404 and a GET for the same URL with 200. We ran both twice, alternating, and got the same pair of answers each time. A verifier that probes with HEAD before fetching would conclude that OpenAI publishes no directory. meta.com answers HEAD with a 301 to www.meta.com, which is harmless but means the bare host needs the redirect followed. Neither behaviour breaks any requirement in the draft, which says nothing about HEAD, and both are the kind of detail that decides whether an implementation works. How this scanner records that sort of thing, and what it refuses to grade on, is set out on the methodology page.
| Response | Hostnames | Served a directory |
|---|---|---|
| 200 with the directory media type | 4 | yes |
| 200 with text/html | 3 | no |
| 404 not found | 40 | no |
| 401 unauthorized | 1 | no |
| 403 forbidden | 1 | no |
| 500 server error | 1 | no |
| 503 unavailable | 1 | no |
The four directories, opened and read
All four documents carried the media type the draft requires, which is the one thing every publisher in the sample got right. They agree on almost nothing else. OpenAI's directory at chatgpt.com is 243 bytes and holds one key. Meta's, identical on meta.com and www.meta.com, is 250 bytes and holds three. You.com's is 90 bytes and holds one. Five keys across three operators, and every one of them is an Ed25519 key declared as kty OKP with crv Ed25519. No other algorithm appears anywhere in the sample, which matches the shape of the draft's own example and of the verified bot work at Cloudflare's web bot auth documentation.
The disagreement is about what else the document says. OpenAI's carries three top level members: keys, signature_agent set to https://chatgpt.com, and purpose set to ai. Its single key carries kty, crv, x, kid, use, nbf and exp. Meta's and You.com's carry one top level member, keys, and their keys carry three fields each: kty, crv and x. Nothing more. So one publisher ships the full shape of the draft's example and two ship the minimum a JSON Web Key Set can be.
Neither extra member is an error, and it is worth being exact about that rather than scoring it. The directory format is a JWKS as defined in RFC 7517, whose Section 5 states that additional members can be present in the JWK Set and that implementations encountering members they do not understand must ignore them. So signature_agent and purpose are legal and inert. What is worth recording is that the draft defines neither of them for the directory, and that its own change log carries the line "Remove the purpose field from the Web Bot Auth example" against an earlier revision. A field the editors took out of the example is still being served in production, by the operator whose crawlers are named in more robots.txt files than anyone else's.
The silence on the other side is the finding this section exists for. OpenAI publishes the most complete directory in the sample, and its crawler documentation at developers.openai.com, fetched and read on 15 September 2026, contains the string signature zero times. It names OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User, and carries no publication date. A site owner who follows OpenAI's own link to learn how to recognise OpenAI's crawlers is told about user agent strings and address ranges, and is not told that a key directory exists. That gap is the same species as the one behind three AI crawler tokens that will never appear in your logs: what a vendor operates and what a vendor documents are two different lists, and only one of them reaches the people configuring servers.
| Operator | Bytes | Keys | Key ids | Validity window | Extra members |
|---|---|---|---|---|---|
| OpenAI, chatgpt.com | 243 | 1 | yes | nbf and exp | signature_agent, purpose |
| Meta, meta.com | 250 | 3 | no | none | none |
| Meta, www.meta.com | 250 | 3 | no | none | none |
| You.com, you.com | 90 | 1 | no | none | none |
One of the five keys carried an identifier, and it was correct
The draft ties the directory to the request through a single value. Section 5.2 states that the keyid parameter in a signature must be a base64url JWK SHA-256 thumbprint, computed as defined in RFC 8037 for Ed25519 keys. Section 5.5 then says that a key served at the well-known URI may carry a kid, and that if it does, the kid must be set to that same thumbprint, so a verifier can select a key by matching one string against the other. The draft explains why in a sentence worth quoting: deriving kid from the key material keeps it globally unique and lets a verifier check the directory's own labelling rather than trusting it.
So we checked it. Computing the thumbprint from each published key on 15 September 2026, OpenAI's kid of otMqcjr17mGyruktGvJU8oojQTSMHlVm7uO-lrcqbdg matches the value derived from its own key material exactly. The one directory in the sample that labels its key labels it correctly, which is a small result and the right kind of small: it is the only claim in this post that could have come back false and did not. The other four keys carry no kid at all, so a verifier matching a keyid against them has to compute all four thumbprints itself before it can match anything. That is permitted, since the draft makes kid optional, and it is more work at verification time for every server that ever checks a Meta or You.com signature.
The validity fields tell a second story. OpenAI's key carries nbf 1735689600, which is 1 January 2025 at midnight UTC, and exp 1790048292, which is 22 September 2026 at 03:38:12 UTC, seven days after this measurement. Section 5.5.2 asks operators to rotate by publishing the new key alongside the old one before its use date and removing the old one after it expires, so a directory holding one key that expires in a week is a directory that has to change within the week. Meta's three keys and You.com's key carry no nbf and no exp, which means nothing in those documents says when a key stops being current. The draft is direct about the consequence: removing a key deactivates it, verifiers stop accepting it once their cached copy expires, and the document states that this is not a revocation mechanism and that it defines none.
Cache lifetime therefore has teeth, and the three operators chose three different answers. You.com sends Cache-Control public, max-age=300. Meta sends must-revalidate, max-age=3600. OpenAI sends no Cache-Control header on the directory at all, leaving every verifier to pick its own interval, against a draft example that shows max-age=86400. Five minutes and no header are not small differences when the cache is what bounds how long a withdrawn key keeps verifying. Detection work that leans on this layer is the same discipline described in a bot declares three things and proves one, and the order in which a server applies its checks is the subject of AI crawler detection as an ordering problem.
Key material and computed thumbprints, 15 September 2026
- chatgpt.com kid=otMqcjr17mGyruktGvJU8oojQTSMHlVm7uO-lrcqbdg computed thumbprint matches
- meta.com key 1 of 3, no kid published Iy80tzAcbbb85lkjfKieYYwxAqinLDqDUBXjVMEnI_s
- meta.com key 2 of 3, no kid published PZZZiz0rLbE9RZJ1DyryOFQzzqVwf96kCcr44MMKP0M
- meta.com key 3 of 3, no kid published t8VHBHRHuPs0g3WFixNY2qbjohTnOWMIcd3neRZTkbE
- you.com key 1 of 1, no kid published 5hbjTA0RjjhxRRquk32fSeRWfTgAdSHCZMK1afI1qvI
What a published directory does not tell you about your own traffic
Everything above measures publication. It does not measure signing, and the distance between those two is the most important limit on this post. A published key directory means an operator has made verification possible. It does not mean a single request that reached your server carried a signature, and no figure here is evidence that any did. We read no access log, we received no signed request, and we observed no crawler at any site other than by asking these 51 hostnames for one document each. An operator could publish a perfect directory and sign nothing, and this method would not notice.
The reverse limit matters too. Seven vendors returned 404, and that is a fact about a path on a hostname rather than a verdict on the company. A vendor could be signing requests today and pointing at a directory on a hostname we did not think to ask for, because Signature-Agent carries a URL of the operator's choosing and we cannot see one without receiving a signed request. The well-known path is reserved and the obvious place to look, which is why it is what we asked for, and it is not the only place a key may live. So read the seven as "published nothing at the reserved path on their main hostnames on 15 September 2026", which is what we can support.
There is one connection between this scheme and the layer this scanner does measure, and the draft draws it rather than us. Its appendix lists the use cases the group has discussed, and records that most are served by a URL alone while two need the crawler's identity bound to a domain name. One of those two is robots.txt alignment, described as matching the crawler against a name in the file. That is the whole point for a site owner: a signature is what would let a server know that the crawler it is applying a robots.txt group to is the crawler that group names. Until then the two layers are joined by a string, which is why 79 of 115 sites refused a crawler their own robots.txt allows and why a crawl-delay directive can cost a crawler its verified status.
If you want the answer for your own traffic rather than in general, it is in your logs and not in any scan, including ours. Search inbound requests for a Signature-Input header carrying tag="web-bot-auth", which is a fact about requests you actually received. What an external tool can tell you is the rest of the picture: whether your file allows the crawlers you think it allows, which the robots.txt tester answers per token, and what an AI crawler receives when it fetches a page, which what GPTBot sees prints as it arrives. Those are questions about your site. Web bot authentication is a question about the crawler, and today, for seven of the nine vendors whose tokens this scanner evaluates, the answer is not published.
-
Directory servedVerification possible Four hostnames. Says the keys are published, says nothing about whether any request was signed. -
404 at the pathNothing published here Forty hostnames. A key could still live at a Signature-Agent URL we cannot see without receiving a signed request. -
200 with HTMLNo directory Three hostnames. A status code alone would read as success, which is why the media type and the body were both checked. -
kid matches thumbprintLabel is checkable One key of five. The other four must have their thumbprints computed by every verifier. -
Signed request observedNot measured No access log was read and no crawler traffic was received. This run fetched documents and drew no conclusion about signing.
Lantad
Published .
A crawler that arrives calling itself GPTBot is making an assertion about itself, and nothing in the request proves it. That is the starting point for everything this scanner does with an AI crawler, and it is the subject of an earlier post arguing that a user agent is a claim rather than an identity. Web bot authentication is the proposal that closes the gap by having the crawler sign its request with a private key, so the server can check the signature instead of believing the string.
Common questions
What is web bot authentication?
It is a scheme in which an automated HTTP client signs its own requests with a private key so the receiving server can verify who sent them, rather than trusting the user agent string. It is a profile of RFC 9421, HTTP Message Signatures, and is specified in draft-ietf-webbotauth-httpsig-protocol-00 of 1 September 2026. The signature travels in a Signature header, the covered components and the parameters travel in Signature-Input with the fixed tag web-bot-auth, and the public keys are published in a key directory at /.well-known/http-message-signatures-directory.
Which AI crawler operators publish a key directory?
Four of the 51 hostnames Lantad asked on 15 September 2026 served one: chatgpt.com, meta.com, www.meta.com and you.com, which are three distinct operators. Of the nine vendors behind the 15 AI crawler tokens this scanner evaluates, OpenAI and Meta publish a directory and Anthropic, Perplexity, Google, Apple, ByteDance, Common Crawl and Amazon returned nothing at the reserved path on their main hostnames that day.
Can a website scanner tell me whether a crawler signed its requests?
No. A signature arrives as a header on an inbound request, so it exists only in traffic your own server received, and nothing published on your pages records it. An external scanner can fetch an operator's key directory, which is what this post did, and that establishes only that verification is possible. To answer it for your own site, search your access logs for an inbound Signature-Input header carrying tag="web-bot-auth".
Does a key directory replace IP address verification?
Not yet, because seven of the nine vendors measured here publish no directory, so address ranges and reverse lookups remain the only method available for most AI crawlers. The draft itself notes that a signature does not depend on the requesting IP address, which is the advantage over address lists, and it also states that removing a key from a directory is not a revocation mechanism and that the document defines none.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
What Web Bot Auth actually specifies, and what it does not
The IETF working group on crawler authentication listed ten active Internet-Drafts on 29 July 2026, two of them filed on 19 July. The core protocol draft carries intended status Informational, and every mechanism it describes sits in places an external scan of your site cannot reach.
A user agent is a claim, not an identity
Every per-crawler number you have ever seen, including ours, counts requests that said they were a crawler. Verifying the claim needs an IP check, and most tools do not do one.
LLM crawler detection: a bot declares three things and proves one
Cloudflare's rebuilt bot submission form, launched on 28 August 2026, asks an operator to declare what its bot does, how it uses what it reads, and who runs it. The automated review that follows fetches an IP list, confirms reverse DNS or validates a Web Bot Auth signature, so the one claim a machine settles is identity, and the three that describe intent are taken on the operator's word.
Crawler IP range files carry two dates, and six of thirteen disagree
Thirteen published crawler IP range files from Google, OpenAI, Apple, Perplexity and Microsoft, read on 19 August 2026, each answer the question of when they last changed twice, and on six the two answers differ by between 13 and 942 days.
