An IETF draft makes an AI crawler's compliance claim falsifiable, not true

What a publisher cannot determine about a crawler that already left

The document opens by naming the problem more precisely than most writing about AI crawlers manages. Both halves of the working group's charter, it observes, are concerned with the publisher side of the exchange: a publisher can state a preference and can make that statement discoverable. What the publisher cannot do, in the draft's words, is determine whether the statement was read, which interpretation was applied to it, or what the processing entity subsequently did with the asset.

Everything a site owner can check today sits on one side of that line. RFC 9309 standardises robots.txt and states in its first section that the rules are not a form of access authorization, which is the specification saying out loud that it describes a request rather than a lock. The vendor documentation has the same shape. OpenAI documents four tokens, OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot, each with a stated purpose. Anthropic's crawler page, carrying a last updated date of 7 April 2026, documents ClaudeBot, Claude-User and Claude-SearchBot and states that its bots respect do not crawl signals by honouring industry standard directives in robots.txt. Perplexity documents PerplexityBot and Perplexity-User and says of the user triggered fetcher that since a user requested the fetch, this fetcher generally ignores robots.txt rules.

Read those three pages together and the pattern is commitments about fetching, published by the party doing the fetching, with at least one documented exception among them. None of the three produces an artefact afterwards. A site owner who wants to know whether a commitment held has to go and measure it, which is what a readability scan of an AI crawler request does from the outside, and even a flawless outside measurement stops at the response. It can establish what was served. It cannot establish what was kept.

The gap is not theoretical, and the effort required to close it even partially is worth looking at. A canary token study from Duke University, the University of Pittsburgh and Carnegie Mellon, posted to arXiv on 13 May 2026, served unique tokens to each visiting scraper across purpose built websites and then queried 22 production AI chatbot systems to see which tokens came back in their answers. We covered the result when it found that a robots.txt block did not stop 12 of 18 measurable chatbots from returning the sites' content. The method deserves attention separately from the finding: to answer the question of what an AI system did with a page, three universities had to build decoy websites and interrogate the models. That is currently the state of the art in verifying a preference, and it is not something a publisher can run on its own site on a Tuesday afternoon.

What an AI Usage Compliance Record actually contains

Section 4 of the draft defines the record. It is an information model rather than a wire format, and it carries nine required members.

Three of them pin the asset. The member named asset-digest is a digest of the representation of the asset as received, computed over the octets of the response body after decoding of any content codings, which is a careful way of saying the bytes that arrived rather than the bytes on anyone's disk. The member named asset-locator is the URI the asset was retrieved from, and the draft is explicit that it is informational with respect to identity: the digest identifies the content, the URL only says where it was found. The third, retrieval-time, records when the response was received as an integer number of seconds since the epoch.

Three more pin the preference. The member named preference-source says where the preference expression was obtained, drawn from a registry the draft defines. Its companion preference-source-digest is a digest over the octets of that source as retrieved, and it is the member that would let a later argument about what your file said on a given day settle rather than continue. The third, preference-observed, is the set of preference terms the entity determined to be applicable to that asset. Together those three record not only the file but the reading of it, and the reading is where a surprising share of robots.txt disagreements actually live: two parsers looking at one file and resolving different rules for the same token, which is why the robots.txt tester takes a crawler name as input rather than answering yes or no.

The last three are the interesting ones. The member named usage-category is the usage category the processing entity assigned to the asset, and the draft describes it in a single line as the member that makes the record falsifiable. The member named disposition summarises the relationship between the preference observed and the usage assigned. The member named agent-identity is an identifier for the signing identity under which the retrieval was performed, which is the hook into the separate work on proving a request came from who it claims rather than from anyone with a text editor and a user agent string. Two optional members follow: a timestamp for the preference source, and a retention statement.

Compare that shape with what a robots.txt line carries, which is a path pattern and a verb. The preferences vocabulary adds two usage categories, train-ai and search, each set to yes or no, and we covered its state alongside the expired attachment draft in the AI preferences standard you cannot deploy yet. None of those carries a timestamp, a digest or an identity, because none of them is a record of an event. This draft is the first document in the area whose subject is the event rather than the rule.

How one signature can cover a day of fetches

One record per fetch is a great many records. A crawler operating at web scale retrieves millions of assets a day, and signing each record on its own would produce a volume of signatures nobody would ever verify. Section 5 handles that the way certificate transparency does. Records are aggregated into a Merkle tree, the root of each epoch is signed, and the result is a Compliance Attestation covering an arbitrary number of records at constant cost. An epoch must not exceed 24 hours. Leaves are ordered by a tuple of the asset locator's authority component, then the locator, then the digest, compared as octet strings, and that ordering is what makes one publisher's records a contiguous range in the tree rather than a scatter through it.

The attestation is a COSE_Sign1 structure whose payload carries the epoch identifier, the epoch's start and end times, the Merkle root, the leaf count, the hash algorithm identifier, and the identifier of the immediately preceding epoch's attestation. That last field is what turns a pile of signed statements into a chain. An entity that quietly drops an epoch leaves a hole a verifier can see, which is a different and better property than an entity that simply publishes less.

Section 6 defines the audit itself. A publisher asks for the records whose asset locator authority component matches an origin over which it has demonstrated authority, and receives them together with proofs of their inclusion in that epoch's attested root. Because the leaves are sorted, the response can prove it is complete rather than merely consistent: the draft requires inclusion proofs for the leaves immediately preceding and following the returned range, which establishes the boundaries of that range and so shows that nothing inside them was held back. That completeness property is the part most transparency schemes get wrong, and getting it right is the strongest engineering in the document.

Section 7 says where any of this is found. A processing entity producing attestations publishes a JSON document at the well known path /.well-known/ai-usage-compliance carrying its verification keys as a JWK Set, the most recent attestation identifier with a means of enumerating prior ones, and the audit interface URI, with a transparency service identifier as an option. The domain that document sits on must be the same domain the entity's request signing keys are published from. Section 8 then makes the composition explicit: where an entity already signs its requests using HTTP Message Signatures, the two mechanisms compose directly and, in the draft's wording, SHOULD be deployed together, using the same key or publishing both keys in one JWK Set so that their common control is evident.

Taken together it is a coherent design, and the coherence is a reason to read it closely rather than to assume it changes anything about AI visibility yet. The draft lists thirteen sections, ending with security, privacy, IANA considerations and open issues, and none of the thirteen is an implementation status.

Falsifiable is not the same as true

One sentence in the abstract should govern how this draft is reported anywhere. The mechanism, it says, is deliberately confined to evidence: it makes claims of compliance falsifiable and non-repudiable, and takes no position on the legal effect of any preference or any record.

Falsifiable is a precise word and it is not a synonym for true. The usage category inside a compliance record is assigned by the processing entity itself. Nobody stands behind the model checking that a page filed under search was not also used for training. What the signature buys is that the entity cannot later say it never made the claim, and what the digests buy is that the claim is anchored to specific bytes and a specific preference file rather than to a general assurance about company policy. If anyone can later show that same asset inside a training set, the record becomes a contradiction with a signature on it. That is a real improvement on the present position, where a vendor's account of its own conduct is a paragraph on a documentation page that can be edited without trace.

It is still a self-report, and the draft says so by listing what it excludes: the legal effect or enforceability of any preference in any jurisdiction, licensing and compensation and commercial arrangements, enforcement, the preference vocabulary itself, and any attestation about model internals, training procedures, dataset composition, or whether an asset is present in a model. That final exclusion is the one that matters to anyone hoping this settles the training argument. It does not settle it. It records what an entity says it did, in a form where saying something different later has a cost.

A second boundary deserves attention even though the draft does not dwell on it. The scheme describes what a participating entity publishes. It says nothing about a client that publishes nothing, and participation here is voluntary in exactly the way robots.txt compliance is voluntary. An operator willing to ignore a Disallow line is not an operator that will stand up a transparency service and invite audits of it. We reported the measured version of that problem in sites whose GPTBot ban was served a 200 anyway. The population this mechanism can describe is largely the population that was already behaving well enough to be describable, which is a familiar shape in this field: the RSL licensing directive binds only clients that have adopted RSL, and its own specification states that the directive does not modify access permissions at all.

None of that makes the draft uninteresting. It makes it a document about evidence rather than a document about control, which is precisely what it says it is on its first page, and it is far more useful read that way than as the thing that finally holds AI companies to account. Most of what circulates as generative engine optimization advice would be improved by the same distinction between what a mechanism records and what it compels.

What you can check on your own site today

Nothing above is deployed, and the sensible thing to do with a proposal at revision 00 is read it rather than plan around it. What is available today is narrower and completely determinate, which is a trade most site owners would take if anybody framed it that way.

Three things about your own site are measurable from outside, repeatably, with no cooperation from any vendor. The first is how your robots.txt resolves for a specific named token, which is a parsing question with one correct answer per token rather than a matter of opinion. The second is what a client without a browser actually receives: the status code, the headers and the bytes. Two layers decide whether AI can read your site, and the file you wrote is the weaker of them, because an edge rule can return a 403 to a request your robots.txt welcomes and nothing reconciles the two automatically. The third is how much of the text a person reads is present in that response before any JavaScript executes, which is prose parity and is the largest weighted component of the score described on the methodology page.

Those three are worth separating from the question this draft addresses, because they fail in different ways. Access failures are binary and loud once somebody looks. Rendering failures are quiet: a 200 carrying an empty shell measures as success on every check that stops at the status code, which is why a scan that reads only headers reports a healthy site that no answer engine can quote. Provenance failures, meaning what happened to the content after it left your server, are not measurable by anyone outside the vendor today, and this draft is the first serious attempt to change that.

It also explains why this site withholds a grade rather than guessing when something could not be measured, which we set out in why we withhold a grade. The same discipline applies to a post about somebody else's specification. Lantad has audited no vendor, holds no compliance record, and has no view into what any AI system retains after a fetch. What a scan can see is one side of one request, and saying that plainly is more useful to somebody making a decision than an implied claim to see both sides.

If the draft is adopted, the thing to watch is not the cryptography, which is conventional and well chosen. It is whether any processing entity publishes a well known document at all. A transparency scheme with no participants is a specification rather than a check, and the answer to that will arrive as a file at a fixed path on a vendor's domain, which is the one part of this whole design that anyone will be able to verify for themselves the moment it exists.