AI usage preferences stopped inheriting, and the attachment draft is back

What changed in the two AI preferences drafts on 19 August 2026

The working group publishes two standards-track documents. The vocabulary defines the categories of use and the values a preference can take. The attachment draft defines where a statement of preference is written so that a client picking up a page can find it. Neither is an RFC yet, and the pair only works together: the vocabulary supplies the words and the attachment draft supplies the two places to put them.

Both revisions landed on 19 August 2026 and both now expire on 20 February 2027, which is the first time the two documents have shared an expiry. That matters more than it sounds. An Internet-Draft that passes its expiry date is not withdrawn, but it stops being a current document, and for 110 days the only specification saying what a Content-Usage line in robots.txt means was a document in that state. The attachment draft is authored by Gary Illyes of Google and Martin Thomson of Mozilla; the vocabulary is authored by Paul Keller and Martin Thomson.

The attachment draft's abstract was rewritten, and the rewrite is the clearest statement yet of the document's scope. It now says that the document defines attachment methods using the Robots Exclusion Protocol and HTTP header fields. Two mechanisms, named in the abstract, and no third. The previous revision opened on the softer ground that content creators and other stakeholders might wish to signal their preferences. A site owner reading the new abstract learns in one sentence that the places to write a preference are the file at the root of the host and a response header, which are the same two surfaces that already decide whether a crawler is allowed in at all. Our own robots.txt tester reads the first of those, and the crawler tokens each vendor publishes are listed on the free AI crawler reference.

The vocabulary itself did not grow. It still defines exactly two categories of use, AI Model Training with the label train-ai and Search with the label search, and exactly two values, y for allow and n for disallow. That is unchanged from draft-ietf-aipref-vocab-06 and it is the same two-category, two-value shape reported in an AI usage preference cannot name a party. What changed is how those two categories relate to each other, and to any category a future extension might add.

Why a preference for a broad category no longer covers a narrow one

The previous vocabulary revision carried a paragraph describing how categories nest. It said that some categories describe a proper subset of the usages of other categories, and that a preference stated for the more general category applies if the preference is unknown for the more specific category. A worked example followed it: allow the broad category, say nothing about a narrower one that sits inside it, and the narrower usage is allowed too, unless you disallow it explicitly.

That paragraph is gone from draft-ietf-aipref-vocab-07, along with its example. The resolution process went with it. The old text set out three steps: take an explicit preference if there is one, otherwise recurse into the enclosing category and take its answer, otherwise return unknown. The new text has two. If the statement of preference contains an explicit preference for the category being asked about, either to allow or to disallow, that is the outcome. Otherwise the preference for that category is unknown. There is no middle step any more.

The consequence is worth stating plainly, because it is the opposite of how the robots.txt rules in the same file behave. In robots.txt a group headed by the wildcard covers every crawler that has no group of its own, which is the mechanism examined in robots.txt names the wildcard, not the AI crawler. A statement of usage preference now has no wildcard of that kind. Coverage is not inherited and cannot be implied. Every category you have an opinion about has to be named, and the document is explicit that it takes no position on what a recipient should assume when the answer comes back unknown.

With only two categories defined, nothing in the current vocabulary actually nests, so today the change costs a site owner nothing. It is a change about the future shape of the vocabulary rather than about the words in it now. That shows in the extension rule, which was inverted rather than edited. The previous revision said future extensions MUST NOT introduce categories that include existing ones, so a new category could be a subset of an existing category but never a superset. The new revision drops that prohibition and requires instead that the definition of an extension must define how any potential overlap between usage categories is resolved. Overlap moved from being forbidden to being permitted and documented, which is a reasonable trade once the automatic resolution rule has been removed, since there is no longer a general rule for an extension to contradict.

One definition was also narrowed. AI Model Training read, in the previous revision, as the act of using an asset in the production or refinement of an AI model that can generate content in one or more modalities. It now reads as using an asset to modify the learned parameters of an AI model that is used to generate synthetic content in one or more modalities. Modifying learned parameters is a narrower act than production or refinement, and a preference against train-ai now speaks to that act specifically. Whether retrieval at answer time falls inside or outside it is not something the document settles, and it is the distinction that decides whether a page can be cited by ChatGPT while being kept out of a training corpus.

Where an AI usage preference is written, and what has to read it

The attachment draft defines two places, and it is worth being precise about both because they behave differently. The first is a response header field named Content-Usage, a structured field dictionary that carries the preference for the content of that particular HTTP message. The draft is careful that this is representation metadata rather than a property of the resource, and it notes that the field has no special effect on caching.

The second is a new rule inside robots.txt. The draft extends the rule grammar of RFC 9309 with a content-usage rule that takes an optional path followed by a statement of preference, so a line reads Content-Usage: train-ai=n, or Content-Usage: /ai-ok/ train-ai=y when it is scoped to a path. Paths match by longest prefix, the same way Allow and Disallow do, and the ordering of rules inside a group carries no meaning, so Content-Usage lines can be interleaved with the access rules around them. Anyone who has watched a non-standard robots.txt directive be ignored, as happened to the Cloudflare signal examined in the Content-signal line asks and the Disallow lines block, will recognise why an extension to the RFC 9309 grammar is the substantive part of this document.

The draft calls this a two-stage arrangement that distinguishes acquisition from usage. Acquisition relies on the Allow and Disallow rules, and usage preference relies on the Content-Usage rules. That separation is the whole design, and it is the same division of labour that separates whether a crawler can reach a page from what happens to the page afterwards. Lantad measures the first stage and nothing in the second: our scan reads robots.txt per crawler token and fetches the page, which is the ground covered in our methodology, and it forms no view about downstream use.

One timing property carries over from RFC 9309 unchanged and deserves a mention because it surprises people. A crawler may cache robots.txt for up to 24 hours, so an edit to a Content-Usage line is not visible until that cache expires and the affected resource is fetched again. The draft says so directly: updates to robots.txt do not retroactively apply to resources already obtained. A preference is attached at the moment of the fetch, which means the copy of the file that was current then is the copy that governs. That is the same lag, with the same consequence, described in when a robots.txt edit reaches a crawler, and it is one reason the mechanism sits closer to a declaration than to a control.

A Disallow expresses no usage preference at all

This is the clause with the most practical bite in the whole document, and it survived the revision intact. Usage preferences apply only to those resources that can be crawled according to the Allow and Disallow rules. No preferences are implied for resources that are disallowed.

Read that against how site owners actually use robots.txt today. The common defensive posture is to disallow an AI crawler outright, which is what most published guidance recommends and what the majority of blocking sites do. Under this draft a disallowed path carries no usage preference whatsoever. Not a preference against training, not a preference against search: nothing, because the mechanism that would express one is scoped to paths a crawler is permitted to fetch. The preference and the block are alternatives rather than layers, and a site that blocks has opted out of saying anything.

That is coherent within the draft's own logic, since a preference is metadata attached to content at the moment the content is handed over, and content that is never handed over has nothing to attach metadata to. It is nonetheless the opposite of the intuition most people bring to the file. It also means the strongest declaration and the strongest block cannot be combined on the same path, and the existing evidence on whether blocking works is mixed enough to matter: a canary study we reported in a robots.txt block did not stop 12 of 18 AI chatbots found content still reachable through systems fed by search crawlers rather than by any AI crawler of the vendor's own.

The draft's own worked example makes the arrangement concrete, and it includes a table of three paths that is worth reproducing because it is the specification stating its own outcome rather than anyone's interpretation of it. A file allowing everything except /never/, carrying Content-Usage: train-ai=n at the group level and Content-Usage: /ai-ok/ train-ai=y for one subtree, produces three different answers for three paths. Note also that a preference is broadcast rather than addressed: there is still no way to name a party in a statement of preference, which is why the separate proposals covered in Earmark signs the preference into the content and compliance records are falsifiable, not proof exist at all.

Access conflicts resolve permissively and usage conflicts resolve restrictively

The two stages disagree about what to do when rules collide, and the draft flags the disagreement itself rather than leaving a reader to notice it. Where two Content-Usage rules have identical paths and conflicting preferences, they apply separately through the combining process in the vocabulary. The draft then adds a note: this differs from the Allow and Disallow rules, where a conflict leads to the more permissive option, allowing crawling.

The combining process it points at is unambiguous in the direction it goes. If any statement of preference says a usage is disallowed, the result is disallowed. Otherwise, if any statement allows it, the result is allowed. Otherwise unknown. The vocabulary states the intent in one sentence: this process ensures that the most restrictive preference applies. So a tie inside one file is broken toward crawling on the access rules and toward refusal on the usage rules, in the same group, in the same file, read in the same pass.

Two rules in one file resolving in opposite directions is not a defect, because the two stages are answering different questions and each default is the safe one for its own question. It is a thing to know before writing conflicting lines, though, and it is a genuine complication for anyone building a parser. It also shows up when preferences arrive from more than one place at once, since a header and a robots.txt line can both speak about the same content, and the restrictive rule decides between them. RFC 9309 itself, in a clause we have quoted before in robots.txt 404 and 503 are opposites, is blunt that its rules are not a form of access authorization, and the usage layer inherits that limitation entirely.

There is one more asymmetry between the two documents. The vocabulary carries a note to readers stating that its contents do not reflect consensus of the working group either in whole or in part, and that the document is published solely as a basis of further discussion, with the same warning repeated at the head of its statements-of-preference section and its vocabulary definition section. The attachment draft carries no such note anywhere: the word consensus does not appear in it. That asymmetry is a reasonable signal of where the argument still is. The mechanics of writing a line into robots.txt appear more settled than the meaning of the words that go in it.

What the drafts say they do not do, and three references that do not resolve

The vocabulary replaced its paragraphs about enforcement with an explicit list of what the specification does not address, and the list is more useful to a site owner than the prose it replaced. It says the specification does not ensure that preferences are followed; does not address if, how, or when preferences should be followed or not followed; does not address the technical, legal, contractual or other mechanisms that might create a stronger requirement to follow them; and does not consider situations or purposes that might justify following or not following them. It closes by saying that an entity receiving usage preferences has a choice whether to follow them, and that this specification does not determine how that choice is made.

A separate clause in the attachment draft moved in the same direction. The previous revision said servers MUST retain any preferences associated with a request where that content is used to answer later requests. The new revision says servers can use any preferences in that situation, and adds that servers or resources would need to be updated to understand the field and then decide to use the preferences. A normative requirement in a standards-track document became a description of something an implementer might choose to do. The pattern across both revisions is consistent: fewer obligations, clearer scope, and no claim to be a control. That is the same honest limit reported for the licence directive in RSL writes a licence into robots.txt and says it changes no access and for the reservation protocol in three crawler documentation pages name robots.txt, and none names TDMRep.

Because these are drafts rather than finished specifications, some of the wiring between them has not caught up, and three cross references in the pair published on 19 August 2026 do not resolve. The attachment draft's section 3.1 sends a reader to section 7.1 of the vocabulary for the process that handles conflicting preferences. Section 7 of the vocabulary is Security Considerations and has no subsections; the combining process is at section 5.1. Section 3.2 of the attachment draft cites sections 6 and 7 of the vocabulary for parsing and processing rules, where section 7 is again Security Considerations. Third, the vocabulary's own reference entry for the attachment draft carries the vocabulary's title rather than the attachment draft's, and points at revision 00 while dating it 19 August 2026, which is the date of revision 05.

There is a fourth, smaller one in the attachment draft's worked example. The prose beneath the example file says a crawler using the second group would apply usage preferences of ai=y, while the line in the file above it reads train-ai=y, and ai is not a label the vocabulary defines: its table maps AI Model Training to train-ai and Search to search, and nothing else. None of these four is a reason to distrust the design, and all four are the kind of thing that gets tidied before an RFC. They are recorded here because a site owner implementing from these documents today will hit them, and because the whole point of naming a source a reader can open is that the reader can check it. What Lantad checks is narrower and unchanged: whether an AI crawler can obtain the page at all, which is the question behind what AI visibility means and the one our scanner answers. Nothing in these drafts is measured by any scan we run, and no crawler documentation we track mentions a Content-Usage line yet.