AI agent delegation: Cloudflare named ten mechanisms, and 5 of 1,032 robots.txt files carry a field no specification defines

What is AI agent delegation, and what did Cloudflare announce?

The post defines its subject narrowly. Personal AI Agents, it writes, are automations that take instructions from a human and pursue a goal by controlling a cloud computer or browser, usually hosted and maintained by an Operator as part of a managed service. It names Muse from Meta and dots from OpenAI as examples, and it puts agents that run on the user's own device, naming OpenClaw, outside its scope. The distinction it rests on is that a hosted agent is an intermediary, so the site sees the agent instead of the user, and one agent acts for many users.

From that it derives a single principle, quoted here in full because the rest of the document hangs on it: the business relationship between Users and Sites, and each participant's security responsibilities, must be upheld regardless of intermediation. The body then lists fourteen challenges in alphabetical order, from account sharing and friendly fraud through policy circumvention and traffic commingling to uncompensated consumption and user attribution, and states that in nearly every challenge the site bears the primary cost.

What is actually announced is a forum rather than a format. The Trusted Agent Delegation coalition is described as a group of representatives across the ecosystem who will discuss the problem set, examine existing protocols, and bring more well-defined problems to the IETF as needed, with side meetings and workshops intended in future. The post is explicit that real-world solutions will not emerge from fragmented, vendor-specific mechanisms. Three things are offered as available in the week of publication, and all three are Cloudflare product steps rather than protocol: a Personal Assistants traffic category in Security Analytics, per-agent access policies, and an early-access feature that reports whether a personal agent made a given login or sign-up so one abusive account can be mitigated without blocking the agent for everyone behind it.

None of that is a measurement, and none of it is ours. It is a vendor's reading of a problem, reported here with attribution, and it matters to anyone tracking what an AI crawler is and is not because it proposes to change the terms on which automated clients are served.

Why robots.txt cannot say what an agent may do

The sentence in the post that a site owner can act on is this one: robots.txt says where a crawler may go, but is not fully suited to express what agents may do, and sites need a machine-readable way to publish what actions agents are allowed to do. It adds that the IETF is working on preference expression in the AI Preferences Working Group, that the focus of that group is on expressing preferences for use such as training, inference and search, and that there is still not a group focused on the larger need the document describes.

That is a fair description of the file, and the specification says why. RFC 9309, the Robots Exclusion Protocol, defines three records and no more: the user-agent line that heads a group, and the Allow and Disallow lines that carry path patterns inside it. Both of the rule types take a path, so the only question the format can answer is whether a given client may fetch a given URL. There is no vocabulary in it for an action, a purpose, a price, a rate, or a party.

The specification does anticipate extension, and the terms are precise. Section 2.2.4, Other Records, states that crawlers MAY interpret other records that are not part of the robots.txt protocol, giving Sitemaps as its example, and that crawlers MAY be lenient when interpreting them. It then requires that parsing of other records MUST NOT interfere with the parsing of the explicitly defined records, so a Sitemaps line must not terminate a group. Read together, those two sentences mean a site may write any field it likes into the file without breaking anything, and every crawler may ignore it without breaking anything either.

So the gap the coalition names is real at the level of the format, and it is not a gap anyone has to wait for permission to fill. It is also not the first time this file has been asked to carry more than it defines: we measured the 1,102 robots.txt files that never name Anthropic's Claude-Agent token on 5 November 2026, and the robots.txt tester here resolves a path per crawler precisely because the answer differs by token and by nothing else.

The ten mechanisms the coalition names, and where each one stands

The post names ten existing or emerging mechanisms as the material the coalition will examine. Their maturity is not comparable, and a reader deciding what to deploy needs that difference stated rather than smoothed, so every one was opened at its own source on 12 November 2026.

Two are finished standards with years behind them. OAuth 2.0 is RFC 6749, a Proposed Standard published in October 2012, and the post names it for conveying account identities. The Privacy Pass Architecture is RFC 9576, Informational, June 2024, alongside RFC 9577 and RFC 9578, both Proposed Standard and both published the same month; the post names Privacy Pass for establishing trust without identifying the user. A third published document sits underneath the identity work: HTTP Message Signatures, RFC 9421, Proposed Standard, February 2024.

Three are chartered IETF working groups whose output is still draft. Web Bot Auth, which the post calls its preferred method for agent identity, now has a working group of that name and a protocol draft, draft-ietf-webbotauth-httpsig-protocol, at revision 01, submitted 29 October 2026 and expiring 2 May 2027; the older individual architecture draft that we read when we published what Web Bot Auth actually specifies expired on 3 September 2026. The AI Preferences Working Group has two drafts, a vocabulary for expressing AI usage preferences at revision 08 of 14 September 2026 and an attachment document at revision 05 of 19 August 2026, both marked as intending Proposed Standard and neither published, which is the same position we described in the AI preferences standard you cannot deploy yet. WIMSE, the workload identity group, carries ten drafts including an AI Identity Management System draft first revised on 2 November 2026.

Two are individual submissions rather than group work. MOLE, the Moderation of unLinkable Endorsements architecture, is 23 pages at revision 01, submitted 2 November 2026 and expiring 6 May 2027, and the post names it both for rate limiting without user-specific identifiers and for conveying user reputation. Private Access Tokens, which the post describes as an earlier attempt at the same problem and for which it names MOLE the successor, appears in the datatracker as an individual draft last revised in April 2022 that expired the same month.

The last two are outside the IETF entirely. The x402 payment protocol has no working group document; a name search returns five individual drafts, each filed by an author rather than a group. Poppy, named in the post for having a notion of a session identifier, returns no document at all. Neither absence is a criticism: a specification can be useful without being an RFC, and the post does not claim otherwise. It does mean that of the ten, exactly three are published standards a site can rely on being stable this year.

What 1,032 robots.txt files contained on 12 November 2026

If the file cannot express an action, the next question is what sites are putting in it instead. Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 12 November 2026, every request sent as LantadBot/1.0 with redirects followed and a twenty second timeout. 1,114 answered with a 2xx status, 220 answered a 4xx, which RFC 9309 reads as imposing no restrictions, 54 answered a 5xx, which the same specification reads as a complete disallow, and 31 failed at the transport. 1,032 of the 1,114 parsed into at least one user-agent group using the matcher in core/src/robots.ts, and those files carried 76,908 rules between them.

Counting every line in those responses that takes the shape of a field name followed by a colon gives 93,162 directive lines. 58,378 are Disallow and 18,680 are Allow, so 77,058 of them, better than four in five, are path rules. 11,013 are user-agent lines, and 3,196 are Sitemap lines across 881 files, which is the extension RFC 9309 names as its own example of a record outside the protocol. Everything else is a rounding error by comparison: 361 Content-Signal lines on 53 files, the Cloudflare-authored policy whose reach we measured when we found one of ten Cloudflare endpoints reading it, 278 Crawl-delay lines on 145 files, 28 Host lines on 28, and 25 noindex lines on 7.

One artifact in that census is worth stating plainly rather than reporting as a finding, because it inflates a figure that would otherwise look impressive. The responses carry 305 distinct field names, and most of the long tail is not policy at all: font-size appears 53 times, margin 45, display 40, and eleven of the files carry a CSS property as a field name, because the response was a styled web page rather than a rules file. We measured that failure directly on 3 November 2026 and found 34 of 1,104 sites answering a crawler with an HTML page, and 82 of today's 1,114 2xx responses parsed into no user-agent group whatsoever. A census of what arrives at the client is not the same thing as a census of what sites meant, and the method behind every figure here is the ordinary one: fetch, parse with the shipped parser, count what is there.

Five files already name an agent policy no specification defines

The interesting part of that census is at the bottom of it. Five of the 1,032 parsed files carry a field whose job is to point an agent at a policy document, and four distinct field names do it. Every one was re-read at its own source the same day.

Two sites publish a field named Agentmap. zapier.com carries the line Agentmap followed by the address of a file at /.well-known/ai-catalog.json on its own host, and blindpay.com carries the same field pointing at the same path on its own. Requesting Zapier's returned HTTP 200 with a content type of application/json and 10,052 bytes, and the document inside is exactly the thing the coalition says does not exist: it declares a specVersion of 1.0, identifies its host, points at a documentation URL, carries a trust manifest, and lists entries each with an identifier, a type, a URL and a set of representative queries. The first entry describes a Model Context Protocol server and the second a WebMCP manifest for browser-resident agents. Whatever one thinks of the format, it is a machine-readable statement of what an agent may do here, advertised from the one file every crawler already fetches.

Two more publish a field named schemamap. fly.io carries schemamap followed by the address of an XML document on its own host, and netlify.com carries the same field pointing at one on its www host. Fly's answered HTTP 200 with a content type of application/xml and 19,342 bytes. All four of those files also carry a Content-Signal line setting search, ai-input and ai-train to yes, so every site here that adds a pointer of its own is also using the one published policy vocabulary beside it.

The fifth is the most direct and the least likely to be read. thefurrow.tv serves a six line file whose entire content is a group headed by User-agent: dollar sign followed by llm, one Allow rule for the root, a Citation-Policy line reading allow-with-attribution, a Training-Window line reading 30d, and a Sitemap. Running it through the parser this product ships produces two notes, one for each unknown field, recording that both were ignored. Resolving the site root for each of the 15 crawler tokens in the registry behind our AI crawler reference returns allowed with no group matched for all 15, because the file's only group is headed by a token none of them is. The site has written a citation policy and a training window, in a file a crawler will fetch, inside the one group no documented crawler selects.

Five of 1,032 is not a trend, and it is not being reported as one. It is the same shape we found when we probed for the one discovery path a standards registry does list and 1 of 1,419 sites published an agent card on 19 September 2026. Sites that want to say more than a path rule are already saying it, each in a vocabulary of their own, which is the case for a standard rather than against one, and it is the part of this story a running measurement can see from outside.

What to check on your own site, and what this did not measure

Nothing in the coalition announcement requires a site owner to do anything this week, and the honest summary of the standards position is that three published standards and three working groups are not yet a way to tell a site who an agent acts for. Three checks are worth running anyway, because each one is answerable today and each is a precondition for anything the coalition eventually ships.

The first is whether your robots.txt parses at all. 82 of the 1,114 files that answered with a 2xx status here resolved into no user-agent group, which means every rule in them applies to nobody; a file served as HTML is the common cause and it is invisible from a browser. The second is whether the group you think you wrote is the group a crawler selects. Group selection matches on the product token and nothing else, so a heading no crawler matches makes the rules under it inert, exactly as it did for the citation policy above. The third is whether the file and the response agree, since an edge rule can refuse a request the file permits, and the only way to know is to ask as the crawler and read what comes back, which is what the page describing our own bot commits us to doing.

What this post did not measure needs naming too. No agent and no crawler was observed, because this scanner reads sites rather than server logs, so nothing here says whether any engine fetches Agentmap, reads a citation policy, or honours a training window; all three remain claims by the sites that published them. Each file was read once, on 12 November 2026, so a rule added yesterday and one standing for years look identical. The corpus is a stratified sampling frame built for crawlability rather than a census of the web, so five of 1,032 is a count inside this sample and not a rate for the web. The field census counts lines that take the shape of a field, which is why CSS properties appear in it, and it is reported that way on purpose.

One gap in the sourcing belongs in the open as well. openai.com refused our requests with HTTP 403 throughout 12 November 2026, so OpenAI's own crawler documentation was not opened for this post and nothing here is claimed about it, including about the dots agent the Cloudflare post names. Every other source was read at its publisher on the date given. That is the standard every figure on this site is held to, and the reason AI visibility is worth measuring at all is that the alternative is believing a file rather than a response.