AI agent traffic: Anthropic's hosted browsers identify as Claude-Agent, and 0 of 1,102 robots.txt files name it

What is AI agent traffic, and why does robots.txt not describe it?

The phrase covers requests made by software acting on a named person's behalf, in the moment, rather than by a crawler gathering pages for later use. The distinction matters because the two things want different answers. A site that blocks a training AI crawler has made a decision about its content. A site that blocks an agentic browser has made a decision about a customer who asked a model to do something on that site, and may not realise those are separate decisions at all.

RFC 9309, which standardises robots.txt, is built around the product token: a crawler reads the file, finds the group whose user-agent line names it, and applies those rules. That design assumes the client knows which token it is and that the site has had a chance to write something about it. Neither assumption survives contact with an agentic browser, which is a real browser carrying a real browser's User-Agent string with a vendor token added to it. The structure of the file is fine. The problem is that nobody has written the line.

The vendors are not uniform about this, and the hero table above sets out what each of their own pages says. OpenAI's crawler documentation says of ChatGPT-User that "because these actions are initiated by a user, robots.txt rules may not apply". Perplexity's bots guide is blunter about Perplexity-User: "since a user requested the fetch, this fetcher generally ignores robots.txt rules". Anthropic's crawler article states that its bots respect do not crawl signals by honouring industry standard directives in robots.txt, and lists three bots under that heading. Claude-Agent is not one of the three. It is documented somewhere else, and the page that documents it says robots settings are managed separately.

That is three vendors giving three different answers to the same question, which is the context in which Cloudflare's per-agent policy makes sense as a product. It is also why a reader should treat a user agent as a claim rather than an identity before building anything on top of it.

How Anthropic-hosted browsers identify themselves

Anthropic's crawler article points readers elsewhere for browsers it hosts, naming Claude Duet and the token Claude-Agent in the same breath. The article it points at is the one that matters here, and it is specific in a way that helps. The requests carry, in its words, the User-Agent header of the browser itself "with the addition of the token Claude-Agent". Its worked example shows Claude-Agent/1.0 inside an otherwise ordinary Chrome string, and it tells site owners to match on the token rather than the full string because the other parts can change.

That advice is correct and it is also an admission. A string you must substring match is not an identity, it is a courtesy, and the same page says so plainly: anyone can send the header, so a site should check that the request came from a published address. We have written before about the tokens that never appear in a log at all, which is the opposite failure. This one appears, but only inside a header designed to look like a browser.

Two consequences follow for anybody writing rules today. The first is that a rule keyed on the string ClaudeBot will not fire, because Claude-Agent is a different token and the matching in RFC 9309 is a prefix test against the client's own token rather than a search of the header. The second is that the token is new, and it is missing from the list this site ships, which the last section of this post is about. Anthropic's separate search crawler had the same problem when we measured it: 25 of 1,024 robots.txt files refused Claude-SearchBot and 55 of the 67 files that kept it never named it, letting it fall through to a wildcard group that happened to permit the root.

What 1,102 robots.txt files say about Claude-Agent

Here is the measurement. Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 5 November 2026, as LantadBot/1.0 with redirects followed and no JavaScript executed. 1,102 answered HTTP 200 and parsed into at least one user-agent group using the shipped parser in core/src/robots.ts. Those files carried 2,218 distinct user-agent tokens between them.

Six of those tokens contain the string claude: claude, claude-code, claude-searchbot, claude-user, claude-web and claudebot. The token Anthropic documents for its hosted browsers is not among them. Not one file in the corpus writes claude-agent in a user-agent line.

The verdicts follow from that absence. Evaluating the site root for Claude-Agent against all 1,102 files, 1,018 of them leave the token falling through to a wildcard group, and 13 of those wildcard groups disallow the root, so those 13 sites refuse the agentic browser only because they refuse everything that is not specifically excused. A further 83 files hold no group that applies to Claude-Agent at all, not even a wildcard, which under RFC 9309 means there is no rule to apply and the answer is allowed. Exactly one file has a named group that binds the token, and that file is the subject of the next section.

The comparison that gives the number its meaning is with the crawlers from the same vendor. 178 of the 1,102 files have a named group that binds ClaudeBot, 50 bind Claude-User and 46 bind Claude-SearchBot. Site owners in this corpus are plainly willing to write Anthropic tokens. They have written six different ones. They have not written this one, because it is newer than their files.

A rule naming ClaudeBot does not reach Claude-Agent

This is the part worth acting on, because it is where a site owner's intention and their file come apart. 84 of the 1,102 files name ClaudeBot in a user-agent line and disallow it at the site root. That is a clear decision, written deliberately, about Anthropic. Evaluating the same 84 files for Claude-Agent returns allowed on 80 of them. Four refuse it. Three of those four do so through a wildcard group that was already blocking everything, and the fourth is the single file in the corpus whose named group reaches the token, which the rest of this section is about.

The mechanism is the matching procedure rather than anything exotic. A group token applies when it is a prefix of the client's token, so claudebot is not a prefix of claude-agent and the group never applies. Our own robots.txt tester will show the same result on any file you paste into it, and the behaviour matches what we found when we measured 2,209 crawler tokens across 1,004 files.

One file in the corpus does bind the token, and it does so by accident in a way that is instructive. lapresse.ca writes 52 user-agent tokens, among them claude, claudebot, claude-searchbot and claude-web, and it disallows the root for all of them. Because the bare token claude is a prefix of claude-agent, that group applies, and the agentic browser is refused. The site did not write a rule about Anthropic-hosted browsers. It wrote a rule about a vendor name rather than about a product, and that choice aged better than the specific tokens beside it did.

That is not a recommendation to write vendor prefixes instead of product tokens. It catches things you have not considered, which cuts both ways: where no more specific token is present, a bare claude group also binds Claude-SearchBot and anything else Anthropic ships under that name, which may not be what a publisher who wants citations in Claude intends. On this particular file the point is moot, because it names claude-searchbot explicitly and refuses that too. It is simply the one case in 1,102 where an existing file already says something, and it says it for the wrong reason.

SAIL moves the check from the user agent to the address

If the header cannot be trusted and the file does not mention the token, the remaining control is the address the request came from. Anthropic's documentation points at a path for that, and there is a specification behind it. Draft-kelley-webbotauth-sail-00, titled SAIL: Source Address Identity List, was published on 22 October 2026 by Kevin Kelley of Anthropic as an individual Internet-Draft in the Web Bot Auth working group, and the datatracker page states plainly that it is not endorsed by the IETF. It expires on 25 April 2027.

The draft defines a metadata document served at the well-known path /.well-known/sail in which an operator of automated clients publishes the address ranges its services send from. It uses TOML rather than JSON, and says why in one sentence: so that the document can carry comments. Two optional assertions give the format its teeth. A service marked exhaustive is a statement that it sends from nowhere else, which lets a site treat a request claiming that service from any other address as refuted. A prefix set marked exclusive is a statement that nothing else uses those addresses.

Lantad read Anthropic's document at that path on 5 November 2026. It is 719 bytes, served as application/sail+toml, declares the schema draft-kelley-webbotauth-sail-00 and a modified time of 22 October 2026, and carries an expires_after of 3600 seconds. It holds one prefix set, named agentic-browser-use and marked exclusive, and one service, named Claude-Agent and marked exhaustive. The prefix set lists 9 IPv4 entries and no IPv6 entries, and none of the 9 is written with a CIDR mask. The document URL is https://claude.com/.well-known/sail, written here as plain text because that host is not on this site's registered outbound list.

Two things about that are worth a site owner's attention. The first is the hour. An expires_after of 3600 seconds means an allowlist built from this file is stale after an hour, so this is a fetch your edge does on a schedule, not a list you paste into a config. The second is that Anthropic publishes two first-party files in two different shapes: the crawler addresses at https://claude.com/crawling/bots.json, which Lantad also read on 5 November 2026 and which holds 14 prefixes, all carrying a mask, under a creationTime of 20 October 2026, and the agentic browser addresses in the TOML document above. Same vendor, same question, two formats, two paths, and no IPv6 in either.

What we got wrong, and what this does not measure

Our own crawler registry in core/src/bots.ts holds 15 tokens. Claude-Agent is not one of them, and neither is Claude-Web, which appears in 74 of the corpus files. That is a gap in the thing this site sells, so it belongs in the post rather than in a backlog. Until the registry carries the token, a Lantad scan will not tell you whether your file reaches Anthropic's agentic browser, and the only reason this post can tell you is that the measurement evaluated the token directly rather than looping over the registry. We publish the method and the registry the scanner actually uses so that a gap like this is checkable from outside rather than taken on trust, which is the same reason we withhold a grade when a page could not be measured.

Now the limits, and they are substantial. This measurement reads files. It does not observe a single request from Claude-Agent, so it says nothing about whether Anthropic's hosted browsers honour a rule when one exists. The one file that binds the token tells you what the rule says, not what the client does with it. Behavioural work of that kind is a different exercise, and when we have run it the results were mixed: every web agent we tested was identifiable and robots.txt stopped one of twelve.

It is also a snapshot of one file per host, taken on one day, at the site root only. A site may well permit the root and refuse a path that matters more, and this measurement would not see it. The corpus is the two committed seed files in this repository, stratified by platform and by industry, which is a sampling frame rather than a census of the web, and 317 of the 1,419 hostnames did not return a parseable file at all on the day.

Finally, Cloudflare's announcement is Cloudflare's. Lantad has not tested the Personal Assistants category, the per-agent policy or the analytics behind them, and the post above reports them from the announcement rather than from any measurement of our own. One observation about it is ours, and it is only an observation about dates: Cloudflare's bot concepts documentation carried a last updated date of 1 July 2026 when we read it on 5 November 2026, and still describes the three behaviours Search, Agent and Training without a Personal Assistants class. A product shipped yesterday and a documentation page from July are allowed to disagree. It is worth knowing which one your AI visibility work is relying on, and the same caution applies to the ads-based blocking default we measured in September.