Seven of nine AI coding agents arrived without a product name

What the study logged, and on what page

The design is small and tightly controlled, and both properties matter when reading the result. Borysenko built one publicly reachable developer documentation endpoint on Node.js and Express, served real API documentation from it, and pointed every tool at the same URL. The endpoint required no authentication and, in the paper's words, did not rely on client-side rendering, so nothing a client saw depended on running JavaScript. That removes the most common reason an automated fetch comes back empty and isolates the question the study is actually asking, which is what each client sends rather than what each client manages to read.

Two discovery files sat on that endpoint for the whole measurement period. The robots.txt allowed every user agent everything except a /guide path and declared a sitemap. The llms.txt described the portal and linked its main documentation page and an OpenAPI 3.0.1 specification, which is close to the shape the llms.txt specification describes. Neither file changed while the trials ran. That is worth stating plainly because it makes the discovery-file result later in the paper a clean negative: the files were present, valid and permissive, so a client that did not fetch one was not being turned away.

Every tool received the identical prompt, given in the paper as "Get developer documentation from this URL", with coding agents driven through their normal user-facing workflow and assistant services given the same prompt and URL in their chat interfaces. Each was run three times, from the paper's local network and from external networks, and all logging happened in server-side Express middleware capturing method, HTTP version, URL, the complete header set, the User-Agent string and the client IP. No client-side instrumentation was used, so what the study reports is raw HTTP arriving at an origin. That is the same vantage point a scan takes when it asks what a server hands a machine, and it is the reason the measurement approach here is legible at all: a header set either arrived or it did not.

The sample is purposive rather than representative, and the paper says so. Tools had to be open source, freely available, or reachable on a free or trial tier during the measurement window, and enterprise-only products and anything that could not be tested repeatably were excluded. Selection leaned on a SlashData survey covering responses from 2,393 professional developers. The paper states that it characterises the observed tools and does not claim statistical representativeness of all AI coding products, which is the correct disclaimer and one that most vendor research on this subject omits. The anonymised logs are published in a companion repository at github.com/oborys/AI-Agents-HTTP-level-behaviour, a host this site does not carry on its registered outbound list, so the address is written out rather than linked. If you want to see how the same questions look against your own origin, the robots.txt tester answers the rules half of it without any of this apparatus.

Seven of nine agents named a library, not a product

Here is the finding worth carrying. Of the nine coding agents, the paper records only two sending a User-Agent that identifies the product: Aider, which appends Aider/0.86.2 and a link to its own site onto a Chrome string, and GitHub Copilot in VS Code agent mode, whose string carries Code/1.111.x and Electron/39.x. The other seven identify a runtime or nothing in particular. Cline and Junie both arrive as curl/8.4.0. Claude Code arrives as axios/1.8.4. Antigravity arrives as Go-http-client/2.0. Cursor sends the string got followed by that library's repository address, and Windsurf sends colly followed by the Colly repository address in the same style. OpenCode sends an ordinary Chrome string with no product marker in it at all.

Two of those strings are shared by more than one tool, which is the part that breaks log analysis rather than merely inconveniencing it. A line in your access log reading curl/8.4.0 was Cline or Junie in this study, and in the wild it is far more likely to be a person or a script. There is no version of a user agent allowlist that separates them, because the string carries no information that distinguishes them. This site has argued before that a user agent is a claim rather than an identity, and the usual worry there is impersonation, a request wearing a crawler name it has no right to. This is the mirror case, and it is more awkward: a real AI client wearing no name at all, honestly, because the library it was built on filled the field in.

The User-Agent header is a free string and always has been, so none of this is misbehaviour. It is what a default looks like. But robots.txt matching is defined against a product token, and the matching problem is already an ordering problem with named failure modes even when the client is trying to be recognised. When the client sends a library name, there is no ordering that helps: the group headed by an asterisk is the only group that can apply, and it applies the same way it would to any script.

The assistant services behave better on this axis and it is worth saying so, because the honest version of this finding is not that AI clients hide. ChatGPT sent ChatGPT-User/1.0 with the openai.com bot address, the Claude assistant sent Claude-User/1.0 with an anthropic.com contact, and MistralAI sent MistralAI-User/1.0 with a link to its own robots documentation. Two are less useful: Google NotebookLM sent Google-NotebookLM, and Gemini sent a User-Agent consisting of the single word Google. One further detail is worth flagging rather than smoothing over. The paper's table records the Perplexity assistant sending PerplexityUser/1.0 with no hyphen, while Perplexity's own bot documentation publishes the fetcher as Perplexity-User. Both are quoted here as their sources give them. A matcher keyed on the exact published token would miss the string as the paper prints it, and nothing in this study or on this site settles which one a live request carries today. The crawler reference on this site lists the published tokens, which is the set a rule can actually be written against.

Only two of fifteen clients asked for robots.txt

The discovery-file result is the shortest row in the paper and the one with the most consequences. Across all fifteen tools and all three trials, no tool requested llms.txt. Two requested robots.txt: the Claude assistant, which fetched robots.txt and the page in parallel from separate IP addresses, and MistralAI, which probed robots.txt and then fetched in browser mode. Every one of the nine coding agents fetched the page without asking for either file first.

Take the llms.txt half carefully, because it is a smaller claim than it looks. Fifteen tools on one endpoint is not a population, and no result from it should be generalised into an adoption statistic. What it does is add a consistent data point to a picture this blog has already reported: when Ahrefs looked at the question at scale, the evidence on llms.txt was that the overwhelming majority of valid published files were never fetched by anything. A study measuring publication and a study measuring retrieval are different measurements, and both currently point the same way. We ship a file generator for it anyway, and the reasoning for that is in the llms.txt definition: a file that costs nothing to publish is a defensible bet, and a claimed effect it has not been shown to have is not.

The robots.txt half is the one that changes what a site owner can conclude from their own file. RFC 9309 defines the protocol as a set of rules a crawler is expected to fetch and apply voluntarily, and says in its first section that the rules are not a form of access authorization. A client that never requests the file has not violated anything at the transport level. It simply never entered the conversation your file is one half of. Anthropic's own crawler guidance describes how site owners block its crawler through robots.txt, and in this study the Claude assistant did fetch the file before fetching the page, which is the behaviour that guidance implies.

The practical shape of the gap is this. If your goal is to keep content out of model training, the vendor crawlers are the population that matters and they publish tokens you can write rules against. If your goal is to control what an individual user's agent can pull into a chat window on their behalf, robots.txt is largely not the mechanism, because in this study thirteen of fifteen clients never read it. That is a statement about a controlled sample of fifteen tools on one page, not about the whole category, and the paper's own limitations section makes the same point about external validity.

One request per prompt was the whole visit

The request counts are stable enough to quote individually, which is unusual. Nine of the fifteen tools issued exactly one request per prompt: Aider, Claude Code, OpenCode, GitHub Copilot, Windsurf, ChatGPT, Gemini, NotebookLM and Perplexity. Four issued two: Antigravity and Cursor, each of which sent a HEAD probe before the GET, plus the Claude assistant and MistralAI, whose second request was the robots.txt fetch. Junie issued three, walking pages sequentially. Cline issued four, because it swept for an OpenAPI or Swagger specification alongside the page. Summed per trial that is 16 requests across the nine coding agents and 8 across the six assistant services, and the paper reports no count deviating across the three trials.

One request is not a session in any sense your analytics understands. There are no page views to sequence, no scroll depth, no dwell time and no exit page, because there was only ever one hit. The paper's framing is that this makes session depth, time on page, click path and bounce rate unreliable indicators of documentation consumption, and for a docs team that is the headline. For anyone measuring AI visibility it lands somewhere more specific: whatever these clients understood about the page, they understood from a single response body. Anything your site reveals on a second click was not seen.

That is the same constraint vendor crawlers operate under, arrived at from a different direction, and it is why the shape of a single response matters more than the shape of a site. Where the two populations differ is in who you can address. Vendor tokens can be allowed or denied by name, and even there the granularity is poor: six of the nine vendors in Lantad's registry publish exactly one crawler token, so allow search and block training is not expressible for most of them. OpenAI's crawler documentation is the reference for which of its names does which job, and several tokens across the industry are robots-only, which is why some crawler tokens never appear in your logs regardless of what your file says. Against a client sending axios/1.8.4 there is no name to address at all, so the only levers left are the ones that apply to every request equally.

Two of the nine agents ran a browser, the rest took the server's HTML

The runtime column decides what a client can read, and on this endpoint it did not have to. The paper is explicit that the test portal did not rely on client-side rendering, so every client got the content whatever it was built on. That makes the runtime split a statement about capability rather than an observed failure, and it should be read that way.

Aider and OpenCode drive headless Chromium through Playwright, and the paper notes that a Playwright-driven browser fully evaluates JavaScript before resolving the fetch, which puts those two in a position to retrieve documentation from client-side rendered portals. The remaining lightweight clients, it says, receive only the static HTML delivered by the server and do not execute JavaScript. GitHub Copilot in VS Code agent mode is the interesting middle case: its runtime is Electron and Chromium and its User-Agent looks the part, but the paper describes the requests as originating from the extension host layer rather than from a full browser navigation, bypassing the complete JavaScript rendering pipeline. A Chromium-shaped identity is not evidence of Chromium-shaped rendering, which is a good general lesson for anyone reading user agents for capability.

So the split on this axis is two clients out of nine that would survive a client-rendered page, one hybrid whose string suggests more than it does, and six that would receive whatever the server sent and nothing more. That is the same test prose parity exists to run: compare what a browser assembles against what arrives in the raw HTML, and the difference is the part no lightweight client sees. It is also the same constraint that made Common Crawl's July archive a JavaScript-free capture of 2.14 billion pages, so a client-rendered page contributed whatever its server returned and no more.

The remedy is unglamorous and stack-specific. Server render the content that matters, which on Next.js mostly means not deferring the primary copy to a client component, and check the result rather than assuming it. Lantad's raw fetch view shows the HTML a server actually returns to a plain request, which is the document six of these nine agents would have been working from. Header signals point the same way: the paper records Accept present for seven of the nine coding agents and the Sec-Fetch family for only two, with four of six and one of six respectively among the assistants, so most of these requests do not carry the metadata a browser navigation would.

What this does not show, and what to check on your own site

The limits are large and the paper states most of them itself. One endpoint, built by the author, is a strong design for comparability and a weak one for external validity, and behaviour may differ on a static HTML site, a large documentation portal, or anything behind a CDN that varies its response by client. Fifteen tools chosen for testability is not a census of a category that grows monthly. Three trials establish that a fingerprint was stable in February and March 2026, not that it is stable now: agent releases ship weekly and a User-Agent is one line in a config file. Every figure in this post should be read as what those tools did on that endpoint on those dates.

What it does not show at all is anything about vendor crawlers at scale, which are a separate population with separate documentation and separate behaviour. Nothing here says GPTBot ignores robots.txt, and nothing here contradicts the vendor guidance on how those crawlers work. Fetchers that skip robots.txt are also not a new category: Google publishes a class of user-triggered fetchers of its own, and fourteen named Google agents are not decided by the robots.txt wildcard by design and by documentation. This paper adds a different observation, which is that a large and growing set of clients do not appear in any token list because their builders never made them announce a product.

The check that follows from this is not a new file to publish. It is to stop treating the presence of a rule as the presence of an outcome. Three questions are answerable from outside your own site today. First, what does your origin return to a plain request that runs no JavaScript, because that is the document most of these clients receive. Second, what do your robots.txt rules actually resolve to per crawler, which is only meaningful for clients that read the file and identify themselves. Third, which of your content depends on a second request, because none of these clients made one. Lantad's own crawler identifies itself and states its purpose on the bot page, which is the standard we would want applied to us.

Anyone wanting to verify the numbers above rather than trust this summary can read the paper itself; Tables 1 through 4 hold every figure quoted here, and the raw logs are in the author's companion repository named in the method section. That is the whole point of citing a primary source: the claim and the evidence should be one click apart, and a summary you cannot check is worth nothing on a subject where what AI can read about you is the thing being sold.