Every web agent was identifiable, and robots.txt stopped one of twelve

What the study ran, and over what period

The paper is On the Internet, Nobody Knows You're an LLM Bot: Unmasking Web Agents with Multi-Layer Fingerprinting, posted to arXiv as 2606.30119 on 29 June 2026 by Iliana Fayolle, Sihem Bouhenniche, Samuel Pelissier, Pierre Laperdrix, Clementine Maurice and Walter Rudametkin, working at Inria with Univ. Lille, CentraleSupelec and Univ. Rennes. The design is a honeysite array: a main domain serving an undefended site, plus nine subdomains each carrying a different defence configuration, all behind one nginx reverse proxy that captured network packets and access logs while the pages themselves collected browser fingerprints.

Twelve tools were driven at that array, in three groups. Three HTTP scrapers, cURL, wget and scrapy. Three browser automation frameworks, Selenium, Playwright and Puppeteer. And six LLM-driven web agents, OpenClaw, Anthropic's Claude for Chrome, Crawl4AI, BrowserUse, OpenAI's ChatGPT Agent and Skyvern. Human visits from Chrome and Firefox on the same machines provided the ground truth. The defences were seven individual mechanisms, robots.txt, User-Agent filtering, Google reCAPTCHA v3, Prosopo CAPTCHA, the Anubis proof-of-work challenge, Cloudflare Turnstile and Cloudflare's Bot Fight Mode with Block AI Bots, plus two combined configurations, one built only from free and open-source pieces and one from Cloudflare's.

The dates belong next to every figure. Table 1 records that experiments were conducted between January and May 2026, and the active traffic that produced the dataset was generated between 28 January and 24 February 2026. Of 1,449 initiated visits, some were excluded because a defence stopped them before they reached the servers or because of technical errors: Cloudflare blocked 40 BrowserUse visits, 20 from HTTP scrapers and one from Puppeteer, and OpenClaw timed out three times. The final dataset is 1,383 visits carrying 1,383 browser fingerprints and 1,358 TLS Client Hello records, the difference arising because several visits can share one TLS session.

One exclusion is worth more attention than the paper gives it. The authors deliberately left out the general-purpose assistants, ChatGPT, Gemini and Perplexity, on the grounds that their own tests showed those systems typically retrieve content via simplified HTTP requests similar to cURL, do not execute client-side scripts, and are therefore trivial to detect or block. That is a description of the retrieval fetchers most site owners actually worry about, and it matches what was found when nine AI coding agents arrived as curl and axios. The hard-to-detect category in this paper is the agentic browser, not the AI crawler fetching a page to answer somebody's question.

The robots.txt honeysite stopped one client in twelve

The robots.txt column of the paper's results table has one blocked cell in it. scrapy stopped, and the paper explains why in a sentence that is about configuration rather than about ethics: scrapy ships with ROBOTSTXT_OBEY set to True, the honeysite's file restricted access, and the framework respected the rules and stopped crawling. Every other tool, including all six web agents, reached the page. The User-Agent filtering honeysite produced the same single blocked cell for the same reason.

The detail underneath is sharper than the headline. The paper records that the other HTTP scrapers did not request the robots.txt file before crawling at all. cURL and wget did not consult the file and then decide to ignore it. They never asked. A rule that is never read cannot be disobeyed, and a server log that shows no fetch of /robots.txt from a client is not evidence of a well-behaved crawler making a considered choice. It is evidence that nothing consulted the file.

None of this is a defect in the standard, which says so itself. RFC 9309 specifies rules that crawlers are requested to honor, and states plainly that these rules are not a form of access authorization. The file is a request addressed to a client that has chosen to ask. That is a useful thing to publish and a useless thing to rely on for enforcement, and the two get confused constantly because the same file is the entry point for both conversations.

For a site that wants AI answer engines to read it, this cuts the reassuring way round: the named crawlers whose access you actually want are the ones that fetch the file, and a correct group for them is the whole of the job. Our robots.txt tester evaluates a file per crawler token for that reason, and the crawler registry behind it lists the tokens that answer. For a site that wants to stop an agent, the same file does nothing. Both halves have been covered here before, from the direction of what happens when a GPTBot ban still serves a 200 and from the direction of why a user agent is a claim, not an identity. This paper adds the missing measurement: in a controlled test with the file correctly written, the only client it stopped was the one that had already decided to be stopped.

One further nuance sits in the paper's appendix and complicates the word blocked. Agents relying on Claude models sometimes refused to bypass a protection, citing internal safety guidelines against circumventing human-verification mechanisms, and Claude for Chrome declined the Prosopo CAPTCHA in 20 separate attempts. The authors classified those cases as blocked. A blocked cell in the table can therefore mean the defence detected the client, or it can mean the client read the situation and declined, and the server sees the same outcome either way.

Which layer actually identifies the client

Having established what the defences did, the authors asked a separate question: given a single request, how well can a server say which tool sent it. They trained classifiers on the fingerprints, excluding the HTTP scrapers because those execute no JavaScript, leaving 1,370 fingerprints across twelve classes. The reported numbers come from a Random Forest on an 80/20 split with SMOTE resampling on the training set, run ten times with fresh random seeds and averaged.

The layer ordering is the finding. Network information alone, meaning the IP address and its ASN, scored 0.596 accuracy. TLS alone, meaning the JA4 fingerprint of the Client Hello, scored 0.454. Both are weak, and the paper explains why in a way that generalises: IP attributes characterise the network the tool is sitting on rather than the tool, and local agents inherit their TLS stack from the host browser, so a Playwright-driven Chrome and a human's Chrome present the same handshake. Combined, network and TLS reached 0.806. Browser fingerprinting on its own reached 0.931, and all three layers together reached 0.993, described in the paper as near perfect classification for all classes.

Cloud-hosted agents are the easy case and were the easy case at every layer. ChatGPT Agent was classified perfectly from its JA4 fingerprint alone: it establishes only TCP connections, advertises 28 cipher suites, more than most tools tested, and exposes 12 TLS extensions, fewer than most, a combination the authors read as a deliberate choice favouring compatibility with older server configurations. Skyvern was the only tool in the study running Microsoft Edge. Across the whole dataset the authors observed 21 distinct JA4 values, and the distinctive ones belonged to tools running in tightly controlled cloud environments rather than on somebody's laptop.

Two things follow that are easy to overstate, so it is worth being exact. The first is that 0.993 is accuracy within this dataset, distinguishing twelve known tools and a human baseline that the authors themselves generated. It is not a detection rate on live traffic, where the class list is open and unknown clients arrive constantly. The paper is explicit that its setup was two Linux machines across four locations running Chrome and Firefox, and that results may not generalise to other browsers, operating systems or hardware. The second is that the strongest single layer, browser fingerprinting, requires the client to run JavaScript, which the general-purpose assistants excluded from the study largely do not. Ordering matters as much as capability here, a point covered from our own side in AI crawler detection is an ordering problem.

The paper's own conclusion is the sentence to carry away, and it is not the accuracy figure. All tested tools were identifiable at the time of writing, the authors write, but they were not all blocked. Identification and enforcement are separate systems, and a site can be very good at one while doing nothing about the other. That is also the argument for cryptographic approaches, which the paper names as a likely direction: a signature proves who sent a request rather than inferring it, which is what Web Bot Auth actually specifies and what Cloudflare's bot verification documentation describes deploying.

Why stealth mode made agents easier to spot

Several of the agents tested ship a stealth or undetected mode, and two of them advertise bot-detection bypass in their documentation. The paper's third headline finding is that in this dataset those modes did not reliably work, and in places actively hurt. At the TLS layer the authors found no significant difference: stealth configurations produced dominant JA4 values and distinctiveness scores much like the ordinary ones, so the handshake surface was not meaningfully altered. At the browser layer, enabling stealth made both Crawl4AI and BrowserUse more suspicious to frictionless CAPTCHA systems rather than less.

The mechanism is the interesting part, because it is the same failure in two different places. Crawl4AI's stealth implementation applies overrides such as simulate_user and override_navigator that modify native browser properties, and the authors observed substantial variation in HTTP headers under stealth that was absent without it. Overriding a property to look normal produces a combination that is not normal, and inconsistency is exactly what a fingerprinting check is looking for. Separately, the paper found Crawl4AI spoofing its HTTP User-Agent header, replacing its real version with an older default value, and caught it by comparing that header against the navigator.userAgentData JavaScript property, which reports the same information through a different and harder to edit channel. A mismatch between the two is not just a failure to hide. It is an additional identifying signal that an unmodified client would not emit.

The network figures are blunter still. BrowserUse in its ordinary cloud mode reached the honeysites across 35 distinct autonomous systems with only 0.02 of its addresses flagged as hosting infrastructure and 0.02 as abusive by the reputation service the authors used, consistent with the residential proxy pool the product advertises. BrowserUse-Stealth used three autonomous systems, with 0.98 of its addresses flagged as hosting and 0.98 as abusive. The stealth setting, on this evidence, moved the traffic off the residential proxies and onto low-reputation datacentre addresses. The authors flag it as surprising and say they cannot tell whether it is a misconfiguration or intended, which is the correct thing to say about a closed product.

For anything that renders pages, this is a useful corrective in both directions. Sites do block real browsers on evidence this thin, as the Bamberg study covered here in headless Chromium was blocked on 15.2 percent of the top 10,000 sites showed, and that is a cost paid by legitimate automation including our own renderer, which identifies itself and is described on our bot page. But trying to look like something you are not is measurably worse than being consistent, and the earlier finding that one of seven browsing agents was stopped by self-identification rather than detection points the same way.

What a site owner can check without a fingerprinting stack

Almost nothing in this paper is deployable by a normal site. Capturing JA4 fingerprints needs packet capture at the edge, and browser fingerprinting needs script that many of the clients in question never run. The useful output is not a technique, it is a correction to a mental model, and the correction has a practical shape.

Separate the two questions and answer them in the right order. The question of whether you can stop an agent is an infrastructure question, it belongs to your CDN or your origin rules, and this paper is evidence that the file-based half of it is close to decorative against agentic browsers. The question of whether the clients you want can read you is a content question, it is entirely within your control, and it is the one with a return. A retrieval fetcher that arrives, gets a page and finds no text extractable without JavaScript has been failed by the site, not by a defence.

That second question is what this scanner exists to answer, and the split is the one described in the two layers that decide if AI can read your site: reachability first, then whether the reachable HTML carries the content. Both halves are checkable today. Whether the text a browser shows also exists in the raw HTML is prose parity, and it is the single most common failure we see. Whether a machine can identify what the page is about without parsing prose is structured data. What a scriptless fetch of your own page actually returns is visible through what GPTBot sees, and the weighting behind the whole grade is written out on our methodology page rather than described in general terms.

There is a second-order point in the exclusion the authors made, and it is the most actionable sentence in the paper for anyone thinking about AI visibility. They dropped ChatGPT, Gemini and Perplexity from the study because those systems fetch with simplified HTTP requests and do not execute client-side scripts. The systems that are hardest to detect are agentic browsers running full Chrome. The systems that decide whether you appear in an answer are, on the authors' own testing, closer to cURL. Optimising your site against the sophistication of the first group is a category error: the clients that matter for the ChatGPT citation guide and the Claude guide are the ones that will not run a line of your JavaScript.

The last thing to take from the paper is its own hedge, and it is a good one. The authors describe a cat-and-mouse dynamic, note that several tools changed behaviour noticeably during their four-month evaluation window, and observe that web agents remain expensive to run because they need advanced models and full browsers, which currently limits them to well-resourced operators. Every figure above describes a set of products as they behaved between January and February 2026. That is what a measurement is, and pretending otherwise is how a dated finding turns into a permanent belief. Where we hold our own dated figures rather than somebody else's, they are on the research page.