Blog / AI bot blocking stopped one of seven AI browsing agents

AI bot blocking stopped one of seven AI browsing agents

A controlled study posted to arXiv on 2 May 2026 turned on both of Cloudflare's free AI bot controls and then ran seven AI browsing agents through three tasks on the authors' own site. Six finished. The one that was stopped is listed in Cloudflare's bot directory, which the authors read as self-identification rather than detection.

Findings 3 August 2026 18 min read

In short

FP-Agent, posted to arXiv as 2605.01247 on 2 May 2026, enabled both AI Crawl Control and Block AI Bots on an instrumented honey website and reports that six of the seven AI browsing agents tested still completed the flight booking, shopping and forum tasks.
Manus was the only agent fully blocked in that experiment, and the paper states Manus appears in Cloudflare's bot directory as a verified bot, which suggests the block rested on self-identification rather than on detection.
Browser fingerprint features alone scored an F1 of 0.7969 in the paper's classifier against 0.9993 for behavioural features, so what separated these agents was typing, scrolling and mouse movement rather than anything a server reads from a request header.
Three of the four crawler tokens Cloudflare's rules fired on in that study, ChatGPT-User, ClaudeBot and Perplexity-User, are tokens Lantad's registry already carries, and the paper reports those blocks landed on provider infrastructure rather than on the agent running in front of the user.
Lantad measured none of this and cannot: an external scan sees a named client requesting a URL, and the defining property of this traffic is that it presents no name to resolve.

An AI browsing agent does not fetch a page. It opens one. Atlas Agent and Comet ship as browsers, Claude for Chrome ships as a browser extension, and Browser Use and Skyvern are libraries that drive a real Chrome from a script. Every one of them renders, executes JavaScript, scrolls, clicks and types, and every one of them arrives at a site looking a great deal like a person on a laptop.

That is a different problem from the one this blog usually writes about, and a paper posted to arXiv on 2 May 2026 tested it rather than argued it. Lantad measured nothing in what follows. The figures below are read from the paper, and the part this site can add comes at the end: every access control covered here, from robots.txt to a CDN toggle, is keyed to a client that declares who it is. This is the traffic that declares nothing.

Browsing agent	Detected by the paper's classifier	Blocked by Cloudflare	Cloudflare rule triggered
Atlas Agent	Yes	No	None
Browser Use	Yes	No	None
Claude	Yes	Server side requests only	ClaudeBot
Comet	Yes	Server side requests only	Perplexity-User
Skyvern	Yes	No	None
ChatGPT Agent	Yes	Server side requests only	ChatGPT-User
Manus	Yes	Yes	Manus Bot

Table 5 of FP-Agent, arXiv 2605.01247, posted 2 May 2026, recording each agent against Cloudflare's free bot management with both AI bot controls enabled on the authors' honey website. Reported from the paper, not measured by Lantad.

What the study ran, and on how many trials

The paper is FP-Agent: Fingerprinting AI Browsing Agents, by Ethan Wang, Zubair Shafiq and Yash Vekaria, posted to arXiv as 2605.01247 on 2 May 2026. Its instrument is a honey website: a site the authors built and instrumented themselves, serving each visitor its own copy of the pages so that every recorded event could be attributed to the session that produced it.

Seven browsing agents were run against it. Five are commercial products, listed in the paper as Atlas Browser and ChatGPT Agent, Claude for Chrome, Comet and Manus. Two are open source, Browser Use and Skyvern, pinned at versions 0.9.2 and 0.2.23 so the measurement stays reproducible. Two candidates were excluded and the paper says why: Google's Project Mariner was in beta during the selection period and could not complete the tasks, and xAI had no browsing agent at the time.

Each agent ran 1,000 trials in total, split equally across three tasks, which were booking a flight, shopping online, and interacting with a community forum. The human comparison group was 56 undergraduate students from the authors' university, each performing three repetitions of every task on their own machines. That produced 546 human instances against 1,000 per agent, an imbalance the authors name and say they chose a tree based classifier partly to absorb.

Three properties of that design bound every number in this post, and they are worth stating before the results rather than after them. The tasks are three specific flows, not a sample of the web, and the paper notes that drag and drop was excluded because not every agent supports it. The classification is closed world, meaning each class is one of the seven studied agents or a human, so the classifier is explicitly not built to name an agent it has never seen. And these are seven products at seven versions inside one measurement window, which is the ordinary limitation of measuring software that ships continuously.

None of that makes it a survey of live traffic, and it is more useful for not being one. A traffic survey could not have reported what happened when the defences were switched on, because a survey cannot switch anything on. Naming what a study cannot support is the same discipline this site applies to its own scan methodology, and it is why the numbers below are attached to a date and an experiment rather than offered as the state of the web.

Browsing agents tested Seven Five commercial products and two open source libraries, at pinned versions.
Trials per agent 1,000 Split equally across booking a flight, shopping, and forum interaction.
Human comparison group 56 participants 546 instances collected against 1,000 per agent, an imbalance the paper names.
Live traffic observed None A controlled honey website built by the authors, not a measurement of the open web.

The scope of the FP-Agent experiment as the paper states it, arXiv 2605.01247, 2 May 2026. Reported from the paper, not measured by Lantad.

Both AI bot controls were on, and six of seven agents finished the task

The result that matters to a site owner is a case study near the end of the paper. The authors evaluated Cloudflare's free bot management, which as they describe it offers two AI bot related controls: AI Crawl Control, which lets a publisher write rules blocking specific AI search bots, AI crawlers and AI assistants, and Block AI Bots, a toggle that blanket blocks AI crawlers. They enabled both on the honey website and then asked each agent to perform the three tasks.

The paper's finding is one sentence: only Manus is fully blocked by Cloudflare, whereas the other six browsing agents complete the tasks successfully.

The three partial results in the middle of that table are the interesting part, because they look like enforcement and are not. Cloudflare did block requests attributed to ClaudeBot and to Perplexity-User, but the authors report that IP based analysis places those requests on the browsing agent providers' server side infrastructure rather than on the locally running Claude and Comet themselves. Some ChatGPT-User requests were blocked too, and ChatGPT Agent still completed the tasks despite running from cloud infrastructure. So the rules fired, a dashboard would have shown blocks, and the agent finished anyway. A block count that goes up while the thing being blocked keeps working is the most expensive kind of number, because it reads as evidence that a control is holding.

The one clean block does not weaken that reading, it explains it. The paper notes that Manus appears in Cloudflare's bot directory as a verified bot, and says this suggests its successful blocking is likely based on self-identification. Read together, the seven rows describe a control surface that works exactly as far as a client is willing to announce itself, and stops there. That is the same boundary this site described when the tokens that never appear in your logs turned out to be unobservable for the same structural reason, and the same one behind the older point that a user agent is a claim rather than an identity.

It is worth being precise about what this does not say. It does not say Cloudflare's controls fail against AI crawlers generally, and the paper does not test that: a declared crawler fetching a URL is the case those controls were built for and the case they handle. It does not test paid tiers, custom rules or managed challenges, only the free controls named above. And it is one honey website in one window, not a claim about detection rates across the network. What it does establish is narrower and still uncomfortable: on this site, with the AI bot switches on, six of seven agents got the content.

Requests to the honey website, both AI bot controls enabled

Claude for Chrome, running locally in the user's browser task completed
Requests attributed to ClaudeBot from provider infrastructure blocked
Comet, running locally task completed
Requests attributed to Perplexity-User from provider infrastructure blocked
ChatGPT Agent, running from cloud infrastructure task completed
Manus, listed in Cloudflare's bot directory as a verified bot blocked
Agents completing the tasks with both controls on six of seven

The Cloudflare case study as the paper describes it, with AI Crawl Control and Block AI Bots both enabled. Reported from FP-Agent, arXiv 2605.01247, not measured by Lantad.

Why the browser fingerprint did not separate the agents

If a server cannot rely on the name a client gives, the next thing it reaches for is the fingerprint: the screen resolution, the font list, the plugin set, the timezone, the colour gamut, the pile of attributes a page can read from a browser. The paper trained a classifier on those features alone and reports precision of 0.8674, recall of 0.8179 and an F1 of 0.7969 across the seven agent classes. Adding humans as an eighth class moved that to 0.8840, 0.8407 and 0.8223.

An F1 near 0.8 sounds usable until you look at where the errors come from. The paper's own explanation is that browser fingerprints provide weaker discriminative power because several agents share the same underlying fingerprint. Atlas Agent is the extreme case in the paper's Table 1: one unique fingerprint observed across the whole experiment, 100 percent top one coverage, normalised entropy of 0.00, and that single fingerprint also produced by another class. Claude shows two unique fingerprints, both shared. Human participants, by contrast, produced 57 distinct fingerprints with top one coverage of 12.50 percent.

That inversion is the finding. Fingerprinting was invented to pick one person out of a crowd, and it works on humans precisely because humans have heterogeneous hardware. Agents built on the same Chromium defaults are homogeneous by construction, so the technique loses resolution exactly where you want it. Where an agent does stand out, it stands out for arbitrary reasons rather than principled ones: Skyvern on macOS reported a screen resolution of 1920x1080 against 1440x900 for the others, no HDR support, an srgb colour gamut rather than p3, no plugins, and a New York timezone against Los Angeles. Those are configuration accidents of one deployment, not durable properties of a product.

There is a second reason not to build a wall out of this. The paper records ChatGPT Agent reporting MacIntel as its system while using the Windows keyboard shortcut Ctrl plus V to paste, an inconsistency that is useful today and is a one line fix whenever anyone decides it matters. Any signal that a vendor can normalise away is a signal on a clock, which is the general shape of the argument this site made about Web Bot Auth from the opposite direction: a cryptographic claim is durable because it is designed to be checked, and an incidental artefact is not.

None of this is a scoring signal for an external scan, and it is worth saying so plainly. Fingerprint features are read by a page from a client that has already arrived and executed script. Nothing in what a Lantad scan measures touches them, because a scan is a request going the other way.

Browser fingerprint features, seven agent classes 0.7969 Precision 0.8674, recall 0.8179
Browser fingerprint features, agents plus humans 0.8223 Precision 0.8840, recall 0.8407
Behavioural features, seven agent classes 0.9993 Typing, scrolling and mouse behaviour
Combined features, agents plus humans 1.0000 Reported as 1.0000 on all three metrics

Classifier F1 by feature set, from Table 2 of FP-Agent, arXiv 2605.01247, 2 May 2026. Bars are drawn at 100 times the F1 value. Reported from the paper, not measured by Lantad.

Typing, scrolling and mouse movement did separate them

The behavioural classifier reports an F1 of 0.9993 on the seven agent classes and 0.9994 with humans added, and the combined feature set is reported as 1.0000 on precision, recall and F1. Those numbers are close enough to perfect that the interesting question is which signals carry them.

Mouse movement is the blunt one. The paper reports that across its experiments, browsing agents produce mouse events only at click time: a single mousemove followed immediately by mousedown and mouseup at the click location, with no continuous pointer movement in between. The authors put it as agents teleporting directly to the click location. Humans generate continuous streams of mousemove events, which is what lets the paper compute features like curvature angle and movement direction at all. This is not a subtle statistical difference. One population emits a trajectory and the other emits an arrival.

Typing splits the agents into two families. ChatGPT Agent, Atlas Agent and Comet paste text into fields rather than typing character by character. Browser Use, Skyvern and Manus type with keystrokes, at average inter key latencies the paper gives as 5.31, 9.52 and 1.39 milliseconds and average hold latencies of 10.19, 11.33 and 52.92 milliseconds respectively. For reference on how mechanical that is, the paper set its idle threshold for segmenting mouse movement at 250 milliseconds on the grounds that it sits within the range of human reaction times. The authors also read the source of the two open source agents and confirmed the mechanism: Skyvern drives Playwright and inputs all but the final 20 characters programmatically, then uses keystrokes with a 10 millisecond inter key latency for the tail.

Scrolling is the third axis and the one with an exception. Human participants show significantly greater scroll duration and distance than most agents, with reported effect sizes of 0.67 and 0.39 at p below 0.01, and higher variance than every agent. The exception the paper names is ChatGPT Agent, which exceeds human participants on both average scroll duration and distance. An agent can therefore look more human than a human on a single dimension, which is a good reason to distrust any one signal used alone.

Two caveats travel with these results. The paper's own generalisation test, holding out one task at a time, drops the behavioural classifier to F1 values of 0.7997, 0.6303 and 0.7432 on the unseen task, while the combined classifier stays between 0.9316 and 0.9708. Behaviour alone does not transfer as cleanly as the headline figure suggests. And the authors are explicit that this is an arms race: every artefact described above is a defect that a vendor can fix, and several of them look like the kind of thing fixed in an afternoon.

What humans produced

Continuous mousemove streams between clicks
Curvature and direction computable from the path
Longer and more variable scroll duration and distance
57 distinct browser fingerprints across 56 participants
Top one fingerprint coverage of 12.50 percent

What the agents produced

One mousemove, then mousedown and mouseup at the target
No pointer path between click locations
Pasting, or keystrokes at inter key latencies of 1.39 to 9.52 ms
Atlas Agent: a single fingerprint, entropy 0.00
Fingerprints shared between agent products

The behavioural differences the paper reports between its human participants and the browsing agents it tested. Reported from FP-Agent, arXiv 2605.01247, not measured by Lantad.

What a robots.txt keyed to product tokens can answer about a browsing agent

Here is where this site's own subject starts, and where the honest answer is a boundary rather than a feature.

RFC 9309 defines robots.txt as a file of groups, and a group is selected by matching a product token against the crawler's identity. The mechanism has no other input. A group cannot be selected by intent, by IP range, by how the client scrolls, or by whether a human asked for the page. If the client presents no product token, no group applies to it beyond whatever the wildcard says, and a wildcard group is a rule written for everything rather than a decision about this population.

Lantad's crawler registry carries 15 robots.txt product tokens, and that is a setting rather than a finding: someone chose which vendors to track and the list is edited by hand. Three of the four tokens the paper reports Cloudflare's rules firing on, ChatGPT-User, ClaudeBot and Perplexity-User, are in it. Manus Bot is not. The registry also sorts tokens into training, search and user initiated purposes, and the user initiated class is the closest thing in it to the traffic this paper studied. It is still not the same thing. OpenAI's crawler documentation describes ChatGPT-User as the agent that fetches a page because a user asked, which is a request made by OpenAI's infrastructure carrying OpenAI's name. Atlas Agent running on a laptop is a browser on a residential connection, and nothing obliges it to announce anything.

So the resolution question splits cleanly. Resolving your robots.txt against a named token answers what a compliant declared crawler is instructed to do, and that is a real answer covering real traffic. It does not answer what a browsing agent will do, because there is no name to resolve. The distinction is the one behind the argument that two layers decide whether AI can read your site: the file states an instruction and the infrastructure returns a response, and this paper is a measurement of the second layer failing to catch what the first layer was never addressed to.

That is also why the emerging answer is cryptographic rather than nominal. Cloudflare's signed agents work and the Web Bot Auth drafts move identity from a string anyone can type to a signature that can be checked, and the paper's own assessment is the fair one: it calls this a useful step forward and then notes that it remains voluntary, naming Atlas Agent, Browser Use, Comet and Skyvern as agents that do not participate, and observing that participating agents may still choose when to sign. A voluntary scheme resolves the honest half of the population, which is most of it, and leaves the rest exactly where it was. Pricing schemes inherit the same dependency, which is the structural point underneath pay per crawl answering with HTTP 402: you cannot bill a client you cannot name.

Flow: Request arrives at your origin to Client presents a product token; Request arrives at your origin to Client presents a generic browser UA; Client presents a product token to robots.txt group selected by name; Client presents a product token to CDN rule matches the token; Client presents a generic browser UA to Only behaviour left to judge on; Only behaviour left to judge on (six of seven in this study) to Page served like any browser visit.

Which layer answers a request, by whether the client presents a product token. A description of the mechanism as RFC 9309 and the cited vendor documentation define it, not a measurement.

What to check on your own site, and what nobody can check for you

The useful response to a laboratory result is not to generalise it. It is to work out which of your own questions it changes, and this one changes three.

First, separate the two populations in your own head before you look at any tool. Declared crawlers are the ones your robots.txt addresses and your logs can count, and for those the ordinary work still applies: check which tokens you actually admit, which is what the AI crawler reference enumerates, and check that a token you meant to allow is not being refused at the edge. Browsing agents are the second population, they arrive as browser traffic, and no robots.txt edit will move them either way. Conflating the two produces a plan that fixes one and claims credit for both.

Second, for the population that renders, the thing that matters is the thing that was always going to matter. An agent driving Chrome executes your JavaScript, so a client side rendering gap does not stop it the way it stops a declared crawler. That cuts both ways and it is worth stating the pleasant half: if your site is readable only after JavaScript runs, a browsing agent still reads it, while a declared crawler receives whatever the server returned. It is the wrong reason to feel relaxed. The declared crawlers are the clients that feed retrieval and citation, and what a crawler receives from your server is still the question that decides whether you can be quoted at all. A browsing agent reading your page helps one user with one task and leaves no index entry behind.

Third, do not read a block count as a defence. That is the specific transferable lesson from the table at the top of this post: rules fired against ClaudeBot, Perplexity-User and ChatGPT-User in an experiment where the agents attached to those brands finished their tasks anyway, because the blocked requests came from provider infrastructure and the working client was somewhere else. If you are relying on a bot control, the question worth asking is not how many requests it blocked but whether the outcome you wanted actually happened.

There is a fourth item and it belongs to nobody's dashboard. Lantad measures whether a named client can reach and read a URL, and publishes the boundaries of that in the methodology and its aggregate figures in the research page. It has not measured browsing agent traffic, has no way to observe it from outside a site, and will not be publishing a figure for it. The position on that is the same one argued in why we will not grade a page we could not measure, and it is not modesty. An outside scan that reported a browsing agent number would be reporting something it had no instrument for, and this study is a reminder that the instrument for this population lives inside the page, in event streams a visitor generates, rather than in anything a request from outside can see. What an external scan can do for an AI crawler audience remains worth doing, and it remains a smaller claim than the whole question of who is reading your site.

Can a declared AI crawler reach and read your pages Answerable from outside by resolving robots.txt against named tokens and fetching the page as a crawler would. Unaffected by this study.
Did Cloudflare's free AI bot controls stop the agents tested Answered by the paper for one honey website: one of seven fully blocked, and that one is listed in Cloudflare's bot directory.
Does a block count prove a control is holding No. Rules fired on ClaudeBot, Perplexity-User and ChatGPT-User while the associated agents completed their tasks, because the blocks landed on provider infrastructure.
How much browsing agent traffic your site receives Not in this study and not observable by any external scan. It needs client side instrumentation on your own pages.

Which questions this study answers, which it does not, and where each one can be checked. A description of scope, not a measurement of any site.

Common questions

What is an AI browsing agent, and how is it different from an AI crawler?

An AI crawler is a program that requests URLs and identifies itself with a product token such as GPTBot or ClaudeBot, so a site can address it in robots.txt and a CDN can write a rule for it. An AI browsing agent drives a real browser to complete a task, rendering pages and executing JavaScript the way a person's browser would. The FP-Agent paper, posted to arXiv on 2 May 2026, studied seven of them: Atlas Agent, ChatGPT Agent, Claude for Chrome, Comet, Manus, Browser Use and Skyvern. The practical difference is that the crawler presents a name you can write a rule against and the browsing agent generally does not.

Did Cloudflare block AI browsing agents in this study?

One of seven. The authors enabled both of Cloudflare's free AI bot controls, AI Crawl Control and the Block AI Bots toggle, on their honey website and report that only Manus was fully blocked while the other six agents completed the tasks. Cloudflare did block some requests carrying ClaudeBot, Perplexity-User and ChatGPT-User, but the paper reports IP based analysis placing those on the providers' server side infrastructure rather than on the agent running for the user. The paper also notes Manus appears in Cloudflare's bot directory as a verified bot, which suggests that block came from self-identification.

Can robots.txt block an AI browsing agent?

Not by name, because there is generally no name to match. RFC 9309 selects a group of rules by matching a product token against the crawler's identity, and an agent driving an ordinary Chrome presents a generic browser user agent instead. A wildcard group still applies in the sense that it applies to everything, but it is not a decision about this population and nothing obliges an undeclared client to read the file at all. Cryptographic schemes such as Web Bot Auth address the identity gap directly, and the FP-Agent paper notes they remain voluntary, naming Atlas Agent, Browser Use, Comet and Skyvern as agents that do not participate.

Did Lantad measure any of the figures in this post?

No. Every figure here is read from FP-Agent, arXiv 2605.01247, posted 2 May 2026, and the one repository fact stated, that Lantad's crawler registry carries 15 product tokens, is a setting someone chose rather than a measurement of anything. Lantad has not measured browsing agent traffic and cannot measure it from outside a site: the signals that separate these agents are typing, scrolling and mouse events generated inside a visitor's browser. What a Lantad scan measures is what a named client receives when it requests your URL.

See what AI can read on your site

Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.

Scan your site free How we measure