Headless Chromium was blocked on 15.2 percent of the top 10,000 sites

What the study measured, and on which 10,000 sites

The instrument is one crawl run four times. The authors take the first 10,000 sites of the Tranco top one million list, naming the revision as ID GVL9K dated 25 February 2026, and load each homepage once in each of four conditions: Chromium headless, Chromium headed, Firefox headless and Firefox headed. That is 40,000 page visits. Naming the list revision matters more than it looks, for the reason this site gave when a July enforcement study named its Tranco list: the list regenerates, so a top 10,000 without an identifier is a sample nobody can rebuild.

All four conditions drive Playwright 1.57.0, at Chromium 143.0.7499.4 and Firefox 144.0.2, with the default Playwright User-Agent left unmodified and, in the paper's own table, no stealth measures at all. Scans ran from Hetzner Cloud in Nuremberg between 27 February and 2 March 2026, with the header spoofing follow-up on 3 March. The IP address rotates between batches of 1,000 sites by deleting the virtual machine and restoring it from a snapshot, and the four conditions for any one site run under the same address inside a narrow window, which is what makes the comparison between conditions a paired one.

Three properties bound every figure below and are worth stating before the results rather than after them. The traffic comes from a datacenter address rather than a residential one, and the authors say directly that this affects the absolute blocking rates while leaving the differences between conditions meaningful. The visit is a single homepage load with no interaction beyond it. And the client is default Playwright with nothing concealed, which is the configuration a researcher or a measurement tool reaches for, not the configuration something trying to evade detection would use. Every rate here is therefore a rate for an automated browser that is not hiding.

What the study does not measure is a declared AI crawler. GPTBot arriving with its own product token is a different client on a different path, decided by the robots.txt group its name selects and by whatever verified-bot allowlist the site's CDN keeps, and no condition in this experiment sends such a name. Reading these numbers as the rate at which sites block AI crawlers would be the wrong reading twice over: wrong about the client, and wrong about the mechanism. The paper is about what happens to the browser, and the browser is the tool used by measurement studies, by scanners, and by the agentic products that drive a real Chrome on a user's behalf.

Denominators move around in the results because not every site completed every condition. The paper is explicit: DNS failures, timeouts and SSL errors mean the paired analyses run on the 6,936 Chromium sites and the 7,063 Firefox sites where both display modes returned an HTTP response, and the property-access figures run on the 7,944 sites that produced data at all.

Headless Chromium was the only configuration that stood out

Table 6 of the paper classifies each of the 40,000 visits by HTTP status: a success is any 2xx, a soft block is a 403, 429 or 503, and a navigation timeout is a page that did not respond inside 30 seconds. Headless Chromium recorded 5,550 successes, 1,521 soft blocks and 117 timeouts. Headed Chromium, the same binary with a window, recorded 6,435 successes and 721 soft blocks. Firefox recorded 678 soft blocks headless and 677 headed, a difference of one site.

So the gap is not headless against headed, and it is not Chromium against Firefox. It is the specific combination. The paired comparison makes that sharper than the percentages do: among the 6,936 Chromium sites where both display modes returned a response, 772 sites blocked the headless run only and 14 blocked the headed run only. Among the 7,063 paired Firefox sites the same figures are 15 and 12, which is noise.

The overlap tells you it is one decision being made differently rather than two different populations of site. Of the 1,554 sites that soft blocked at least one configuration, 1,521, or 97.9 percent, blocked headless Chromium. Of the 750 sites that blocked something other than headless Chromium, 717, or 95.6 percent, also blocked headless Chromium. The authors read that as the same infrastructure driving every block and escalating its response for one configuration, which is a much more actionable finding than a raw rate: it means the sites in question are not hostile to automation in general, they are reacting to a particular signal.

There is a quieter number in that table that a scanner operator should not skip past. Even the best-performing conditions succeeded on only about 64.6 percent of the 10,000 sites, because roughly 2,100 to 2,450 infrastructure failures per condition sit underneath, DNS and SSL and connection errors on a toplist that always contains dead entries. Blocking is the part of the loss that correlates with something; the rest is ordinary attrition. Distinguishing the two is exactly the discipline behind reporting what a crawler actually met on a real page rather than reporting a clean number and quietly dropping the failures.

For a tool that compares the server response against the rendered page, a lost browser view is not a smaller measurement, it is a different one. Prose parity is defined as a comparison between two fetches, so losing one of them does not degrade the figure, it removes it. That is why the honest response to a blocked render is to say which half was measured, which is the rule this site states in its scan methodology and applies below.

Two headers caused three quarters of the gap

The paper names the mechanism and then tests it, which is the part that turns a correlation into something you can act on. Chromium in headless mode advertises itself twice. Playwright's headless User-Agent string contains the literal word HeadlessChrome and carries the full build number 143.0.7499.4 where the headed run sends the reduced form 143.0.0.0, and the sec-ch-ua client hint exposes HeadlessChrome as a brand value. Firefox does not implement client hints at all, which the authors point out removes that detection vector from both Firefox conditions and explains why the Firefox rows are identical.

To establish cause rather than coincidence they re-crawled the 798 sites that had returned HTTP 403 exclusively to headless Chromium, this time replacing HeadlessChrome with Chrome in the User-Agent and stripping the HeadlessChrome brand from sec-ch-ua so the headers matched headed Chromium exactly. Of the 784 sites that returned a response, 590, or 75 percent, returned a 2xx. The remaining 194, 25 percent, stayed blocked, which the authors attribute to deeper layers such as JavaScript fingerprinting, behavioural analysis or TLS fingerprinting.

Three quarters of a blocking gap that a scanner would experience as a wall turns out to rest on a string in two headers. That is the same lesson as a user agent being a claim rather than an identity, arriving from the opposite direction: the earlier point was that a name cannot be trusted when a client asserts it, and this one is that a name is acted on anyway, hard, when the client is honest enough to leave it in place.

The headers are not the whole detection surface, only the part currently doing the blocking, and the paper is careful about the difference. Its instrumentation records that navigator.webdriver, a property that reports true in every Playwright-controlled browser regardless of display mode, is read on 2,730 of the 7,944 successfully scanned sites, or 34 percent. Grouping 28 automation signals into eight framework families, 3,649 sites, 46 percent, probe at least one. Some of those checks look for frameworks that no longer exist: 32 percent of sites probe for PhantomJS and 31 percent for NightmareJS, which the authors read as detection scripts accumulating checks without ever removing obsolete ones.

The delivery side is just as concentrated. Counting only vendors that serve detection script from an identifiable domain, and therefore producing a lower bound, the paper finds Cloudflare Turnstile ranked sixth among all third-party scripts on the Tranco top 10,000 at 794 sites, ahead of major analytics libraries, with Google reCAPTCHA at rank 17 on 398 sites. The reading the authors offer is that the JavaScript-based detection already deployed is more extensive than the current blocking rates suggest, which is a forecast rather than a measurement, and worth treating as one.

The block rate followed the hosting provider, not the ranking

The finding with the longest reach is not the headless gap itself but what predicts it. Blocking does not correlate with site popularity in this dataset, reported as a Mann-Whitney U test at p below 0.001 with an effect size of minus 0.07. It correlates with who hosts the site. Of the 2,128 top 10,000 sites the authors resolve to Cloudflare, 788 blocked at least one condition, a rate of 37.0 percent. Akamai blocked 110 of 417, or 26.4 percent. Amazon came in at 16.3 percent, Fastly at 15.6, Microsoft at 14.8 and Google at 5.1.

Those differences track a product default rather than an editorial stance. Cloudflare's bot management is enabled by default on free plans, as the paper notes, while services like AWS WAF are opt-in and paid, so a site can sit on capable infrastructure with nothing switched on. The concentration makes the effect large: the same authors resolved the full Tranco top one million and found Cloudflare accounting for 40.0 percent of AS-matched A records, Amazon 16.9 percent, the top three providers 59.2 percent and the top five 63.5 percent. Cloudflare's share also grows as rank falls, from 14.5 percent of the top 1,000 to 37.6 percent across the full million, which the authors attribute to the free and low-cost tiers that include bot detection.

The consequence for anyone reading a scan is that a refusal carries information about your CDN's defaults and almost none about your content. It is the same category error as reading a robots.txt response without reading its status code, which is why a 404 and a 503 on robots.txt mean opposite things: the file, or the page, is not the only thing talking.

Attribution is where the paper earns the headline. Of the 1,554 blocked sites, response body matching identified a known vendor on 919, or 59.1 percent, of which Cloudflare challenge pages account for 792, DataDome 65, PerimeterX and HUMAN 26, a generic CAPTCHA 23 and Incapsula and Imperva 13. Another 353 sites, 22.7 percent, blocked in a condition-dependent way, and 334 of those 353 blocked headless Chromium, consistent with the header mechanism. The remaining 282, 18.1 percent, blocked uniformly wherever they responded, which the authors treat as geographic restriction, ASN blocks or misconfiguration. Combining the first two groups gives 1,272 sites, 81.9 percent, attributed to bot detection.

One detail in that table deserves more attention than its size suggests. On Akamai, 103 of the 110 blocked sites carried no vendor signature in the response body at all, because Akamai serves generic error pages without branding. A tool receiving those responses has no reliable way to tell a bot block from a genuine server error, which means the sample loss is not merely present but invisible. That is a harder problem than blocking, and it is the reason a report should describe what it received rather than infer why. Naming an AI crawler as blocked when the origin simply failed would be a confident wrong answer of exactly the kind this site refuses to emit.

What a headless scanner inherits from this, including ours

The reason the paper exists is a documentation gap in research, and it is worth repeating here because the same gap exists in commercial tooling. Screening 132 papers across seven venues and keeping 81 published between 2020 and 2025, the authors report that 83 percent omit any discussion of blocking, only 5 percent quantify a blocking rate, 47 percent do not say whether the browser ran headless, and 90 percent do not document rate limiting. Across their seven bot detection criteria the average paper addresses 2.4 of 7, and eight papers address none. A blocked crawl that goes unreported does not look like missing data. It looks like a clean result.

Lantad measured none of what is above, and the honest statement of its own position is a design decision rather than a figure. The scanner runs a crawler view and a browser view, and when the browser view does not come back it does not guess. The pipeline carries a render_blocked outcome, and the verdict logic in core/src/verdict.ts turns that into a partial result whose headline says the site's bot defense blocked the headless browser, sets the composite score and the grade to null, and reports only what the crawler view could establish. That behaviour predates this paper and exists for the reason the paper documents: a grade computed on half the evidence would be a guess wearing a number. The argument for it is set out in full in why we will not grade a page we could not measure, and the boundaries of what a scan claims are listed in the methodology.

There is a smaller coincidence in the repository that makes the point neatly. The log classifier in core/src/requestclass.ts carries a list of generic bot markers matched against a lowercased user agent, and headlesschrome is one of them, sitting alongside curl, wget, python-requests and phantomjs. That is a setting someone chose, not a measurement, and it is on the receiving side rather than the sending side. But it is the same string, in the same position, doing the same job: this site treats HeadlessChrome in an inbound user agent as a reason to classify a request as automated, and the study shows a large share of the top 10,000 treating it as a reason to refuse one. Nobody had to coordinate for that to happen. The string is simply there, in the open, in every headless Chromium request that has not been edited.

What this does not license is a claim that AI visibility measurement is broadly unreliable. The paired design here isolates one signal on one client from one datacenter, and a tool's exposure depends on choices this study did not test, including the user agent it declares, whether it announces itself and publishes an opt-out, whether it requests from residential or cloud addresses, and how it reports a refusal when it gets one. The last of those is the only one fully within a tool's control, and it is the one that decides whether a reader is misled.

What to check on your own site

Four things follow from this that a site owner can act on, and one that nobody can.

Check what your CDN does by default, because on this evidence it is the single largest predictor of whether an automated client reaches your pages. If you are on a plan with bot management switched on out of the box, the 37.0 percent figure above is the population you belong to, and the block is being decided before any rule you wrote is consulted. That is a separate layer from your robots.txt, and confusing them produces a plan that edits a file which was never the thing refusing the request.

Separate the clients before you change anything. A declared AI crawler is not the client this study measured. OpenAI's crawler documentation names GPTBot, OAI-SearchBot and ChatGPT-User with published purposes, and those clients are decided by robots.txt groups keyed to their tokens under RFC 9309, plus whatever verified-bot rules your provider applies to known crawlers. Checking which of them your site actually admits is a different exercise from checking whether a headless browser gets in, and the AI crawler reference enumerates the ones worth resolving. Publishers block these deliberately and in numbers, which is the finding behind the sites that block AI crawlers being the ones with editors, so an unintended block is worth distinguishing from a policy.

Read your own refusals by status code and body. A 403 with a challenge page is a bot decision, a 503 is an availability answer, and a bare Akamai-style error page may be either, which on this evidence is 103 of 110 cases on that provider. Anything reporting on your site, including what a crawler receives from your server, is subject to the same ambiguity, so treat a tool that turns an unlabelled 403 into a confident diagnosis with suspicion.

Do not generalise the headless number to agentic traffic. A browsing agent driving a real Chrome on a user's machine sends headed headers and looks nothing like the blocked condition here, which is consistent with the separate result that six of seven AI browsing agents finished their tasks with Cloudflare's AI bot controls enabled. Two studies, two instruments, one direction: the client that gets refused is the one that says what it is.

The thing nobody can check for you is how much of any published web measurement is missing for this reason. The paper's contribution is to show that the loss is systematic, correlated with infrastructure rather than content, and almost entirely undocumented. The right response is not to distrust every crawl. It is to expect a tool to tell you when it did not get in, and to be suspicious of one that never seems to have that problem.