BlogFindings
Do cookie banners block AI crawlers? 269 of 1,093 home pages ran a consent platform, and it cost 0.44 percent of the words
Lantad requested the home page of all 1,419 hostnames in this repository's committed corpus on 21 September 2026 and read the raw bytes with no JavaScript executed. 1,093 answered HTTP 200 with HTML. 269 of them loaded one of 25 named consent platforms and 257 carried consent wording in the delivered HTML, but that wording came to 6,143 words against 1,381,167 visible words in total, and no page in the corpus gave a cookie notice more than half its text.
This blog has already published the documentation half of the argument, in a post on how a consent gate never opens for a stateless crawler, and that post said plainly in its own takeaways that Lantad had not measured how often the gate actually costs a site its text. This is that measurement. On 21 September 2026 Lantad requested the home page of all 1,419 hostnames in this repository's two committed corpus seed files as LantadBot, followed redirects, executed no JavaScript, and read the bytes that came back. The result does not support the wall. Across the pages that answered, consent notices came to 0.44 percent of the words an AI crawler could read, and the pages carrying a consent platform returned more text than the pages carrying none.
In short
- Lantad requested the home page of 1,419 hostnames on 21 September 2026 and read the raw bytes with no JavaScript executed. 1,093 answered HTTP 200 with HTML, and consent notices accounted for 6,143 of the 1,381,167 visible words on them, which is 0.44 percent.
- Do cookie banners block AI crawlers on the pages that run them? Not on this corpus: the 269 pages loading a named consent platform returned a median of 1,275 visible words against 850 on the 824 pages that loaded none.
- On 137 of the 269 pages running a consent platform, not one word of the notice appeared in the served HTML, because the platform's tag inserts the banner in the browser and a client that runs no JavaScript never reaches that step.
- Nine of the 1,093 home pages gave a consent notice more than a tenth of their visible words and three gave it more than a quarter, the highest being famousbrands.co.za at 373 of 964 words on 21 September 2026.
- Three pages running a consent platform returned no visible words at all, and on all three the cause was client-side rendering rather than the notice: bloomandwild.com, croatia.hr and skyscanner.net each served an empty application shell.
What was measured, and how a banner was counted
Each of the 1,419 hostnames received one request for its home page over HTTPS, from one network location, with redirects followed and a twenty second timeout. 31 never returned a status at all, 19 failing DNS resolution, 10 timing out and 2 failing in the client. 213 answered HTTP 403 to this crawler and 53 answered HTTP 503. 1,094 answered HTTP 200, and 1,093 of those carried an HTML content type. Those 1,093 pages are the denominator throughout, and they are not a random sample of the web: a site that refuses this scanner never reaches the test, so the surviving set leans toward sites that admit crawlers. The request discipline is the same one described on the methodology page and used for every corpus measurement on this blog.
A consent layer was counted two separate ways, because the two answer different questions. The first is whether the page loads a consent platform at all, detected by a string that the platform's own tag or content delivery network leaves in the HTML: cdn.cookielaw.org and optanon for OneTrust, consent.cookiebot.com for Cookiebot, and 23 more signatures covering 25 named platforms in total. The second is whether the words of a notice are actually in the delivered document, detected by running the same extractor the scanner uses for prose parity and then flagging text blocks that use wording only a cookie notice uses, such as we use cookies, strictly necessary, reject all and manage preferences. A page can match one test and not the other, and the gap between the two turns out to be the most interesting number here. Word counts throughout are visible words after script, style, noscript, template, svg and iframe subtrees are dropped, the same basis used when this blog counted how many home pages sent a crawler zero words.
GET /, 1,419 hosts, no JavaScript
- GET https://{host}/ User-Agent: LantadBot/1.0 (+https://lantad.co/bot) 1,419 sent
- HTTP 200 with an HTML content type 1,093
- match 25 named consent platform signatures in the bytes 269 pages
- extract visible text, flag blocks using consent wording 257 pages, 504 blocks
- consent words against all visible words 6,143 / 1,381,167
Do cookie banners block AI crawlers?
On this corpus, on this date, they did not. The 1,093 home pages carried 1,381,167 visible words between them. 6,143 of those words were consent notices, which is 0.44 percent. Of the 257 pages carrying any notice text at all, the median page carried six words of it: enough for a link reading cookie policy in a footer, not enough to displace anything. 165 of the 257 carried ten words or fewer, 68 carried between eleven and fifty, and only 24 carried more than fifty. Not one page in the corpus gave a cookie notice more than half its visible text, and the single largest absolute figure was interactivebrokers.com at 586 words, against 3,921 words of visible text on the same page.
The comparison that cuts hardest against the wall claim runs the other way. The 269 pages loading a named consent platform returned a median of 1,275 visible words. The 824 pages loading none returned a median of 850. Pages with a platform were also far less likely to be nearly empty: 8 of 269 returned fewer than 200 visible words, against 130 of 824 without one. That is not evidence that a consent platform helps, and this post does not claim it is. It is evidence about which sites buy one. A consent platform is a purchase, and the organisations that make it are the ones with legal departments, content teams and budgets, which are the same organisations that publish a lot of text. Consent itself is not optional for them: the Information Commissioner's Office guidance on cookies, read on 21 September 2026 and carrying no date of its own, states that consent must be "freely given, specific and informed" and that "you cannot set non-essential cookies on your website's homepage before the user has consented to them". So the banner correlates with the kind of site that has something to say. Whether a machine can then read it is a question about AI visibility that the banner turns out to have very little to do with, and the mechanism for that is the subject of the earlier post on stateless crawlers.
Which consent platforms the corpus actually runs
OneTrust is not merely the leader here, it is half the market on this sample: 135 of the 269 pages that matched any platform matched OneTrust, which is 50.2 percent. Cookiebot follows at 28, then TrustArc at 14, Usercentrics at 13, and CookieYes and Osano at 11 each. Thirteen further platforms matched between one and five pages each. 21 pages matched more than one platform, which is why the counts sum above 269: instacart.com matched three signatures, and cnn.com, aljazeera.com, hollywoodreporter.com and newegg.com each matched two. Those are usually a migration that was never finished rather than a deliberate pairing, and each one is a second tag loading on every page view.
The rate varies a great deal by the kind of site. Among travel hostnames 37 of 79 loaded a platform, the highest of any stratum, followed by SaaS at 52 of 119 and ecommerce at 28 of 66. At the other end, 6 of 87 government pages matched, 3 of 45 single page app startups, and none at all of the 54 Wix and Squarespace sites. That last figure needs care rather than a conclusion: those builders ship their own consent notice as part of the platform and inject it in the browser, so the absence of a third-party signature is not the absence of a banner. It is a limit of what a signature test can see, which is the same limit that applies whenever a scan reads only what the server sent. A reader who wants to see the delivered bytes for a single page can run what GPTBot sees against it, and the same discipline of reading only what arrived is what produced this blog's count of how many home pages offered a crawler no internal path.
| Consent platform | Home pages | Share of the 269 |
|---|---|---|
| OneTrust | 135 | 50.2% |
| Cookiebot | 28 | 10.4% |
| TrustArc | 14 | 5.2% |
| Usercentrics | 13 | 4.8% |
| CookieYes | 11 | 4.1% |
| Osano | 11 | 4.1% |
| iubenda | 8 | 3.0% |
| Quantcast | 7 | 2.6% |
| HubSpot banner | 7 | 2.6% |
| Complianz | 7 | 2.6% |
| cookie-law-info plugin | 7 | 2.6% |
| Didomi | 6 | 2.2% |
On 137 pages the banner never reached the HTML at all
This is the finding that explains all the others. Of the 269 pages that loaded a consent platform, 137 carried none of that platform's words anywhere in the delivered document. The tag was there. The notice was not. That is slightly more than half of every page running a platform, and it is not a fault: it is how these products are built. The platform ships a script, the script fetches a configuration, and the banner is constructed in the browser and inserted into the page after load. A client that executes no JavaScript receives the article and never receives the notice, because the notice does not exist until a step that client does not perform.
The same property is what makes a consent gate harmless to a crawler in the first place. Google's own documentation on fixing Search-related JavaScript problems, carrying Last updated 2025-12-18 UTC, states under the heading "Don't rely on data persistence to serve content" that "WRS does not retain state across page loads" and that "HTTP Cookies are cleared across page loads". A gate that reveals content once a stored decision exists is therefore waiting on something the renderer discards every time. Whether the AI crawlers execute JavaScript at all is a separate question with a separate answer, and this blog found that 2 of 9 operators say either way. For the 137 pages here the answer does not matter much: a crawler that renders sees the banner drawn over an article it has already received, and a crawler that does not render never sees the banner. What neither gets is a page withheld. It is worth contrasting this with the cases where client-side machinery does hide real content, such as when text sits in hydration JSON and scores the same as no text at all. A consent tag is the opposite shape: it adds something in the browser rather than withholding something from the bytes.
Flow: GET / as a crawler to Server returns HTML; Server returns HTML to No banner words in bytes; No banner words in bytes (in a browser) to Browser runs the CMP tag; Browser runs the CMP tag to Banner drawn over page; No banner words in bytes (for LantadBot) to Crawler reads article text.
Where the notice did take words, and how few pages that was
Nine pages of 1,093 gave a consent notice more than a tenth of their visible words, and three gave it more than a quarter. famousbrands.co.za is the worst case in the corpus at 373 of 964 words, or 38.7 percent, a full cookie table rendered server-side into the document. plannedparenthood.org follows at 310 of 960 words and cshl.edu at 359 of 1,178. These are the pages where the notice is not injected by a script but written into the template, so a crawler receives the cookie policy and the page content as one undifferentiated run of text.
Whether that is a problem depends on where the words land, and the extractor answers that too. On 56 of the 1,093 pages the consent wording fell inside blocks classified as main content rather than boilerplate, which is the case that matters: text in a navigation or footer region is already discounted, and this blog has measured how much of a page that region accounts for, finding that 1,395 of 2,729 text blocks were navigation. interactivebrokers.com is the clearest example, with all 586 of its consent words landing in main. The practical cost is dilution rather than blocking. A passage about cookie categories sitting inside the main region competes with the page's actual subject for whatever the extracting system weighs, in the same way that container choices change what survives extraction: deleting one element cost a captured page 1,665 of 13,615 words. None of that is a claim that an answer engine ranks these pages lower, which is not something this measurement can see. What it can say is that on 1,084 of 1,093 pages the notice took under a tenth of the text, so for almost every site the cookie banner is not where the structured data and prose budget is being lost.
Three pages sent no words at all, and the banner was not the reason
Three of the 269 pages running a consent platform returned zero visible words to this crawler, and it would be easy to file them as the wall working exactly as advertised. They are not that, and checking why is the difference between a measurement and a headline. bloomandwild.com runs Cookiebot and returns an Angular application shell whose body holds eighteen script elements and no prose. croatia.hr runs Cookie Control and returns a body containing a noscript element that says in plain words that the application does not work properly without JavaScript. skyscanner.net runs Ketch and returns a loading spinner inside an empty application root. In all three cases the consent platform is present and irrelevant: the page withheld its text from a non-rendering client because it is built to assemble itself in the browser, which is the failure this blog measured when JavaScript supplied 7.6 percent of the prose across 271 pages. The fix for these three is a rendering decision of the kind covered in the per-stack guide for Next.js, not a consent decision.
One page did visibly react to the missing cookie, and it is instructive that it is only one. nature.com redirected the request to its own home page carrying the query string error=cookies_not_supported, which is a site explicitly noticing that the client kept no state. It then served 1,433 visible words including its full run of article headlines, of which 5 words were consent wording. The site detected the stateless client, announced the detection in the URL, and delivered the page anyway. That is the whole finding in miniature. Elsewhere the missing text has an entirely different cause, as when text inside shadow DOM reaches the browser and not the extractor, and attributing any of those absences to the cookie notice would be wrong.
-
bloomandwild.com0 words Cookiebot present. The body is an Angular application shell carrying eighteen script elements and no prose, so the cause is client-side rendering. -
croatia.hr0 words Cookie Control present. The body carries a noscript element stating the application does not work properly without JavaScript. -
skyscanner.net0 words Ketch present. The body holds a loading spinner and an empty application root, with the page assembled in the browser. -
nature.com1,433 words OneTrust present. Redirected to a URL carrying error=cookies_not_supported and then served the full home page, 5 words of it consent wording.
What this measurement does not cover
Four limits, stated because the figures are worth less without them. First, no JavaScript was executed, so every count describes what a non-rendering client received. A crawler that renders would meet the banner as a visual overlay, and this measurement says nothing about whether such an overlay affects what an agent extracts or clicks. Second, the platform test is a signature match against 25 named products, so a hand-built banner or a builder's bundled notice is invisible to it. That cuts both ways: 125 pages carried consent wording with no platform signature matched at all, which is why the two tests are reported separately rather than merged into one number. Third, 31 hostnames never resolved and 213 refused this crawler with HTTP 403, so 326 sites are missing from every figure and they are not missing at random. Fourth, only the home page of each site was read, so nothing here describes an article template, a paywall, or a regional variant that behaves differently.
The check worth running on your own site is short and needs no tool. Fetch your own home page the way a crawler does, with no browser and no cookie jar, and read what comes back: if your article text is in those bytes, the banner is not your problem, whatever the advice says. If it is not, the cause will almost certainly be rendering rather than consent, which is a different repair with a different cost. For the neighbouring question of which crawlers your rules actually admit, the robots.txt tester evaluates a file per crawler token rather than in general. Everything in this post came from one request per host on one day from one network location, and the underlying corpus and method are the same ones used across this blog's research. A figure here supports a statement about these 1,419 hostnames and nothing wider.
- Consent notices took 0.44 percent of visible words 6,143 words of 1,381,167 across the 1,093 home pages that answered HTTP 200 with HTML.
- Pages with a consent platform returned more text Median 1,275 visible words against 850 without one, which describes who buys a platform rather than what it does.
- Rendered behaviour was not measured No JavaScript was executed, so nothing here describes what a banner overlay does to an agent that renders the page.
- Builder-bundled notices are invisible to the test None of the 54 Wix and Squarespace pages matched a third-party signature, which is a limit of signature matching rather than an absence of banners.
- 326 hostnames are absent from every figure 31 never resolved and 213 answered HTTP 403 to this crawler, so the surviving corpus leans toward sites that admit crawlers.
Lantad
Published .
Almost every commercial website in Europe opens with a question about cookies, and there is a widely repeated claim about what that question does to a machine reader. The claim is that the banner is a wall. A crawler arrives, meets the notice, and never reaches the article underneath, so the site is invisible to the systems that answer questions about it. The claim is plausible, it is cheap to repeat, and it is almost never accompanied by a count.
Common questions
Do cookie banners stop ChatGPT or other AI crawlers reading a page?
Not on this corpus. Across 1,093 home pages measured on 21 September 2026, consent notices accounted for 0.44 percent of the visible words, and on 137 of the 269 pages running a consent platform the banner's words were not in the delivered HTML at all because the platform builds the notice in the browser. No page in the corpus gave a cookie notice more than half its text.
Why did pages with a consent platform return more text, not less?
Because a consent platform is a purchase, and the organisations that buy one tend to be the ones that publish a lot. The 269 pages running a platform returned a median of 1,275 visible words against 850 on the 824 that ran none. That is a fact about which sites buy consent software, not evidence that the software helps, and this measurement cannot separate the two.
Which consent platform is most common?
OneTrust, on 135 of the 269 pages that matched any of the 25 named platforms on 21 September 2026, which is 50.2 percent. Cookiebot was second at 28 pages and TrustArc third at 14. 21 pages matched more than one platform signature, usually the residue of a migration.
If my home page shows a crawler no text, what is the likely cause?
Client-side rendering, not the cookie banner. All three corpus pages that ran a consent platform and returned zero visible words were application shells that assemble themselves in the browser, and the one site that explicitly detected the stateless request, nature.com, still served 1,433 visible words. Fetch your own page with no browser and read the bytes before blaming the notice.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Prose Parity
Prose Parity is Lantad's metric for the share of a page's rendered, human-visible text that is present in the raw HTML an AI crawler receives, where 100% means a crawler sees everything a human visitor sees.
Google clears cookies between page loads, so a consent gate never opens
Google's guidance on Search-related JavaScript problems, carrying Last updated 2025-12-18 UTC, states under the heading Don't rely on data persistence to serve content that its Web Rendering Service does not retain state across page loads, and that Local Storage, Session Storage and HTTP Cookies are cleared across page loads. A gate that waits for a stored consent decision is therefore waiting on something the crawler discards, and the crawler pages published by OpenAI, Anthropic and Perplexity, read on 13 August 2026, contain the word cookie zero times between them.
Client side rendering: JavaScript supplied 7.6 percent of the prose, and all of it on 11 of 271 pages
Lantad loaded the home page of 392 hostnames twice on 20 September 2026, from the same browser at the same moment, changing one thing between the two loads: whether JavaScript was allowed to run. On the 271 pages that answered HTTP 200 both times, the unrendered document held 173,138 words and the rendered one held 187,422. The average is undramatic and the distribution is not: 100 pages were word for word identical, and 11 returned nothing readable at all until the bundle ran.
AI SEO: 17 of 380 home pages sent a crawler zero words
Lantad requested the home page of 392 hostnames on 11 September 2026 and read the bytes with no JavaScript executed. 380 answered HTTP 200, and 17 of those carried no visible text at all. Every one of the 17 carried a title element, 13 carried a meta description, and the two together came to a median of 17 words per site.
