# Google clears cookies between page loads, so a consent gate never opens > Google's guidance on Search-related JavaScript problems, carrying Last updated 2025-12-18 UTC, states under the heading Don't rely on data persistence to serve content that its Web Rendering Service does not retain state across page loads, and that Local Storage, Session Storage and HTTP Cookies are cleared across page loads. A gate that waits for a stored consent decision is therefore waiting on something the crawler discards, and the crawler pages published by OpenAI, Anthropic and Perplexity, read on 13 August 2026, contain the word cookie zero times between them. - Canonical page: https://lantad.co/blog/consent-gates-never-open-for-a-stateless-crawler - This file: https://lantad.co/blog/consent-gates-never-open-for-a-stateless-crawler.md - Last substantive update: 2026-08-13 ## Key facts - **Published:** 2026-08-13 - **Category:** Findings - **Author:** Lantad - **Length:** 3067 words - **Takeaway 1:** Google's Fix Search-related JavaScript problems documentation, carrying Last updated 2025-12-18 UTC, states that the Web Rendering Service does not retain state across page loads, and that Local Storage and Session Storage data are cleared across page loads and HTTP Cookies are cleared across page loads. - **Takeaway 2:** The same Google page carries the heading Don't rely on data persistence to serve content, which is guidance about consent gates, age gates and preference walls whether or not it names them. - **Takeaway 3:** A consent gate is a state machine with two states, and a client whose cookie jar is emptied between page loads occupies the first state permanently, so every URL on the site is answered with the same interstitial rather than the document. - **Takeaway 4:** Read on 13 August 2026, the crawler documentation published by OpenAI, Anthropic and Perplexity names nine agent tokens between them and contains zero occurrences of the words cookie, JavaScript and local storage, so no vendor rule describes what any AI crawler does with a stored decision. - **Takeaway 5:** Lantad has not measured how often a consent gate withholds a page from a crawler. Everything reported here is read from four vendor documentation pages and the ICO's cookie guidance on 13 August 2026, and the check any site owner can run is described in the last section. ## Summary Almost every site in Europe asks a question before it shows you anything. The question is about cookies, the answer is stored, and the storing is the point: nobody wants to be asked twice. That design assumption, that a visitor who answers once is remembered, is so ordinary that it disappears into the furniture. It is also the assumption that decides whether an [AI crawler](https://lantad.co/glossary/ai-crawler) ever reaches the page underneath. The mechanism is worth stating plainly before any documentation is quoted. A consent gate withholds or reveals content according to a decision it stored earlier. A crawler that begins every page load with an empty cookie jar has no stored decision, has never had one, and cannot acquire one, because acquiring it requires a click. So the gate is not defeated by clever markup or by a crawler that runs JavaScript well. It is defeated, or rather never opened, by statelessness. This post reads what Google publishes about state in its renderer, what OpenAI, Anthropic and Perplexity publish about state in theirs, and where the line falls between a banner that covers a page and a gate that replaces it. Lantad has measured none of this. What we have measured is [which of two layers decides whether AI can read your site](https://lantad.co/blog/two-layers-decide-if-ai-can-read-your-site), and stored state is a third thing sitting behind both of them. ## What a consent gate actually depends on Strip the legal language away and a consent gate is a small state machine with two states and one transition. State one is the visitor who has not answered. State two is the visitor who has. The transition is a click, and the record of that click is written somewhere the next request can read it: an HTTP cookie, a Local Storage entry, or a value the server keeps against a session identifier. Everything else about the banner, its wording, its buttons, its position on the screen, is presentation over that one bit of stored state. The storage mechanism is ordinary HTTP. A server sets a value with [the Set-Cookie response header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Set-Cookie), the client keeps it, and the client sends it back on subsequent requests to the same origin. Nothing about that exchange is unusual, and nothing in it is optional for the gate: without a client that keeps and returns the value, there is no second state to reach. The reason the gate exists at all is regulatory rather than technical, and it is worth being precise about what the rule requires. [The ICO's guidance on cookies and similar technologies](https://ico.org.uk/for-organisations/direct-marketing-and-privacy-and-electronic-communications/guide-to-pecr/cookies-and-similar-technologies/) states that you must tell people if you set cookies, clearly explain what the cookies do and why, and get the user's consent, and that consent must be actively and clearly given. It locates the rule in regulation 6 of the Privacy and Electronic Communications Regulations, and notes an exception for cookies that are essential to provide an online service at someone's request, giving an online basket and security in online banking as examples. The same page carries a notice that it is based on the previous version of the detailed cookies guidance and that the ICO is consulting on a revised version, so this is guidance in motion rather than a settled text. Read that requirement next to the mechanism and a gap opens up. The rule is about a person: tell them, explain to them, obtain their consent. A crawler is not a person, has no preferences to record and cannot click. The regulation does not require anybody to withhold an article from a machine, and most implementations do not intend to. They withhold it as a side effect of a design that treats every stateless caller as a first-time human visitor. That is the same class of failure as [prose parity](https://lantad.co/glossary/prose-parity), where the words a reader sees and the words a fetcher receives diverge for reasons that have nothing to do with the subject of the page. ## What Google documents about state across page loads Google publishes the answer for its own renderer, and it publishes it as advice rather than as trivia. The page is [Google's guidance on fixing Search-related JavaScript problems](https://developers.google.com/search/docs/crawling-indexing/javascript/fix-search-javascript), it carries Last updated 2025-12-18 UTC, and one of its section headings reads: Don't rely on data persistence to serve content. Underneath that heading the documentation says the Web Rendering Service loads each URL, following server and client redirects, same as a regular browser. Then it draws the distinction that matters here. WRS does not retain state across page loads. Local Storage and Session Storage data are cleared across page loads. HTTP Cookies are cleared across page loads. Three storage mechanisms, named individually, all emptied at the same boundary. The phrasing repays attention because of what it does not say. It does not say the renderer refuses cookies, or ignores Set-Cookie, or fails to execute the script that writes the value. Within the loading of a single page the machinery presumably works. What does not survive is the boundary between one page load and the next, which is precisely the boundary a consent gate is built across. The gate's entire purpose is to remember an answer given on a previous page. A renderer that begins each page load empty is not refusing to consent; it is arriving, every single time, as a visitor who has never been asked. This is a different failure from the one usually discussed. The familiar question about crawlers and modern sites is whether the crawler executes JavaScript at all, which is the subject of [an earlier post here on headless Chromium being blocked outright](https://lantad.co/blog/headless-chromium-blocked-on-15-percent-of-sites). Statelessness is orthogonal to that. A crawler with a full browser engine, executing every script on the page correctly, still hits this wall, because the wall is made of memory rather than of capability. Rendering harder does not help. Google's own advice, on Google's own page, is not to build content delivery on persistence in the first place. For a site that wants to appear in [Google AI Overviews](https://lantad.co/how-to-get-cited/google-ai-overviews), the guidance and the consequence are published on the same domain. ## What the AI crawler documentation says about stored state Google is the outlier here for publishing anything at all. The three vendors whose crawlers matter most for citation say nothing on the subject, and the silence is complete enough to be worth quantifying rather than characterising. Three pages were read on 13 August 2026: [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots), [Anthropic's crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), and [Perplexity's crawler documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers). Between them they name nine agent tokens: OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User from OpenAI, ClaudeBot, Claude-User and Claude-SearchBot from Anthropic, and PerplexityBot and Perplexity-User from Perplexity. Searching the rendered text of each page for three terms returns the same result on all three: the word cookie appears zero times, the word JavaScript appears zero times, and the phrase local storage appears zero times. The Anthropic page carries a last-updated date of 7 April 2026; the OpenAI and Perplexity pages display no date at all. What each page does document is the opt-out. They name their tokens so that a site owner can write a robots.txt group against them, and that is the contract on offer: you may refuse us, and here is the name to refuse. It is a real contract and a useful one. It is also a contract about whether a crawler may fetch a URL, which leaves entirely open the separate question of what arrives when it does. Those are different questions, and a site can pass the first and fail the second without any signal that anything went wrong. The nine tokens are catalogued alongside the rest in [the AI crawler reference](https://lantad.co/tools/ai-crawlers). The nearest thing to an answer from a non-Google operator is not from an AI vendor at all. Common Crawl's own FAQ states that JavaScript is not executed and cookies are not used, which [an earlier post here read against the July archive](https://lantad.co/blog/common-crawl-july-archive-no-javascript). That is an operator describing its own behaviour in the present tense, and it is the shape of statement the AI vendors have not made. Until one of them does, the honest position for a site owner is that nothing is published about what GPTBot or ClaudeBot does with a Set-Cookie header, and that planning a content delivery mechanism around an undocumented behaviour is planning around a guess. ## Not every banner withholds the page, and the difference is in the response It would be wrong to conclude that every cookie banner hides a site from AI. Most do not, and the distinction between the ones that do and the ones that do not is mechanical rather than visual. Two banners can look identical to a person and behave completely differently towards a fetcher. The benign shape is an overlay. The server returns the complete document, the article text is present in the HTML, and a script paints a panel on top of it. A crawler that reads the response body has the content regardless of what is drawn over it, because the drawing never happened for that client and the text was in the bytes either way. This is the common case and it costs nothing. The damaging shape is a gate. Here the response body genuinely does not contain the article. Either the server checks for a consent cookie and returns an interstitial when it finds none, or the page ships as a shell whose content-loading script is deliberately held back until a consent decision exists. In both variants the document a crawler receives is the gate. Nothing is hidden by CSS, so nothing can be recovered by ignoring CSS. There is a third shape between the two, where the content is present but the scripts that fetch images, embeds or comments are held back, which degrades the page without emptying it. Which shape a site has is not usually a decision anyone made deliberately. It falls out of where the consent library was installed. A tag manager that gates other scripts behaves differently from a server middleware that gates the response, and a framework rendering on the server behaves differently from the same framework rendering in the browser, which is why the stack-level guidance for [Next.js](https://lantad.co/fix/nextjs) and for [Shopify](https://lantad.co/fix/shopify) diverges on questions that look identical from outside. The useful instinct is the one that applies to any request-dependent response: a page whose bytes depend on the caller as well as on the URL has stopped being a document at an address. That is the same structural point made about locale in [Googlebot sets no Accept-Language](https://lantad.co/blog/googlebot-sets-no-accept-language), and about metering in [what paywalled content markup names](https://lantad.co/blog/paywall-markup-names-two-google-crawlers). Consent is a third variable of the same kind, and unlike the other two it has no [structured data](https://lantad.co/glossary/structured-data) vocabulary for declaring itself. ## What to check on your own site, in one request The good news about this failure mode is that it is cheap to test, and the test needs no vendor cooperation, no log access and no tooling beyond a command line. Every claim above concerns what a server returns to a client carrying no stored state, and that is a request anyone can send. Request one of your content URLs with no cookies, no session and no logged-in browser, and read the response body rather than a rendered screenshot. Search it for a sentence you know appears in the middle of the article. If the sentence is there, you have an overlay and there is nothing to fix. If the response is the consent interstitial, or a shell with no prose in it, then that is what every stateless fetcher receives on every page, forever, and no amount of correct robots.txt configuration changes it. Repeat with a second and third URL, because a gate applied by middleware often behaves differently on the homepage than on an article. Two adjacent checks are worth running at the same time, because they answer neighbouring questions and get confused with this one. Fetching a page under a named crawler user agent, which is what [our GPTBot view tool](https://lantad.co/tools/what-gptbot-sees) does, tells you whether a server treats that agent differently from an anonymous client. Testing your rules per crawler with [the robots.txt tester](https://lantad.co/tools/robots-txt-tester) tells you whether the fetch is permitted at all. Neither answers the consent question, and mistaking one for another is easy: all three failures present identically as an absent page. State clearly what this post does and does not establish. It establishes that Google documents its renderer clearing cookies, Local Storage and Session Storage between page loads, and that three AI vendors document nothing on the subject. It does not establish how many sites gate rather than overlay, which crawlers in practice retain a cookie within a single page load, or whether any AI vendor's fetcher behaves like Google's renderer. Lantad has run no survey of consent implementations and holds no figure for their prevalence, and the sampling rules that would have to be met before publishing one are set out in [our methodology](https://lantad.co/methodology). The general principle is the one worth keeping: [AI visibility](https://lantad.co/glossary/ai-visibility) begins with what a server hands to a caller that has no history with it. ## Questions and answers **Does a cookie consent banner stop AI crawlers reading my site?** Only if it withholds the content rather than covering it. If the server returns the full article and a script paints a banner over it, a crawler reading the response body has the text. If the server returns an interstitial until a consent cookie is present, or the page ships as a shell whose content loads only after a decision is stored, then the crawler receives the gate instead of the article on every page. **What does Google say about cookies and its renderer?** Google's guidance on fixing Search-related JavaScript problems, carrying Last updated 2025-12-18 UTC, states that the Web Rendering Service does not retain state across page loads, that Local Storage and Session Storage data are cleared across page loads, and that HTTP Cookies are cleared across page loads. The section is headed Don't rely on data persistence to serve content. **Do GPTBot, ClaudeBot and PerplexityBot keep cookies?** Nothing published says. Read on 13 August 2026, the crawler documentation pages from OpenAI, Anthropic and Perplexity name nine agent tokens between them and contain zero occurrences of the words cookie, JavaScript and local storage. Those pages document how to allow or refuse each crawler, not what any of them does with a Set-Cookie header. **How do I test whether my consent implementation hides my content?** Request one of your article URLs with no cookies and no session, then search the raw response body for a sentence from the middle of the article. If it is there, the banner is an overlay and nothing is hidden. If the body is the consent screen or an empty shell, that is what every stateless fetcher receives. Repeat on two or three URLs, because gates applied by middleware often exempt the homepage. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot