Google's paywall markup names two crawlers, and both are Google's

What the paywalled content markup actually declares

The contract is small. One property is required and three are recommended, and between them they say two things: that this item is not free, and which part of the HTML is the part that is not free.

The required property is isAccessibleForFree, a Boolean, documented as whether the article is accessible to everyone or is behind a paywall, a subscription or a registration. It is not a Google invention. schema.org defines isAccessibleForFree as a flag to signal that the item, event or place is accessible for free, expects a Boolean, lists CreativeWork, Event and Place as the types that carry it, and records that it supersedes an earlier property called free. That vocabulary page also carries a usage figure from Google's monthly web index aggregation, dated July 2026, placing the property in the 100,000 to 1,000,000 domain band. For scale, counting the same published dataset in July put only sixteen schema.org types in the top band of more than ten million domains, which is where the common types sit and where this property does not.

The three recommended properties sit inside a hasPart block and are what turn a page-level flag into a region-level one. The hasPart type is set to WebPageElement. Its own isAccessibleForFree states whether that section is behind the paywall. And hasPart.cssSelector, typed as a CssSelectorType, is documented as a CSS selector referencing the class name that the author set in the HTML to mark the paywalled section. Without the hasPart block the markup says the article is not free. With it, the markup points at the exact element and says this is the part.

Two rules in the guidelines section are the ones an implementation gets wrong. The selector must be a class selector, and paywalled sections must not be nested inside one another. Both are constraints on precision rather than on intent: Google is asking the page to identify a region unambiguously, because the whole value of the declaration is that it can be checked against what the crawler received.

Nine types are supported: Article, NewsArticle, Blog, Comment, Course, HowTo, Message, Review and WebPage, all subtypes of CreativeWork. That range is worth noticing, because the discussion of paywalls is almost always a discussion of news, and the list includes Course and HowTo. A gated tutorial or a members-only guide is inside the specification, not outside it. Either way this is structured data doing the same job it does everywhere else on a page, which is making a claim about content that a machine can check rather than infer.

The declaration exists because the same behaviour is otherwise cloaking

Serving a crawler more than you serve a visitor has a name in Google's rulebook, and the name is not flattering. Its search spam policies, a page carrying Last updated 2026-05-15 UTC, define cloaking as presenting different content to users and search engines with the intent to manipulate search rankings and mislead users, and give inserting text visible only to crawlers as one of its examples. A paywalled page does exactly the first half of that sentence. The markup is how a publisher supplies the missing half, which is the intent.

Google's flexible sampling guidance, a separate page also carrying Last updated 2025-12-10 UTC, states the relationship directly: paywalled content is enclosed with structured data in order to help Google differentiate it from cloaking, where the content served to Googlebot differs from the content served to users. Two sampling models are named there. Metering gives a reader a quota of articles before requiring a subscription or a login. Lead-in offers a portion of an article without showing it in full. For most daily news publishers, that guidance expects the metering value to fall between six and ten articles per user per month.

The two models are not equivalent from a crawler's point of view, and the difference is the whole subject of this post. Metering is a decision about a reader, tracked in a cookie or a session, and a crawler carrying neither is not a reader who has used up a quota. Lead-in is a decision about a document, and a lead-in page is short for everybody.

There is a wider principle underneath the specific rule. A declaration only has value where somebody is in a position to verify it, and Google is in that position because it operates both ends: it sends the crawler and it decides the ranking. That is the same reason a claim in markup is worth checking against the page rather than trusted on sight, which is the discipline behind measuring structured data twice, once in the served HTML and once after rendering. A property asserting something about content is only as good as the content it sits beside.

Who the declaration is addressed to, and what it reaches

The instruction that decides the scope is one sentence in the crawling section: if you want Google to crawl and index your content, including the paywalled sections, make sure Googlebot, and Googlebot-News if applicable, can access your page. That is an instruction about access control, and it names the two clients that access is being granted to.

Follow the chain forward and it does reach AI surfaces, but through Search rather than around it. The documentation carries a section headed Generative AI in Search considerations, and it says that AI Overviews and AI Mode offer a preview of a topic or query based on a variety of sources including web sources, and that as such they are subject to Search's preview controls. The controls it then names are the ordinary ones: the data-nosnippet attribute to keep a section out of snippets, and the max-snippet robots meta tag to cap snippet length. So a publisher who has granted Googlebot access to paywalled text and wants to limit what is previewed uses the same two levers that govern a blue link, which is consistent with the broader point that the opt-out for AI Overviews does not live in a file on your site but in the snippet controls Search already had.

What the chain does not do is fork. There is no second branch of it in this documentation for a client that is not Googlebot. Read on 4 August 2026, the page names Googlebot and Googlebot-News and no other crawler, from Google's own estate or anyone else's: no Google-Extended, no Gemini, and none of GPTBot, ClaudeBot or PerplexityBot. That is not an oversight to complain about. It is Google documenting Google Search, which is what the page is for. It only becomes a problem when a publisher reads the markup as a general-purpose statement to the machine web, because it is not one, and nothing about implementing it changes what a non-Google client is served.

The narrowness has a precedent inside Google's own crawler documentation, where the class a client belongs to decides which rules apply to it, and fourteen named Google agents are not governed by the robots.txt group headed with an asterisk. Scope is stated per client there too. If you want to know how the paywall reads to a specific engine, the honest starting point is the platform guidance for Google's AI surfaces rather than an assumption that one declaration covers the field.

What an AI crawler receives from a paywalled page

Strip the markup question away and the mechanism is ordinary. A crawler with no subscription, no session cookie and no login sends a plain request, and the origin answers it. Whatever bytes come back are the page as far as that client is concerned. The declaration is metadata inside those bytes, not an instruction that changes them.

Which means the paywall implementation, not the markup, decides the outcome, and the same guidance says so from the other direction. The flexible sampling page tells a publisher that if they do not want the content accessible to the browser at the time of serving, they should choose a paywall implementation that does not supply the paywalled content to the browser. That sentence is written about a browser and a reader, but it describes exactly the fork that matters here, because a crawler is a client on the same side of the wire.

Three broad implementations produce three different answers. A server-side paywall omits the restricted text from the response entirely, so an anonymous crawler receives the lead-in and nothing more, whatever the JSON-LD says about the rest. A client-side paywall ships the full text and hides it with CSS or removes it with JavaScript, so a plain HTTP fetch may receive the whole article while a rendering client does not, which inverts the usual assumption that the rendered view is the fuller one. A metered paywall keyed to a cookie treats an anonymous client as a first-time reader and serves the article, until the identity check tightens and it does not.

This is the same two-layer problem this site keeps returning to, because what the server returns and what a browser assembles are two different documents and only one of them is what a JS-blind crawler reads. On a paywalled page the gap between them is not a bug to fix. It is the product working. The measurement question is only ever which of the two a given client got, and the way to answer it for a specific crawler name is to fetch the URL as that crawler and read what came back rather than to reason from the markup.

One consequence deserves stating plainly, because it cuts against the instinct to add the property and move on. Adding isAccessibleForFree to a page whose server already withholds the text does not make that text available to anyone. It correctly labels a page that a non-Google crawler will still receive as a lead-in. The label is right and the content is still absent, and no amount of accurate metadata is a substitute for bytes on the wire.

Why robots.txt has no way to express a paywall

A reasonable next question is whether the access rules a site already publishes can carry any of this, and the answer is no, for a structural reason rather than a missing feature. RFC 9309 gives a crawler allow and disallow rules matched against a product token. The vocabulary is binary and it is about paths. There is no rule that says this path is available in part, or available to a subscriber, or available for a fee. A publisher can disallow a section and a crawler that honours the file will not fetch it, but that is a closed door rather than a declared partial view, and a robots.txt group is selected by a name the client chooses to send rather than by any credential it holds.

The layer that does have vocabulary for conditional access is HTTP, and it is worth being clear about which codes actually mean what. HTTP 401 indicates a request that failed because it lacked valid authentication credentials, and MDN documents it as sent with a WWW-Authenticate header naming the scheme the server expects. That is the honest answer to an unauthenticated client, and it is unambiguous: nothing was served, and the reason was credentials. A 403 refuses without inviting a retry. And the code that reads as if it were designed for exactly this case is the one that cannot be relied on, because 402 Payment Required is reserved rather than defined, which is a problem Lantad has already written about at length.

In practice most paywalls answer 200 with a short page, which is the least informative option available and the one that makes external measurement hardest. A 200 carrying four hundred words is indistinguishable, from outside, from a thin page that simply has four hundred words. No status code was spent saying otherwise. The structured data is the only signal that separates the two, and it is a signal aimed at one search engine.

None of this is an argument that publishers should be doing something different. It is an argument that the vocabulary available to them is thinner than the discussion assumes, and that the gap sits in the same place as several others: the names a crawler publishes are few, and the controls keyed to those names are fewer. A paywall is an access decision made per request, and robots.txt was never a per-request instrument.

What to check on your own paywalled site

The check that answers the most is also the cheapest, and it is not a markup validation. Request one paywalled URL with no cookies, no session and no logged-in browser, and read what comes back. Everything else in this post is context for interpreting that response. If the restricted text is in it, every client that fetches without credentials has it too. If the text is not in it, no property in the JSON-LD puts it there for a crawler that is not Googlebot.

Then check the three things that make the declaration internally consistent, because a declaration that does not match the page is worse than none. The class named in hasPart.cssSelector should exist in the HTML that is actually served, not only in the version a logged-in reader sees. The page-level isAccessibleForFree should be false on gated articles and absent or true on free ones, since a site that marks everything as paywalled has told Google nothing. And the type should be one of the nine supported ones. All three are checkable by reading the response, which is the same standard this site holds itself to when it reports what a crawler met on a real page.

It is worth being clear about what a scanner can and cannot tell you here, including this one. Lantad reads robots.txt and the served and rendered page as an identified crawler, and it publishes the user agent it uses on its bot page. It has no subscription to your site, so it sees what any unauthenticated client sees, which is useful for exactly the question above and useless for the question of whether Googlebot is receiving more. Nobody outside Google can answer the second one from the public web.

That is also why the tool refuses to grade these pages rather than scoring them low. When the served HTML carries a password input and the body is under four times NEAR_EMPTY_TOKENS, a constant set to 30 in core/src/config.ts and therefore a threshold of 120 tokens, the verdict logic in core/src/verdict.ts returns the state not_publicly_scannable and the headline says the page sits behind a login or paywall so a grade would be a guess. Those numbers are settings that somebody chose, not findings about the web. The reasoning behind them is the same one this blog opened with, which is that a confident wrong grade costs more than a withheld one, and a paywall is the clearest case there is: the content the score would be measuring was never sent. Reporting a low AI visibility score for a page that is doing its job would be a measurement of the scanner's ignorance, dressed as a measurement of the site.