Do AI crawlers see paywalled content? 23 of 35 pages marked not free served 500 words or more

What Lantad asked for, and what came back

The corpus is the 1,419 hostnames in this repository's two committed seed files, the same sampling frame behind the AI visibility work published on the research pages. The run has two stages, because paywall markup does not live where a site's front door is.

Stage one asked every one of the 1,419 hosts for robots.txt as LantadBot/1.0. 1,107 answers parsed into at least one user-agent group, 221 returned a 4xx, which RFC 9309 says leaves the site unrestricted, 61 returned a 5xx, which the same specification says a crawler should read as a complete disallow, and 30 failed at the transport. The site root was then resolved for the LantadBot token against every parsed file using the matcher shipped in core/src/robots.ts, which closed 73 home pages. 1,056 hosts answered the home page with HTTP 200 and an HTML content type, and 625 of those carried at least one JSON-LD block that parsed. Eleven of the 1,056 carried the property isAccessibleForFree anywhere in that markup, which is the first thing worth saying: on a home page the flag is close to absent, and that is expected, because a home page is not the thing behind the wall.

Stage two went looking for the article pages. The candidate set was 95 hosts: the news stratum, the local media platform stratum, and any host whose home page markup named a schema.org Article type. Same-host links were taken from each home page and ranked by shape, preferring a path carrying a four digit year or a long hyphenated slug and penalising the archive furniture that an internal link graph is mostly made of. Each candidate path was then checked against that host's own robots.txt for the LantadBot token before it was requested, which is why no page in this measurement was fetched against a rule telling the crawler not to.

257 article URLs were requested, at most three readable pages per host. None was skipped by a robots rule, none failed at the transport, and 11 did not answer with both HTTP 200 and an HTML content type. That leaves 246 pages on 85 hosts, and 192 of those 246 declared a schema.org Article type, spread across 69 hosts. Every figure below is drawn from that set of 246, read once each on 8 November 2026. The scanner's own request policy is published at the bot page.

Do AI crawlers see paywalled content?

71 of the 246 pages, spread over 26 hosts, declared a page-level isAccessibleForFree. 35 of those declared it false, on 13 hosts, and 36 declared it true, on 14 hosts. The 35 that said false are the population this question is about, because they are the pages that assert in machine-readable form that what they hold is not free.

Those 35 pages delivered 35,987 words of main content to an anonymous crawler between them. The median was 789 words, 23 of the 35 cleared 500 words, 16 cleared 1,000, and the single largest was 3,756 words on an nzz.ch politics article. At the other end, 11 of the 35 delivered under 200 words and two delivered none at all, both on cnbc.com, where the markup declares the page not free and the bytes carry no main content for the crawler to take.

The contrast with the 36 pages that declared themselves free is smaller than the framing of the question suggests. Their median main-content delivery was 1,046 words against 789 for the not-free pages. A difference of roughly 250 median words is not the difference between an open article and a closed one, and nobody reading only the markup would guess which group was which from what reached the crawler.

Three hosts are worth separating out because they show the two honest implementations sitting side by side. On scmp.com all three pages read delivered between 160 and 171 words of main content, so the main article region genuinely was withheld, and yet the JSON-LD on those same pages carried an articleBody property of 773, 583 and 797 words whose opening text was present in the delivered bytes. The body was supplied to the crawler through the markup rather than through the page. On nzz.ch and inquirer.com the body simply arrived in the HTML, thousands of words of it, with the markup naming the region a reader would be stopped at. Both are consistent with what Google's paywall markup documentation asks for. Neither is detectable by a crawler that does not read the flag.

This is a different measurement from the login walls counted in the gated content run, which found 2 of 66 password pages tripping the condition this scanner uses to withhold a grade. A password form is a wall a crawler can see. A metered paywall declared in JSON-LD is a wall a crawler is told about, and only if it is reading the right property. 18 of the 246 pages here showed a password input, which is the overlap between the two populations and a small one.

Which crawlers document reading the flag

The property is not a Google invention. schema.org defines isAccessibleForFree as a flag to signal that the item, event or place is accessible for free, expects a Boolean, lists CreativeWork, Event and Place as the types that carry it, and records that it supersedes an earlier property called free. That page also carries a usage band from Google's monthly web index aggregation, dated August 2026, putting the property on between 100,000 and one million domains. For scale, the same published dataset puts only sixteen schema.org types above ten million domains, so this is a property in use but not a common one.

Google's paywalled content documentation, read at source on 8 November 2026 and carrying Last updated 2026-09-08 UTC, requires isAccessibleForFree and recommends three more: a hasPart block whose type is WebPageElement, that block's own isAccessibleForFree, and a cssSelector naming the class of the region behind the wall. It applies the markup to CreativeWork or to the subtypes Article, NewsArticle, Blog, Comment, Course, HowTo, Message, Review and WebPage. It names two crawlers, Googlebot and Googlebot-News. And it states the purpose in one clause: the markup helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies.

Read that clause again, because it decides how the numbers above should be taken. The markup exists so that handing a crawler the full text of a paid article is not treated as deception. A publisher doing exactly what the documentation asks will therefore serve the words and declare the wall, and the words will reach anything else that asks politely.

So the question becomes which other crawlers read the declaration, and the answer on 8 November 2026 is none of the three largest AI vendors, at least not in anything they publish. OpenAI's crawler overview documents four tokens, GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot, and mentions paywalls, paywalled content, subscriptions, isAccessibleForFree and structured data nowhere on the page. Anthropic's crawler page documents ClaudeBot, Claude-User and Claude-SearchBot, and does the same. Perplexity's bots page documents PerplexityBot and Perplexity-User, and does the same. These are statements about what four published pages say, not about what any crawler does in private, and the difference is the whole caveat.

121 of 192 article pages said nothing either way

The larger group is the silent one. 192 of the 246 pages declared a schema.org Article type, and 121 of those 192 carried no page-level isAccessibleForFree at all. Their markup asserts that the page is an article, who wrote it and when it was published, and says nothing about whether reading it costs money.

For a free article that is harmless, and most of the 121 are presumably free. The cost lands on the paid ones, because a missing Boolean is not a default. Nothing in the markup distinguishes an open article that never needed the flag from a metered article whose publisher never added it, and the crawler sees the same thing in both cases, which is prose it is free to take. This is the same shape as the gap measured in the structured data run on home pages, where absence rather than error was the dominant finding.

The recommended region block is rarer still. 56 of the 246 pages, on 21 hosts, carried a hasPart node typed as WebPageElement. Of the 35 pages declaring themselves not free, 32 carried one and 3 did not, and all three of those were scientificamerican.com, where the page-level flag says false with no region named at all. Under Google's documentation a page-level false with no hasPart block is a claim about the whole page, which on those three pages sat above 1,027, 1,170 and 1,204 words of delivered main content.

Seven pages ran the arrangement in reverse, carrying a region block that declares a part not free while the page level says nothing, on slate.com, irishtimes.com and thenationalnews.com. Thirteen hasPart nodes declared their region free rather than paid, and seven carried no cssSelector, so there was no class for a reader of the markup to resolve. Defects of this kind are the same family as the ones counted in the schema markup validator run and the headline mismatches in the Article schema run: the markup parses, the vocabulary is real, and what it asserts is not quite what the page does. Whether any of it changes an answer engine's behaviour is a separate question this run cannot reach, and the methodology page is explicit about which claims the scanner is entitled to make.

Four selectors named a class that is not on the page

A cssSelector is the one property in this block that can be checked against the page it describes, because it names a class and the class is either in the delivered HTML or it is not. Of the 50 paywall selector declarations, 46 named a token that appears in a class or id attribute of the bytes the crawler received. Four did not.

Three of the four are slate.com, and they are worth setting out in full because the combination is unusual. Each of the three pages carries a hasPart node typed WebPageElement whose isAccessibleForFree is false and whose cssSelector is .slate-blur, and none of the three carries a page-level isAccessibleForFree. The token slate-blur appears ten times in the delivered HTML of the article read most closely, and not once as the class or id of an element: it occurs inside inline JSON rather than on the markup it is meant to select. Those three pages handed the crawler 1,455, 2,345 and 2,598 words of main content. The markup names a paywalled region, the region is not in the document under that name, and the article text is.

The fourth is a cnn.com page on an election forecast path, which declares not free and names .article__content, a class that is present on the other two cnn.com pages read and absent on that one. That reads like one template among several rather than a systematic error, which is why it is reported as one page and not as a property of the host.

A selector pointing at nothing is a familiar failure in this vocabulary. The speakable schema run found 5 of 12 resolvable selectors pointing at no readable text, and the mechanism is identical: a class name written into JSON-LD by a template that does not know what the renderer emitted. Worth noting for anyone grepping their own markup, cssSelector is used by more than one property. 19 declarations on 11 of these 246 pages sit inside a speakable block rather than a hasPart block, which is an unrelated purpose, and counting the two kinds together inflates the paywall figure. cnbc.com writes the same class string, .xyz-data, into both blocks on the same page, which is what made the conflation easy to miss on a first pass of this measurement.

What this measurement does not show

Four limits, and the first is the one that would change the headline if it were different. Every request was sent as LantadBot/1.0, an honest token with a URL in it that no vendor has an agreement with. No request impersonated GPTBot, ClaudeBot or Googlebot, because a scanner that lies about who it is cannot report on who is trusted. So this run shows what a publisher serves to an unknown and well-behaved crawler, which is the relevant case for most of the AI fleet and is not the case for Googlebot, whose access many of these publishers have negotiated or configured deliberately.

Second, three pages per host at most, and 246 pages in total. That is enough to say what these publishers' templates do and not enough to put a percentage on the news industry. The strata behind the sample are described on the research pages, and the industry split measured in the blocking run is the better guide to how differently news sites behave from the rest of the corpus.

Third, metered paywalls move. A reader's third article of the month is not the reader's first, and a crawler is nobody's third anything. A page that served 2,000 words to LantadBot on 8 November 2026 may serve a human a form, and nothing here measures what a subscriber or a logged-out browser would see. The one firm statement is about the bytes: on these pages, on that date, those words were in the initial HTML, which is the same quantity measured across the corpus in the answer-bearing facts run and the same thing prose parity scores.

Fourth, nothing here measures citation. Whether text a crawler received ends up quoted in an answer is a different pipeline, and the gates in front of it were counted in the ChatGPT citation run. A publisher who wants the open half of this story rather than the paid half can see what a crawler gets from any single page with the GPTBot view tool, and the reason this site declines to grade a page it could not read is set out in why we withhold a grade.