Can AI crawlers read gated content? 2 of 66 password pages tripped our own login wall test

Can AI crawlers read gated content?

The short answer this run produced is that almost nothing in a 1,419 site corpus was gated in a way a crawler could detect, and the one page confirmed as a genuine wall was not recognised as one. Getting to that answer took two hops rather than one, because a home page is not where anybody puts a gated asset.

The first hop is the one every study on this site starts with. Lantad requested /robots.txt from all 1,419 hostnames in the two committed corpus seed files, as LantadBot/1.0, the user agent documented on the bot page. 1,101 answered HTTP 200 and 1,019 of those parsed into at least one user-agent group. 1,007 left LantadBot allowed at the root, and those are the hosts whose home pages were read. 934 of the 1,007 answered HTTP 200 with HTML. The methodology page describes the fetch path, and the sampling frame is the same one behind the crawlability study.

The second hop is new to this post. From each readable home page, Lantad took the site's own same-host links whose URL path or anchor text named a resource, report, guide, whitepaper, ebook, webinar, case study, download, template, toolkit, playbook, datasheet, brochure or checklist. 538 of the 934 home pages offered at least one. Up to three per host were requested, 1,227 in total, each checked against that host's own robots.txt for the specific path first. 5 were disallowed to LantadBot and were not fetched. 1,186 answered HTTP 200 with HTML, across 521 distinct hostnames. That set of 1,186 pages, plus the 934 home pages, is the 2,120 every figure below is drawn from.

What this scanner counts as a login wall

Lantad does not grade a page it could not measure, and a page behind a login is one of the cases that outcome exists for. The state is named not_publicly_scannable, and when it fires the report says the page sits behind a login or paywall and declines to print a letter. Prose parity, retrievability, structure and schema are all nulled, because all four would otherwise be computed against a login form and reported as facts about the page. Access survives, because access is read from robots.txt and the response headers before the body matters.

The condition that decides it is one line in core/src/verdict.ts. It requires two things at once: that the HTML carried a password input, and that the visible text of the page tokenised to fewer than four times NEAR_EMPTY_TOKENS, a constant set to 30 in core/src/config.ts. Four times thirty is 120 tokens. Both halves are decisions somebody made rather than anything measured, and the constant is a setting, so nothing here says 120 is the right number. What the measurement says is what that number does in practice.

The first half is cheap and reliable. An HTML password input is the control the specification provides for entering a secret, described by MDN as a way for a user to enter a password securely, with the typed value obscured in the field. A site that wants a password has very little choice but to ship one, which makes it a good signal for an AI visibility check to key on. An earlier post on this blog described the same outcome from the other direction, reading Google's paywall markup and noting that Lantad withholds the grade on a page it finds behind a login. That sentence was true about the code. This post is about how rarely the code gets the chance to act on it.

The 68 pages that showed a crawler a password input

68 of the 2,120 pages carried a password input in the HTML served to LantadBot: 27 of the 934 home pages and 41 of the 1,186 resource pages. That is 3.2 percent of the set, and it is the ceiling on how often the login wall outcome could possibly fire, because the outcome requires a password input before it looks at anything else.

Each of the 68 was read a second time the same day to record the figure the condition actually uses, which is the token count of the whole visible text rather than the main content count the first pass stored. 66 answered again with HTML and a password input still present. Two did not and are excluded from every figure below, which is why the denominator is 66 rather than 68.

Two of the 66 fell under 120 visible tokens. Both were facebook.com's own login screen at 78 tokens, and neither was a page belonging to a corpus site. One was reached from cam.ac.uk, whose home page carries a link anchored "Facebook"; the other from quantamagazine.org, through a social sign-in URL on its own host that redirected out to facebook.com. Redirects were followed, as they are in a normal scan, so the crawler walked off the site under measurement and the login wall it found was Facebook's. That behaviour is visible in the what GPTBot sees tool as well, and it is worth naming because a login wall detected on a third party domain tells a site owner nothing about their own pages.

The remaining 64 cleared the floor, most of them by a wide margin. The median visible token count across the 66 was 1,591 and the maximum was 8,361. At those volumes the question of whether the content is readable is a prose parity question rather than an access question, and the grade goes ahead.

Why a password protected page still gets a grade

The clearest case in the set is sunstonemontessori.org/staff/, linked from that site's home page with the anchor "Staff Resource Page". The page returns HTTP 200, and its entire main content is the sentence "This content is password-protected. To view it, please enter the password below." followed by the label "Password:". A reader arriving without the password gets no staff resources, and neither does a crawler.

This scanner graded it. The visible text tokenised to 128, eight tokens above the floor, so the condition did not hold and the page went through the normal path. The reason the count reached 128 is the whole point: 14 of those tokens were the main content, and the other 114 were the navigation, the header and the footer. The extractor read 50 boilerplate blocks on that page, 32 of them distinct, with the main menu served three times over. A site menu, a phone number and nine repeated nav links were enough on their own to make a page with fourteen words of body text look populated to the condition.

That is a design flaw rather than an accident of one site, and it follows from which field the condition reads. The extractor produces two corpora from every page: the visible text, which is every block including the boilerplate containers, and the main text, which is only the blocks it classifies as main content. The login wall condition reads the visible one. Earlier measurements on this corpus put that gap at scale: 379 of 1,066 home pages declared no main landmark at all, and on five captured pages 1,395 of 2,729 text blocks were navigation rather than content. A floor set against the larger of the two corpora is a floor a navigation menu can clear by itself, and on this evidence it usually does.

Three of the 66 pages had under 120 main content words while clearing 120 visible tokens. Sunstone is one. The other two were a news site voter guide carrying 31 main content words behind 308 visible tokens, and a financial calculators index carrying 62 behind 1,035. None of the three was reported as a login wall.

An email form is not a gate

The first version of this measurement counted email inputs instead of password inputs, on the assumption that a lead capture form is what gating looks like in practice. That assumption does not survive contact with the corpus, and the way it fails is worth publishing because it is the trap any automated gate detector falls into.

290 of the 1,186 resource pages carried an HTML email input, spread across 144 hostnames. 261 of those 290 also carried 120 or more main content words, so on nine pages in ten the form sat beside the page's own prose rather than in place of it. The crawler got the text either way. The form was gating a download, a newsletter or a demo request, not the page.

The 29 pages that did carry an email input and under 120 main content words were checked one by one against their URL, title and anchor text, and they were not gates either. They were resource and report index pages, size and care guides, app download pages, and two Facebook login redirects. Most were thin for a reason that has nothing to do with gating: an index of links to other pages has few main content words because its content is the links. Counting email inputs would have reported a gating rate of 24 percent on resource pages. The honest figure for anything resembling a wall is far smaller, and an email input is simply the wrong signal to compute it from.

There is a second reason the count would have been wrong, and it cuts the other way. No JavaScript was executed in this scan, so a form injected by a marketing script after load was invisible to it, and so was any overlay that script puts over the text. Cookie and consent gates behave the same way for a stateless crawler. Whether a given AI crawler would see such an overlay is not something the vendors settle: 2 of 9 operators state either way on rendering, and OpenAI's own bots documentation names GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User without saying anything about JavaScript at all.

What the crawler reaches once it is past the form

The pages in this set link their assets directly. From the 1,186 resource pages, 234 unique same-host document URLs were collected, ending in a PDF, Word, PowerPoint, Excel or CSV extension. Each was requested once as LantadBot/1.0 with no form submitted, no cookie set and no session of any kind.

229 of the 234 answered HTTP 200, and 220 of those began with the %PDF signature, so they were real documents rather than a login page served under a document URL. The 5 that did not answer were two 404s on seattle.gov, one 404 on cshl.edu and two timeouts. Whatever the form on the page was protecting, it was not the file.

The more interesting number is what the sites' own rules said about it. Each file path was evaluated against that host's robots.txt for the GPTBot token using this repository's own evaluator, and 214 of the 229 were allowed. 15 were disallowed, and the rules doing the work are worth reading: Disallow: / on two hosts, Disallow: *.pdf on two more, and then Disallow: /*.pdf$, Disallow: /Portals/, Disallow: /cassets/* and Disallow: /content/ on one host each. Only two of those patterns name the file type, and the wildcard forms they use are the ones Google documents for path matching, where * stands for zero or more characters and $ anchors the end of the URL. The other rules caught the documents incidentally, by blocking a directory or the whole site.

So on 214 of 229 documents the site had published a file, linked it from a page carrying a form, and left it allowed to a named AI crawler in its own robots.txt. An earlier post measured the home page layer of the same problem and found 332 PDFs on 124 home pages with 297 carrying no X-Robots-Tag; this is the layer below it, one hop in from the home page, and the pattern holds. Anyone wanting to check their own rules against a specific path can run them through the robots.txt tester.

What this measurement does not cover

The denominators here are small and the method has edges, so they belong in the post rather than in a footnote.

No JavaScript was executed. Every figure describes the HTML an origin served to a plainly identified crawler, which is the right view for the question but blind to any gate a script applies afterwards. A modal that covers the text in a browser does not appear in these counts, and neither does a form that is only mounted on the client.

Nothing here classifies the 64 pages that cleared the floor. They showed a password input and plenty of text, and no claim is made that they are or are not gated. One page in the set, Sunstone's staff page, was confirmed as a genuine wall from its own words, and that is a sample of one. The right reading of this post is not that gating is rare on the web; it is that the condition in this scanner's code fired twice in 2,120 pages, that both times it fired on somebody else's login screen, and that the one confirmed wall missed the floor on its navigation menu.

Each page was read once, or twice for the 68, on a single day. Sites change, and a page serving a form at 11:00 may serve the article at 15:00. The resource-page sample was capped at three per host, so a site with forty gated reports contributed three of them, and the 1,186 is not a census of anything.

What the corpus-wide scan does support is the shape of the risk. When a partial wall clears the floor, the grade proceeds and the four sub-scores are computed against a page whose body is a form: parity against a form, structured data against a form. That is the confident wrong grade the outcome was written to prevent, and on this evidence the threshold is set where it will rarely prevent it. The constant is a setting and changing it is a product decision, not a finding, so this post reports the measurement and stops there. The crawler registry lists the tokens evaluated, and what a full scan reports is on the pricing page.