AI Overviews and the one crawler our scanner does not model

The short answer, split into what we check and what we do not

Three things have to be true before a page can be quoted in an AI answer that Google assembles, and they are worth separating because a Lantad scan covers one of them well, one of them partially, and one of them not at all.

First, Google's crawler has to be allowed to fetch the page. That is a robots.txt question about Googlebot, and Googlebot is named on Google's list of its own common crawlers. Our free robots.txt tester will not answer this for you. It evaluates your file against fifteen named tokens and prints a row for each, and if you count the rows you will find no Googlebot among them. That is the honest state of the tool, not a rendering bug.

Second, the words you want quoted have to survive the trip. Google's JavaScript basics documentation, last updated 2026-03-04 UTC, states that "Googlebot queues all pages with a 200 HTTP status code for rendering, unless a robots meta tag or header tells Google not to index the page", and that "Once Google's resources allow, a headless Chromium renders the page and executes the JavaScript". Read as written in Google's JavaScript SEO basics guide, that is Google saying it does render, on a queue, when resources allow, and conditioning the whole guarantee on a directive. Hold on to that conditional clause, because a later section is about the fact that we parse exactly that directive and then do nothing with it. This is also the half we measure well, because Prose Parity is exactly a measurement of how much of your rendered text is already present in the server response. A page that passes it is legible to a renderer and to a crawler that never renders. A page that fails it is betting on somebody else's queue.

Third, the text must not be marked as off limits for quoting. Google documents several ways to do that, our scanner detects one of them, and the distance between those two counts is the subject of a later section.

What follows is not a list of tips. Tips belong on the reference pages, where the how to get cited hub keeps one page per answer surface. This is the part of the picture that a vendor blog usually leaves out: which of those three things our own product actually observes, and where the reader has to go and look for themselves because we have not built it. Every claim below is a line you can open in a file in this repository. The scoring rules behind them are published on our methodology page, which already carries a section on what the scan cannot tell you.

Fifteen tokens in the registry, and Googlebot is not one

Every per-crawler verdict Lantad prints comes out of one array in core/src/bots.ts called BOT_REGISTRY. It holds fifteen tokens: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot. Nine of the fifteen belong to OpenAI, Anthropic and Perplexity, which is a fair description of where the AI crawler conversation started and a poor description of where Google sits in it. If you want the category defined properly rather than described in passing, what an AI crawler is is the reference entry.

The only Google row in that array is Google-Extended, and the comment above it explains why that token has no user agent to probe: its requests come from ordinary Googlebot infrastructure, so there is nothing distinct to send. There is no Googlebot data row, no fallback, and no special case further down the file.

The string does appear elsewhere in core/src, and it is worth being exact about where, because "we never mention Googlebot" would be an easier sentence than the true one. It appears in a comment in the robots parser, illustrating how the most specific matching group token wins. It appears in the comment on the Google-Extended row already quoted. And it appears once as real data, in a list called SEARCH_CRAWLER_TOKENS inside core/src/requestclass.ts, which classifies inbound requests to our own zone so that ordinary search crawling does not pollute the human count on our own dashboard. That list is about traffic arriving at lantad.co. It is never consulted when a scan evaluates somebody else's robots.txt, and the file's own comment says it is kept deliberately separate from the registry, which is AI specific by charter.

There is a sharper way to put the gap. The robots parser is token agnostic: it takes a token string and selects the most specific matching group for it, so nothing about the parser is incapable of answering for Googlebot. The parser is simply never called with that token, because the caller loops over BOT_REGISTRY and the registry has no such entry. The limitation is a data limitation rather than an engineering one, which is what makes it embarrassing rather than difficult.

That single absence then propagates everywhere. The scan pipeline loops over BOT_REGISTRY to build the per-bot access matrix. The report matrix imports the same list. The free AI crawler reference is generated from it, the free robots tester builds its rows from it, and the bulk audit path reads it too. Even the crawler count printed on our own 404 page is that array's length. So a Lantad scan is not merely quiet about Googlebot on one screen; it is structurally incapable of holding an opinion about Googlebot at all.

A second registry fact sharpens this. A function called probeableBots filters the array down to the tokens that have a user-agent string, because those are the ones we can send a live differential request as. Twelve of the fifteen pass that filter. Three do not, and Google-Extended is one of them, alongside anthropic-ai and Applebot-Extended. We wrote about that trio in the crawler tokens that never appear in your logs, so the short version is enough here: Google is represented inside our product by a token that never sends a request. It can be read in a robots.txt file and evaluated as a rule, but it cannot be observed arriving.

Our own scanner does send requests, and it sends them honestly. The user agent is LantadBot/1.0 with a link to the page explaining what our bot does, and it never claims to be somebody else's crawler. That policy is at least the reason we can publish this post without a second embarrassment: we have never told you a Googlebot number, because we have never had one to tell. The gap is in the registry, and the registry is the whole of it.

The one Google token we do carry costs nothing to block

Google-Extended sits in the registry with a purpose field set to training. That word does real work in our scoring. A constant named PENALIZED_PURPOSES lists exactly two purpose classes, search and user_agent, and the function that computes the Access sub-score iterates over that constant and nothing else. A robots.txt Disallow aimed at a training token therefore reaches no branch that subtracts anything.

The arithmetic is easy to follow. Access is worth 25 of the 100 composite points. Inside Access, a robots.txt disallow that lands on a penalized purpose class costs 0.4 of a sub-score that starts at 1.0, which is 10 composite points per class blocked. Google-Extended is not in a penalized class, so blocking it costs 0.4 times zero. The code comment sitting above that constant says the intent out loud: blocking a training crawler stays a deliberate owner choice that costs nothing. Adding a purpose class to the penalized list is deliberately a compile error until somebody decides the wording the report will use, which is how the score and the sentence explaining the score are kept from drifting apart.

The reason that arithmetic belongs in an AI Overviews post is that Google-Extended is frequently reached for as the AI Overviews lever, and Google's own documentation says it does not govern inclusion in Search. Our scoring already agrees with the narrow reading, and it agrees in the only way a score can, which is by declining to move. What that token governs, what the Search Console setting introduced in June 2026 governs, and why neither leaves anything on your site for an external scanner to read, is set out with the dates in the AI Overviews opt out does not live on your site, published the same day as this post. There is no reason to argue it twice here, and the surrounding conflict between edge tooling that files Google under training bots and Google's own narrower design is in the two layers that decide whether AI can read your site.

What is new here is the consequence for a Lantad report specifically, and it is not a flattering one. A site owner who disallows Google-Extended and then runs our scan sees no change in the number, and that is correct behaviour rather than a missed detection. A site owner who disallows Googlebot and then runs our scan also sees no change in the number, and that is a gap. The two cases look identical in the report and they could hardly be less alike, because one is a deliberate preference about model training and the other is a site removing itself from Search. A report that renders both as silence is not lying, but it is letting the reader supply a meaning we never measured, which is the failure mode this whole product exists to avoid.

Counted honestly, Lantad's coverage of Google's AI answer controls comes to one token we score at zero on purpose, one crawler we do not model, one account setting no external tool can see, and one snippet directive we detect out of the several Google documents. That is three gaps, one control we detect, and several controls we do not.

The controls Google documents for limiting quotes, and the one we detect

Google's robots meta tag documentation, last updated 2026-03-24 UTC, describes several distinct ways to limit what Google may show. The nosnippet rule is documented as "Do not show a text snippet or video preview in the search results for this page." The max-snippet rule is documented as "Use a maximum of [number] characters as a textual snippet for this search result." The data-nosnippet attribute is documented as an HTML-element-level control, described on the same page as something you can place on span, div and section elements to designate textual parts of a page not to be used as a snippet. And noindex is the separate, blunter instrument that keeps the page out of results altogether.

Here is what our scanner does with that set. It builds one set of directive tokens from the meta robots tag and the X-Robots-Tag response header, then tests that set for exact membership of two strings: noai and nosnippet. Nothing else can trigger it. A third token, noimageai, is named once when the defect's evidence string is composed, so it can appear in the wording of a finding that something else raised, but it can never raise one. The consequences follow mechanically from that test.

A max-snippet directive never matches, because max-snippet:50 is not the string nosnippet and the test is exact-token equality rather than a prefix check. A data-nosnippet attribute never matches either, and for a deeper reason: it is an HTML attribute on an element rather than a directive in a meta tag, and our extractor parses no such attribute anywhere. Searching core/src for data-nosnippet returns nothing at all. A noindex directive is the strangest case of the three, because we do parse it. The extractor splits the meta robots content on commas and pushes every token into the raw extraction, and the pipeline does the same for the header. So noindex is sitting in memory during every scan we run, and then nothing reads it. Search core/src for the word and you get no hits at all, because it is only ever handled as an anonymous member of a list. It is never scored, never turned into a defect, and never rendered into a report row.

When the one directive we do detect fires, the penalty is 0.15 of the Access sub-score, and since Access is a quarter of the composite that is 3.75 points out of 100. It also raises a named defect in the report rather than only moving a number, which is the pattern we hold to across the whole scoring surface: a deduction the report cannot explain in words is a deduction we do not ship. That rule is written up in why we withhold a grade and the full deduction table is on the methodology page.

If you came here to protect an AI Overviews citation, this section is the most actionable one in the post, and the action is not to run our scan. Search your own templates and your CDN configuration for max-snippet, for data-nosnippet, and for noindex, then check whatever you find against Google's documentation directly. A single data-nosnippet wrapper left on a summary block by a previous developer is invisible to us and, per Google's own description of the attribute, meaningful to Google. The Google AI Overviews reference page lists the controls in one place, and Google's robots meta tag reference is the source of record for all of them.

What the scan is still worth if AI Overviews are your reason for reading

Having spent four sections on what we do not measure, here is the honest case for the thing we do.

Prose Parity is 50 of the 100 composite points, exactly as much as Access, Structure and Schema put together. The measurement is simple to describe and awkward to fake. We fetch the page once over plain HTTP with our own honest user agent, LantadBot/1.0, no spoofing and no header games. We then load the same URL in a real browser and take the visible text again. Parity is the share of the rendered text that was already present in the raw response, scored over eight-token shingles so that boilerplate phrases rarely collide by accident. If your parity is 1.0, everything a browser shows was in the server response. If it is 0.3, most of your page arrives only after JavaScript runs.

That number is directly load-bearing for a crawler that does not execute JavaScript, and it is a weaker predictor for one that does. We should say so plainly on a page about AI Overviews, because Google's JavaScript documentation states that it renders with headless Chromium once resources allow. A renderer will eventually see text that a raw fetch misses. The word doing the work in Google's own sentence is queues, and a queue is a thing you are waiting in rather than a thing you have arrived at. The rendering on the web guidance published on web.dev, last updated 5 January 2026, defines server-side rendering as "Rendering an app on the server to send HTML, rather than JavaScript, to the client", which is the same recommendation from the other direction.

The numbers we have are small and dated, and we present them that way. Six real pages were captured on 15 July 2026 and stored as fixtures. Five of the six produced grades, with parity between 0.958 and 1.000 and composites between 88.3 and 96.7, and the sixth produced no grade at all. Those captures were the subject of what a crawler meets on a real storefront, which is where the page-by-page reading belongs. Six pages is not a survey of anything and we do not report it as one, which is also why our research page still declines to publish a picture of the web.

The reason to raise those five numbers in this particular post is narrow. Every one of them was produced without asking anything about Googlebot, and every one of them is still true of a page a Google renderer will meet, because parity is a property of what your server sends and what your server sends does not vary by which crawler asked. Unless, of course, you have gone out of your way to make it vary, which is a separate finding our scan does raise and calls cloaking. So the half of the AI Overviews question we can answer is the half that is stack-shaped rather than vendor-shaped,.

Parity is not the whole score either, and a page can deliver its text perfectly and still lose points elsewhere. Structured data is a separate ten points, and pages that ship none are not rare in the one sample we have actually counted: of thirty pages fetched on 26 July 2026 for the entity-signal work, six returned HTTP 200 with no JSON-LD at all, developer.mozilla.org among them. That count is recorded in core/src/config.ts and used in the five signals post. Thirty pages is a sample rather than a census and it is quoted here as one. If you want to check the structured-data half against Google directly rather than against us, the Rich Results Test tests a publicly accessible page to see which rich results its structured data can generate. And if your parity problem turns out to be a framework problem, the Next.js fix guide is more use than another scan, as is the React guide when the framework is fine and the rendering strategy is not.

No AI Overviews measurement exists here, and no proxy for one

Lantad has a second product surface that queries answer engines, and it would be easy to imply that it covers this ground. It does not, and the registry is again the proof.

The answer-engine registry lists five ids: workers-ai, gemini, perplexity, openai and anthropic. The Google row among them is labelled "Gemini with Google Search grounding". That is the Gemini API called with its Search grounding tool: a live-retrieval model call that returns citations. It is a different product from an AI Overview, served by different machinery to a different audience, and treating one as a proxy for the other would be the exact kind of quiet substitution this blog exists to avoid. There is no results-page scrape in this repository, no AI Overviews parser, and no stored observation of an AI Overview. Nothing in what AI says about you reports an AI Overview, because nothing can.

The naming convention in that registry deserves a sentence of its own, because it is the same discipline applied one layer up. Every engine is labelled by the model and the retrieval mode actually called, never by the consumer product a reader might have in mind. The OpenAI row says GPT-4o mini with web search rather than ChatGPT, and the Anthropic row says Claude Haiku 4.5 with web search. A report that said ChatGPT would be claiming to have observed a product we never called. The Google row obeys the same rule, and obeying it is precisely what makes the AI Overviews gap visible instead of papered over with a plausible label.

That is a genuine capability gap against tools built for a different job. Our comparison notes, re-verified against the vendors' own sites on 24 July 2026, record that Profound's site names Google AI Overviews among the engines it lists, and that the Otterly.AI features page names Google AI Overviews and Google AI Mode. What is checkable from here is that they publish the coverage and we have none of it. How well either measures that surface, how often, or from which country is not something we have verified, so we do not characterise it.

The same discipline applies when a scan itself fails. On 15 July 2026 the identified crawler asked allbirds.com for its homepage and received HTTP 429. That capture carries a null composite, a null grade and a state of access_fatal: the report says what happened and refuses to grade the page. A page behind a login gets the same treatment, with a stored verdict that reads "This page sits behind a login or paywall, so it is not publicly scannable. No letter grade: a grade here would be a guess." That rule is set out in why we withhold a grade, and it is this post's rule applied to a different input. If we will not print a grade we could not measure, we cannot print an AI Overviews verdict we did not observe.

It is also why the ChatGPT reference page and the Perplexity reference page can be more concrete than the Google one. Those surfaces have named, probeable crawlers with published user agents, so we can send a real request carrying each user agent and report what came back. Even that is a smaller claim than it sounds, because what we report is a response to a string we chose to send rather than proof of who anybody is, which is the argument of a user agent is a claim, not an identity taken from the other end. Small claims are the only kind the evidence in this category currently supports, and the definition of the field itself is younger than most of the tooling sold into it.