BlogFindings
1,395 of 2,729 text blocks were navigation, not main content
Counted on 25 August 2026 from five real pages this scanner captured and stored on 15 July 2026, the crawler view holds 16,636 visible words and 2,729 text blocks. 1,395 of those blocks sit inside nav, header, footer or aside, and between them they carry 2,920 words. More than half the blocks hold under a fifth of the text.
The five pages are the golden fixtures this scanner captured on 15 July 2026 and has stored since: the English Wikipedia article on web crawlers, the MDN HTML reference, astro.build, webflow.com and the gymshark.com homepage. Two of the figures below need no new instrument at all. Each stored expectation already carries rawVisibleTokens and rawMainTokens in its evidence block, so the headline counts are read out of files the golden fixture tests check on every push. The rest were produced by running the shipped extractor over the same stored captures on 25 August 2026, and they reproduce the stored numbers exactly.
In short
- Across five real pages captured on 15 July 2026, the stored golden expectations record 16,636 crawler-visible words, of which 13,615 are classified main content and 3,021 sit outside it.
- 1,395 of the 2,729 text blocks on those five pages are classified boilerplate, so 51.12 percent of the blocks carry 17.55 percent of the words: a boilerplate block averages 2.1 words against 10.4 for a main block.
- The share of words outside main content ranges from 4.99 percent on the English Wikipedia article to 38.18 percent on the MDN HTML reference, measured on the same day under the same rule.
- Lantad computes prose parity as containment of the rendered main text inside all raw visible text, so boilerplate sits in the corpus that is searched and is absent from the text that has to be found.
- Nothing here observes an AI crawler. These are counts from five stored HTML files, and no claim is made about how any crawler weights navigation text.
How much of a page is main content?
Across the five captures the crawler view holds 16,636 visible words. 13,615 of them are classified main content and 3,021 are not, which is 18.16 percent of the text on the page falling outside the corpus that the report treats as the document. The split is not a new measurement layered on top of the stored fixtures. It is the same pair of numbers those fixtures already publish: every expected.json in core/fixtures carries rawVisibleTokens and rawMainTokens, and summing the five gives 16,636 and 13,615 without opening a single HTML file.
The per-page spread is the part worth sitting with, because it is wide enough that an average over five pages describes none of them. The English Wikipedia article on web crawlers puts 4.99 percent of its words outside main content. The MDN HTML reference puts 38.18 percent outside. The two captures are timestamped 28 seconds apart, they were parsed by the same code and judged by the same four-line rule, and they differ by a factor of almost eight. A page that is mostly a long article and a page that is a short reference wrapped in a large navigation sidebar are not the same shape, and no single ratio is going to describe both.
Of the 3,021 words outside main content, 2,920 are boilerplate, meaning they sit inside nav, header, footer or aside. The remaining 101 are what the extractor calls neutral: visible text that is in none of those containers and also in no heading, paragraph or list item. That second bucket is the one we expected to be a problem and it is not. Copy dropped into a bare div with no semantic wrapper is the failure mode that client-rendered marketing pages are supposed to be full of, and across 16,636 words there are 101 of them, six tenths of one percent. On these five pages, almost everything a reader can see is inside an element that says what it is. That is an inconvenient result for a certain kind of advice and it is what the count says.
One caveat belongs here rather than at the end. The gymshark.com raw capture is exactly 3,145,728 bytes, which is the cap set in core/src/config.ts, and its stored confidence notes record that the response exceeded 3 MB and was truncated with parity scored on the captured portion. Its 2,353 words are what the captured portion holds, so every gymshark figure in this post is a floor rather than a total. The reasons a cap like that exists, and what the one search vendor that publishes a size limit says about the boundary, are set out in what Googlebot documents about reading the first two megabytes.
| Page | Visible words | Main content | Boilerplate | Neutral | Outside main |
|---|---|---|---|---|---|
| en.wikipedia.org | 7,974 | 7,576 | 387 | 11 | 4.99% |
| developer.mozilla.org | 1,794 | 1,109 | 685 | 0 | 38.18% |
| astro.build | 956 | 784 | 169 | 3 | 17.99% |
| webflow.com | 3,559 | 2,383 | 1,161 | 15 | 33.04% |
| gymshark.com | 2,353 | 1,763 | 518 | 72 | 25.07% |
| All five | 16,636 | 13,615 | 2,920 | 101 | 18.16% |
What the extractor calls boilerplate
The classification is worth stating precisely, because the word boilerplate carries a judgement the code does not make. Nothing in core/src/extract.ts decides that a run of text is unimportant. It decides where the text sits. Anything inside script, style, noscript, template, svg or iframe is dropped before classification begins, which is why script payloads never reach the text corpus and are routed instead to the JSON harvesting path described in our note on text that scored the same as no text at all. Everything that survives is then sorted into three kinds.
Main content is anything inside a main or article element, plus any h1 to h6, p or li anywhere in the document. Boilerplate is anything inside nav, header, footer or aside. Neutral is everything else, which in practice is text sitting loose in a div or in the body. When a block qualifies as both, boilerplate wins unconditionally, and that precedence is not a detail: it is the same rule that turns a real h1 wrapped in a header into a reported count of zero, which we took apart in an h1 inside a header counted as none. That post was about heading counts. This one is about words, and the two go different ways, which is the reason to count both.
Read as a description of where text sits, the rule is accurate on all five pages. Read as a judgement about value, it would be wrong in at least one direction on every one of them. A footer holding a company address, a jurisdiction and a registration number is boilerplate by position and is exactly the sort of machine-readable identity claim that entity confidence exists to look for, and the same footer is where a good deal of a page's structured data tends to live. A nav element holding a reference sidebar of 584 words of links, which is most of what MDN contributes to this count, is boilerplate by position and is also the page's table of contents. Neither is noise, and the rule does not say they are. It says they are not the document.
There is a second thing the count settles. Boilerplate is not evenly distributed across container types either. On MDN and on the Wikipedia article, the largest single source of boilerplate words is the li element, at 584 and 332 words respectively: navigation lists, which is what a sidebar and a contents rail are made of. On webflow.com the largest source is the p element at 587 words, most of it sitting inside an aside rather than a nav, which is to say real sentences rather than labels. Those are different documents doing different things with the same four container tags, and a rule that treats an aside paragraph and a nav list item identically is going to be right about one of them more often than the other.
-
main13,615 words in 1,313 blocks Inside main or article, or in any h1 to h6, p or li anywhere in the document. This is the corpus prose parity is computed over. -
boilerplate2,920 words in 1,395 blocks Inside nav, header, footer or aside. Boilerplate wins over main when a block qualifies as both, which is unconditional. -
neutral101 words in 21 blocks Visible text in none of those containers and in no semantic block element. Six tenths of one percent of the text on these five pages. -
droppednot counted at all script, style, noscript, template, svg and iframe subtrees are removed before classification. Script contents go to JSON harvesting instead.
Parity looks for main text inside all text
The reason the split matters more than a tidy statistic is that the two sides of the parity calculation do not use the same corpus. Prose parity is 50 of the 100 composite points, written as a weight of 0.5 in SCORE_WEIGHTS in core/src/config.ts, which makes it worth as much as access, structure and schema together. It is computed in core/src/verdict.ts as the containment of one token list inside another, and the two lists are built by different rules.
The list being searched for is the rendered main text: parityCorpus applied to the browser view, which is main content only. The list being searched is every visible token in the raw view, boilerplate included. So the 3,021 words outside main content are in the haystack and not in the needle. A crawler that fetches the page gets them; the score never has to find them.
That asymmetry is deliberate and the code says why in its own comment. Shared chrome appears in both the raw and the rendered view by construction, since it is the same template on both sides. Folding it into the corpus that must be matched would let a repeated navigation menu carry the parity score and mask genuinely invisible main content on a page whose real text arrives only after JavaScript runs. A site with a large footer and no server-rendered article would score well on a measurement that counted the footer twice. Excluding boilerplate from the needle is what stops that, and it is the right call for the failure mode the score exists to catch, which is the one described in what a crawler meets on a real storefront.
The cost of the call is worth naming anyway. Parity is a statement about the main content of the page and not about the page, and on the MDN capture that is a statement about 61.82 percent of the words. There is one guard against the pathological case: parityCorpus falls back to all non-boilerplate text when the main corpus comes in under NEAR_EMPTY_TOKENS, set to 30 in core/src/config.ts, so a table-and-bare-div layout with no semantic containers is not scored against an empty needle and handed a vacuous zero. On these five pages the fallback never fired. The smallest main corpus in the set is astro.build at 784 words, twenty-six times the threshold, so every parity figure quoted here was computed the ordinary way. When any of this changes a stored score, it gets re-baselined in the open for the reasons in why we withhold a grade.
Flow: Raw HTTP response to All visible text 16,636 words; Rendered DOM to Rendered main text only; All visible text 16,636 words (searched) to corpusContainment; Rendered main text only (searched for) to corpusContainment; corpusContainment to Parity, 50 of 100 points.
Half the blocks, a fifth of the words
The block counts point somewhere the word counts do not. The extractor emitted 2,729 text blocks across the five raw captures, and 1,395 of them, 51.12 percent, are boilerplate. Those 1,395 blocks hold 2,920 words. The 1,313 main blocks hold 13,615. A boilerplate block on these pages averages 2.1 words and a main block averages 10.4, which is the difference between a menu label and a sentence.
Put plainly, most of the text nodes on a real page are navigation, and navigation is made of two-word fragments. That is not a complaint about the pages. It is what a site menu, a language switcher, a breadcrumb rail and a footer link column are: many short labels. But it does mean that any measurement of a page that counts elements rather than words will report something close to a fifty-fifty split between content and chrome, and any measurement that counts words will report something closer to four to one. Both are true and they are answers to different questions.
The extreme case in the set is MDN, and it is worth reading as a shape rather than a fault. 403 of its 479 text blocks are boilerplate, which is 84.13 percent of the blocks holding 38.18 percent of the words. The page is a reference entry with a very large navigation sidebar, and that sidebar is genuinely useful to a human reading the HTML reference. The Wikipedia article is the mirror image: 154 of 488 blocks are boilerplate, 31.56 percent of the blocks holding 4.99 percent of the words, because the article body is one long run of prose and the chrome around it is comparatively small. The same rule, applied to two well-built documents, produces two very different pictures of the same site quality.
None of this counts a link, which is a separate question we have already put numbers on. The anchor text side of the same five captures is in 85 links carried no anchor text, the table markup in three table elements and all three were navigation, the head of each document in 85 meta elements and two addressed a crawler, and the difference JavaScript made to the link graph in JavaScript added no new crawl paths. Those posts and this one are five files being asked five questions, and the reason to keep asking is that the answers have not been consistent with each other.
Counted as text blocks
- 2,729 blocks in total
- 1,395 boilerplate, 51.12 percent
- 1,313 main content
- 21 neutral
- MDN: 403 of 479 blocks boilerplate
Counted as words
- 16,636 visible words in total
- 2,920 boilerplate, 17.55 percent
- 13,615 main content
- 101 neutral
- MDN: 685 of 1,794 words boilerplate
What the specifications say about main content
The positional rule is not something this scanner invented. It is a reading of what the elements are for, and the documentation for the main element says so directly. MDN's reference for the main element, last modified 24 April 2026, states in its usage notes that the content of a main element should be unique to the document, and that content repeated across a set of documents or document sections, such as sidebars, navigation links, copyright information, site logos and search forms, should not be included unless the search form is the main function of the page. That is the same distinction the extractor draws, arrived at from the other direction.
Worth being precise about what that does and does not authorise. It is guidance about how to author a document, not a statement about how any crawler reads one. The HTML specification's own definition of the element is one sentence, that the main element represents the dominant contents of the document, and it attaches no retrieval behaviour to that. An author who follows the MDN note produces a document where the boundary this scanner reads is where they put it. An author who does not still produces valid HTML, and the scanner still has to grade it.
There is exactly one first-party mechanism in this area that a search engine documents as something it acts on, and it is narrower than people expect. Google's robots meta tag and inline directives documentation, carrying Last updated 2026-03-24 UTC, describes the data-nosnippet attribute, which can be set on span, div and section elements and prevents the text inside them being used in a search result snippet. It is a boolean attribute, so any value on it is ignored. Two things follow. It is opt-in, so a page that marks nothing gets no exclusion. And it governs snippets rather than reading: it is a statement about what may be shown, not about what is parsed or indexed.
So the honest position on the main content boundary is that there is a standards-backed authoring convention, a Google-specific snippet control that does something else, and no published statement from any AI crawler vendor about how it separates a document from its navigation. The crawler documentation from the three vendors whose tokens this scanner probes is thin on parsing behaviour generally, which is the pattern behind our note on how AI crawler documentation names no file type at all. If you want to know how an AI crawler treats your footer, nobody has written it down, and what we can say about a specific engine is limited to observable behaviour of the sort collected in our notes on getting cited by ChatGPT.
| Source | What it states | Date on the page | Binds a crawler? |
|---|---|---|---|
| MDN, the main element | Content repeated across documents, such as sidebars, navigation links, copyright information, site logos and search forms, should not be in main | Last modified 24 April 2026 | No, authoring guidance |
| HTML specification, the main element | The main element represents the dominant contents of the document | Living standard, read 25 August 2026 | No retrieval behaviour attached |
| Google, robots meta tag and inline directives | data-nosnippet on span, div and section keeps the enclosed text out of a search result snippet, as a boolean attribute | Last updated 2026-03-24 UTC | Snippets only, opt-in |
| OpenAI, Anthropic and Perplexity crawler documentation | No published statement on how a document is separated from its navigation | Read 25 August 2026 | Nothing to bind |
What to check on your own page
The check is cheap and it needs no scanner. Fetch your own page with curl so you get the response a crawler gets rather than the one your browser assembles, and look for the boundary. Count how many words sit inside your main element and how many sit outside it, and if the answer is that most of your copy is in a footer or a sidebar, you have learned something before any tool has an opinion. The same fetch answers a second question at no extra cost, which is whether your text is there at all: if the response is mostly script, the problem is not the boundary, and the per-stack fixes for that are in the guides for React and Shopify. You can also run the fetch through what GPTBot sees if you would rather read the crawler view than build it.
Then check the three specific things this count keeps turning up. Is your h1 inside a header element, which will cost you a heading on more than one measurement. Is there real, unique copy inside a footer or an aside, which is fine for a reader and invisible to any tool drawing the boundary where the specification suggests. And is there a main element at all: all five pages here have exactly one, which is the easy case, and a page with none is the case where a scanner has to guess.
Two limits on everything above, stated plainly because the numbers are precise enough to be mistaken for more than they are. Five pages is not a sample of anything. They were captured on a single day, they are two documentation pages and three marketing or commerce homepages, and a page changes between deploys. Where we have tried to say something about a population rather than a handful of files, it is in the crawlability study and it is reported with its sampling stated. The counts are exact for these five files and they support no estimate of the web. Second and more important, nothing here observes a crawler. This post reports how one extractor classified 16,636 words in five stored files, and it contains no evidence that GPTBot, ClaudeBot or any other agent treats a nav element differently from a paragraph. Our published scoring methodology says which parts of a grade are measured and which are inferred, and this boundary is on the inferred side.
What the count does change is what we will claim. A parity figure is a statement about the main content of a page, and on the five pages we have, that is between 61.82 and 95.01 percent of the words in the response. That is a caveat the report should carry in the customer's own copy rather than in a blog post, in the same way the truncation caveat already does, and it is the kind of correction that comes out of counting something rather than assuming it. Anyone building for generative engine optimisation on the assumption that a tool reads the whole page should check where the tool draws the line, and that includes this one.
- Exactly one main element in the served HTML All five captures have one. A page with none is the case where a scanner has to guess where the document starts, and where the near-empty fallback exists.
- The h1 is not inside a header element Boilerplate wins unconditionally, so an h1 in a header is discarded from the heading list and from h1Count on the same pass. Two of the five captures record an h1Count of 0 for this reason.
- Unique copy is not stranded in a footer or aside 685 of MDN's 1,794 words sit in its sidebar. Useful to a reader, outside the corpus that parity is computed over.
- Body copy is inside p, li or a heading rather than a bare div Only 101 of 16,636 words across the five pages were loose text. This is the failure everyone warns about and it barely appeared.
- The response is not truncated before the end The gymshark.com capture hit the 3 MiB cap exactly, so every figure for that page is a floor rather than a total.
Lantad
Published .
An AI crawler that fetches your page receives every word in the response. A scanner that grades the page has to decide which of those words are the page, and Lantad makes that decision with a positional rule in core/src/extract.ts. Text inside main or article is main content, as is any heading, paragraph or list item anywhere in the document. Text inside nav, header, footer or aside is boilerplate, and the comment at the top of the file states the precedence in three words: boilerplate wins. That rule is a few lines of set membership, and it decides what the heaviest score on the report is computed over. It had never been counted, and a rule that has never been counted is an assumption.
Common questions
What counts as main content in a Lantad scan?
Text inside a main or article element, plus any h1 to h6, p or li element anywhere in the document. Text inside nav, header, footer or aside is classified boilerplate instead, and that classification wins when a block qualifies as both. Everything else visible is neutral. The rule is positional and lives in core/src/extract.ts.
Does text in a footer or navigation menu count toward my prose parity score?
Not on the side that has to be matched. Parity is the containment of the rendered main text inside all visible text from the crawler view, so boilerplate is in the corpus that gets searched and absent from the text being searched for. On the five pages measured here that is 3,021 of 16,636 words, or 18.16 percent, which the score never has to find.
How much of a typical page is navigation rather than content?
There is no typical answer in this data and the spread is the finding. Across five real pages captured on 15 July 2026 the share of visible words outside main content ran from 4.99 percent on the English Wikipedia article to 38.18 percent on the MDN HTML reference. Counted as text blocks rather than words the picture changes again: 51.12 percent of all blocks were boilerplate, because navigation is made of two-word labels.
Do AI crawlers ignore navigation and footer text?
No published documentation from OpenAI, Anthropic or Perplexity says either way, and nothing in this post measures it. The only first-party control in this area that a search engine documents as acting on is Google's data-nosnippet attribute, and its documentation, Last updated 2026-03-24 UTC, describes it as keeping text out of a search result snippet rather than out of parsing or indexing.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
Prose Parity
Prose Parity is Lantad's metric for the share of a page's rendered, human-visible text that is present in the raw HTML an AI crawler receives, where 100% means a crawler sees everything a human visitor sees.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
An h1 inside a header element counted as no h1 at all
The raw HTML of five pages this scanner captured on 15 July 2026 holds 245 heading elements and our extractor recorded 166, because it discards every heading inside nav, header, footer or aside. On two of those pages the rule turned a real h1 into a reported count of zero.
Three table elements on five real pages, and all three were navigation
Lantad's own component library states that a table is the shape an AI crawler extracts most reliably, so the claim was checked against the pages this scanner has actually stored. Counted with an HTML parser on 21 August 2026 over five real pages captured on 15 July 2026, the whole set holds three table elements, all three on one page, all three navigation boxes, and none of the five pages carries a single caption element.
85 links carried no anchor text, and 34 were named only by aria-label
Counted with an HTML parser over five real pages this scanner captured and stored on 15 July 2026, the rendered DOM holds 1,951 links with an href and 85 of them carry no text between the tags. Google's link documentation names two fallbacks for an empty link, the title attribute and a nested image's alt text. Thirty four of the 85 carry neither, and 23 carry none of the three.
