Text inside shadow DOM reaches the browser and not the extractor

Where a web component keeps its text

A custom element can hold its content in three places, and the difference between them is invisible on screen.

The first is the light DOM. The element's children are ordinary children of the document, exactly as a div's children are, and nothing is hidden from anything. A crawler that never runs a line of JavaScript reads that text straight out of the response body, and prose parity between what a crawler receives and what a browser paints is high for the same reason it is high on any server rendered page. If every component on your site worked this way, none of the rest of this post would apply to you.

The second is an imperatively attached shadow root. A script calls attachShadow on the host element and then fills the root it gets back. MDN's attachShadow reference describes the method as attaching a shadow DOM tree to the specified element and returning a reference to its ShadowRoot, and documents a mode option whose closed value means elements inside the shadow root cannot be accessed from JavaScript via the element's shadowRoot property, which is set to null. None of that content exists until the script has run, so this is the familiar client rendering failure wearing different clothes. It fails for an AI crawler that does not execute scripts in exactly the familiar way, and the remedy is the familiar one.

The third is declarative shadow DOM, and it is the interesting case, because the markup ships in the response. MDN's template reference documents a shadowrootmode attribute as a declarative version of the attachShadow method that accepts the same enumerated values, and states that if the template element carries shadowrootmode with a value of either open or closed, the HTML parser will immediately generate a shadow DOM, with the element replaced in the DOM by its content wrapped in a ShadowRoot attached to the parent element. The same page states, of the template element in general, that by default the element's content is not rendered.

Read those two statements together and the shape of the problem appears. The words are in the bytes. The words are inside a template element. A parser building a DOM turns that element into a shadow root and paints what is inside it. A parser that is only trying to collect text sees a template, and a template is the one element in HTML whose contents are conventionally not content at all.

What a text extractor does with a template element

Dropping template elements is the correct default, and it has been for as long as the element has existed. A template is the standard place to park markup that is not content: a row prototype a script will clone once per record, a dialog nobody has opened, a fragment a framework will stamp out later. Counting that text as page content would inflate every measurement taken over a page that uses one, and it would count the same sentence twice on any page that stamps the template into the document as well as shipping it. An extractor that kept template contents would be wrong far more often than it was right.

Lantad's extraction ruleset does the conventional thing. The rules are stated at the top of core/src/extract.ts and the drop list is a single set in that file: script, style, noscript, template, svg and iframe subtrees are dropped entirely, and every text measurement the product takes runs through the same sink. Script contents are routed to structured data and hydration payload harvesting rather than to text, which is why structured data can still be read from a page whose visible prose is empty. Template contents are routed nowhere. They are discarded before a token is counted.

The consequence follows mechanically rather than mysteriously. A page that uses declarative shadow DOM for its main content ships that content to every client, including one that runs no JavaScript at all, and Lantad's extraction does not count a word of it. The text is in the response we fetched and it is absent from the corpus we grade. Nothing in the pipeline is broken and no exception is thrown: a rule that is right for templates is wrong for one specific use of a template, which is the category of defect that no error message will ever raise and no test will catch unless someone thinks to write it.

This matters more than it would on a dimension worth a handful of points. Prose parity carries the largest single weight in how the score is built, because whether a crawler receives the words is the question the product exists to answer. An extraction that silently omits a component's text is not measuring a slightly smaller version of the page. It is measuring a different page, and every number computed downstream, the word counts, the headings coverage, the main content share, describes that different page with no marker to say so.

Why the browser view loses the same text a second time

The natural answer to all of this is that the browser view will catch it. Render the page in a real browser, serialize the result, compare the two corpora, and any text the crawler side missed shows up as a parity gap. That is exactly how the measurement is supposed to work, and on a shadow DOM page it does not, for a reason that is a documented default rather than an accident.

MDN's Element.getHTML() reference states that the method provides an options argument enabling the serialization of child nodes that are shadow roots, and that without arguments, child nodes that are shadow roots are not serialized and the method behaves in the same way as reading the value of Element.innerHTML. The serializableShadowRoots option that would include them carries a default value of false. The attachShadow reference documents the matching serializable option on the root itself, a boolean that when set to true indicates the shadow root may be serialized by getHTML with that parameter set, and whose default value is also false.

Two defaults of false, and the ordinary ways of pulling HTML back out of a live page inherit both. A rendered snapshot taken as markup therefore contains the host element and not the shadow content underneath it, whatever the visitor is looking at on screen at the time.

Now put the two halves together, because the combination is the part worth understanding. The crawler side loses the text because the extraction drops the template. The browser side loses it because the serialization omits the shadow root. Parity is a comparison of those two corpora against each other, so a page that keeps most of its content in shadow DOM can produce a parity reading that is not obviously wrong. It can look good. The defect is not a gap between the two views, which is the thing parity is built to detect; it is that both views are smaller than the page, and a ratio between two undercounts carries no signal that either of them was one.

That is the opposite failure from the one described in why we withhold a grade, where a fetch fails outright, the honest answer is to publish no composite at all, and the report says which measurement is missing. Here everything succeeds. A status code arrives, a document parses, a score computes and a grade prints. The number is arithmetically correct over a corpus that nobody chose and no field records.

What Lantad does about it, and in which renderer

There is a fallback for this, and it does not run everywhere.

The CLI renderer takes the serialized page, counts the tokens the extractor keeps, then runs a shadow piercing text walk over the live DOM and counts those tokens as well. When the walk sees far more text than the serialization did, it grades the walked text instead of the serialized document, and the report says so. The threshold is two exported constants in core/src/render-notes.ts: SHADOW_FALLBACK_RATIO is 3 and SHADOW_FALLBACK_MIN_TOKENS is 50, so a pierced walk has to see more than three times the serialized token count, and more than fifty tokens in absolute terms, before the fallback fires. Both numbers are settings somebody chose rather than findings: the ratio keeps ordinary pages, where the two counts sit within a small factor of each other, from tripping it, and the token floor stops a very short page tripping it on noise. When it does fire, the report carries a note stating that the rendered text came from a shadow piercing snapshot because the page keeps most of its content in shadow DOM, which many extractors miss.

The production renderer does not implement it. Lantad renders through the Cloudflare Browser Rendering API in production and through Playwright in the CLI and in development, both behind one interface, and only the second one pierces. core/src/renderparity.ts exists so that this is a declaration rather than a discovery: it lists every renderer behaviour that can change a verdict, names which renderers implement it, and requires a written reason whenever they differ, with a test that fails when a divergence is neither closed nor declared. The record for the shadow fallback states that it is not in the Worker renderer and is left open on purpose as of 2026-08-04, that exposure was measured rather than assumed across a random sample of 24 scanned corpus sites in which no page used declarative shadow DOM and none called attachShadow, and that porting it would change what production measures inside the window covered by the crawlability study.

Three things about that record deserve saying in plain words. Twenty four pages is a small sample and it establishes nothing about the web; it establishes that the sites this scanner has been pointed at did not use the feature. The direction of the error is the bad direction: without the fallback, a shadow heavy page measures lower than it should, which is a confident wrong low grade rather than an absent one, and a wrong low grade is the failure mode this product is least entitled to. And the calibration standard has the capability that production lacks, because the captures in core/fixtures/ are taken through the CLI path. Publishing that is the same commitment as naming the crawler our scanner does not model: a limitation someone can read is worth more than a number nobody can check.

What to check on your own site

Start with the bytes rather than with the browser, because the bytes are the thing every client agrees on. Fetch the page without a browser and search the response for the string shadowrootmode. A hit means the component content ships in the HTML, which is the good case for anything that does not run scripts, and it also means every extractor that drops template elements is failing to read it. Then search the same response for attachShadow. A hit there means content assembled by script, which is the ordinary version of the two layers that decide whether AI can read your site with a component shaped front end sitting on top. What a crawler receives for a URL shows you the response side without your having to set up a fetch of your own.

Then decide what you want a client that does not render to see, because the vendors differ and only one of them has told you. Google is explicit that it renders: Google's JavaScript SEO basics documentation, carrying Last updated 2026-03-04 UTC, states that Google supports web components and that when Google renders a page it flattens the shadow DOM and light DOM content, adds that this means Google can only see content that is visible in the rendered HTML, and recommends checking the rendered HTML in the Rich Results Test or the URL Inspection Tool. That is Google describing Google, on Google's own page, and it is worth reading there rather than in anybody's summary. It says nothing about a client that fetches your HTML and never opens a browser, and Lantad has not measured what any AI crawler does with a declarative shadow root. The archive Common Crawl publishes without running JavaScript is the standing reminder that such clients are not hypothetical.

The remedy is the one Google's own page demonstrates, which is to keep the content in the light DOM and project it into the component through a slot, rather than writing it inside the shadow root. Text in the light DOM is in the response, in every serialization and in every extractor, and putting it there costs a component nothing structurally. That is the same instruction as the per stack advice for React, Next.js and Shopify, stated for a different mechanism: the words a machine is meant to read belong in the markup the server sent, and every layer you put between the server and those words is a layer that can drop them.

Last, check what your measuring tools do before you trust a number one of them gives you, and that includes this one. Our crawler identifies itself and the developer documentation sets out what it fetches and what it does not. An AI visibility figure is only ever as good as the corpus it was computed over, and generative engine optimisation advice built on a corpus that quietly dropped your component text will point at the wrong problem with complete confidence. The captures in what a crawler meets on a real storefront are useful precisely because those pages kept their text where an extractor could find it, which is a property worth checking before you read anything into a score.