BlogFindings

JavaScript added no new crawl paths, and rendering dropped nine locale links

Counted with an HTML parser over five real pages this scanner captured and stored on 15 July 2026, the raw HTTP response held 991 distinct same-host URLs and the rendered DOM held 982. Three URLs existed only after JavaScript ran, and every one of the three was a rewrite of a link already present. Nine of the twelve that disappeared were one page's translations of itself.

17 August 2026 17 min read Lantad

So the question is worth asking separately from parity: on real pages, does running JavaScript reveal onward links that a plain fetch does not have? This post counts that on the five real captures already stored in this repository, using the same stored HTML that produced their published grades, and reports what the count says including the part that argues against the premise. The short answer is that on these five pages, running JavaScript revealed nothing new, and on three of them it took links away.

In short

Across the five real captures stored in core/fixtures/ and taken on 15 July 2026, the raw HTTP response carried 1,957 anchor elements and the rendered DOM carried 1,951, resolving to 991 and 982 distinct same-host URLs.
Rendering introduced three same-host URLs absent from the raw HTML, all three on the English Wikipedia article, and each was a rewrite of a link already present rather than a route to content a plain fetch could not reach.
Rendering removed twelve same-host URLs, of which nine were developer.mozilla.org's links to the German, Spanish, French, Japanese, Korean, Brazilian Portuguese, Russian, Simplified Chinese and Traditional Chinese versions of the same page.
Of the 1,957 anchor elements in the raw HTML, 85 held no text at all and 204 pointed only at a fragment of the page they sat on, while none of the 1,957 was missing an href attribute.
Lantad's structure checks in core/src/verdict.ts are seven booleans and not one of them counts a link, so no figure in this post reaches the grade this scanner publishes.

Captured page	Raw anchors	Rendered anchors	Raw URLs	Rendered URLs	Added	Lost
astro.build	133	133	52	52	0	0
gymshark.com	334	334	184	184	0	0
developer.mozilla.org/en-US/docs/Web/HTML	463	452	353	344	0	9
webflow.com	258	259	89	88	0	1
en.wikipedia.org/wiki/Web_crawler	769	773	313	314	3	2
Total	1,957	1,951	991	982	3	12

Anchor elements and distinct same-host URLs in the raw HTTP response against the rendered DOM, counted with htmlparser2 over the five real captures in core/fixtures/, all stored 15 July 2026. Added and lost are distinct same-host URLs present in one document and not the other.

What was counted, and how a link was defined

Six of the golden fixtures in core/fixtures/ are captures of real sites rather than synthetic test documents. One of them, allbirds.com, answered 429 on the day of capture and was never rendered, which leaves five pages that returned 200 and hold both a raw.html and a rendered.html: the astro.build homepage, the gymshark.com homepage, the developer.mozilla.org landing page for HTML, the webflow.com homepage and the English Wikipedia article on web crawlers. All five were captured on 15 July 2026 within about four minutes of each other. These are the same five pages a count of alt text on 466 captured images used, for the same reason: they are the only real HTML this repository stores twice, once as a plain client received it and once as a browser finished with it.

The counting used htmlparser2, which is the parser core already depends on, walking each document and collecting every element whose tag name is a. Nested anchors were counted once, at the outermost element, because that is the link a client would follow. Each href was then resolved against the capture's final URL under the reference resolution rules in RFC 3986, which is what makes a relative path comparable with an absolute one, and the fragment was stripped so that two links to the same document differing only by anchor collapse into one URL rather than counting twice.

Four kinds of href were excluded before comparison, on the grounds that none of them is a route to another document: an empty or whitespace-only value, a javascript: URL, a mailto: or tel: URL, and a value consisting only of a fragment. What survives is a set of absolute URLs. The set was then filtered to the capture's own host, because a same-host URL is the only kind whose discovery matters for whether the rest of your site gets crawled, and because counting outbound links would mix a question about your architecture with a question about your editorial choices.

The comparison itself is a set difference in both directions, which is the part worth being precise about. Added means a distinct same-host URL present in the rendered DOM and absent from the raw response. Lost means the reverse. Neither figure is a count of anchor elements, and the two columns move independently of the element counts in the hero table, which is why webflow.com can gain an anchor element and lose a URL at the same time. The method here is narrower than the one behind the composite score described at our methodology, which compares visible prose rather than link graphs.

Flow: Stored HTML document to Parse with htmlparser2; Parse with htmlparser2 to Outermost a elements; Outermost a elements to Resolve href per RFC 3986; Resolve href per RFC 3986 to Drop empty, javascript, mailto, tel, fragment; Drop empty, javascript, mailto, tel, fragment to Keep same host only; Keep same host only (compare both ways) to Set of distinct URLs.

The counting pipeline applied identically to raw.html and rendered.html for each capture. A description of the method used in this post, not of the production scoring pipeline.

Did JavaScript reveal a crawl path a plain fetch would miss?

On four of the five pages, no. The raw response and the rendered DOM resolved to identical sets of same-host URLs on astro.build and gymshark.com, and on developer.mozilla.org and webflow.com the rendered set was a strict subset of the raw one. Only the Wikipedia article produced any URL that rendering added, and it produced three.

Those three are worth printing in full rather than summarising, because the summary would be misleading. Two of them are the account creation and login URLs with a query parameter appended naming an interface experiment, against the same two URLs without that parameter present in the raw HTML and absent after rendering. That is one link rewritten, counted twice by a set difference that treats a changed query string as a different URL, which it is. The third is the article's own address with action=edit appended. All three are interface affordances on a page whose article text and whose 313 other same-host links were already in the raw response.

So across 991 distinct same-host URLs on five real pages, the number that rendering revealed as a route to content a plain HTTP client could not otherwise reach is zero. That is a null result and it cuts against a claim this site's own product line rests near: that a crawler which does not execute JavaScript is missing your site. On these five pages, for links specifically, it was not. The prose numbers say something different and are not in conflict, because they measure a different thing. Parity on these captures runs 1.0 on astro.build, 0.994 on Wikipedia, 0.988 on developer.mozilla.org, 0.980 on webflow.com and 0.958 on gymshark.com, so text was very nearly all present too. What this repository holds is five pages that are already well built, not a sample of the web.

The honest boundary is therefore narrow and specific. A page that renders its navigation entirely on the client would fail this count badly, and none of these five does. Frameworks make that outcome a configuration rather than a certainty, which is what the stack guides for Next.js and React are about. What this measurement establishes is that being built on a client-side framework does not by itself hide your links: webflow.com and gymshark.com both ship substantial JavaScript and both had their entire same-host link set in the first response. A finding about deep research agents reported here separately, that cited links worked above 94 percent while facts did not, points the same way from the other end of the pipeline.

URL added after rendering	What it is	Already reachable in raw HTML
/w/index.php?title=Special%3ACreateAccount&returnto=Web+crawler&experiments%5B0%5D=...	Account creation link with an interface experiment parameter appended	Yes, without the parameter
/w/index.php?title=Special%3AUserLogin&returnto=Web+crawler&experiments%5B0%5D=...	Login link with the same parameter appended	Yes, without the parameter
/wiki/Web_crawler?action=edit	The same article's edit interface	The article itself, yes

The three same-host URLs present in the rendered DOM of en.wikipedia.org/wiki/Web_crawler and absent from its raw response, captured 15 July 2026. No other capture produced any.

Rendering removed twelve links, and nine were translations of the same page

The interesting direction turned out to be the other one. Twelve distinct same-host URLs were present in a raw response and gone from the rendered DOM, and they are not randomly distributed. Nine of the twelve are on developer.mozilla.org, and all nine are the same page in another language: the German, Spanish, French, Japanese, Korean, Brazilian Portuguese, Russian, Simplified Chinese and Traditional Chinese versions of the HTML landing page. The raw HTML advertises all nine. After the page finished rendering, none of the nine anchors was in the DOM.

That is a real asymmetry with a practical reading, and it needs stating carefully because the mechanism is not measured here. Something in the page's own script collapsed a language list that the server had sent as markup, which is ordinary behaviour for a language picker that becomes a menu. The consequence is that on this page a client which runs JavaScript sees fewer localised alternates than a client which does not, and the direction of that surprise is the opposite of the one the rendering argument predicts. It also sits next to a separate finding on this blog that Googlebot sets no Accept-Language header and that no AI crawler's documentation describes any locale behaviour at all, which together mean the discovery of a translated page depends more on the markup than on anything negotiated at request time.

The remaining three losses are smaller and each explains itself. webflow.com's raw HTML links to its own dashboard and the rendered DOM does not, which is what a signed-out session looks like once the client decides what to show: the same class of state-dependent difference behind consent gates never opening for a stateless crawler. Wikipedia's two are the unparameterised login and account creation URLs, the other side of the rewrite described above, so on that page the added and lost columns are largely the same two links counted at both ends.

None of this means rendering destroys crawl paths as a general rule, and twelve URLs out of 991 is a small effect. What it does mean is that the raw response and the rendered DOM are two different documents in both directions, not one document plus additions. Treating the rendered DOM as a superset is a modelling assumption, and on three of these five pages it was wrong. That is the same lesson as reading one page's JSON-LD two ways, where the choice of which document to parse changed the answer completely.

Capture	URLs lost	What they were
developer.mozilla.org	9	The de, es, fr, ja, ko, pt-BR, ru, zh-CN and zh-TW versions of the same HTML landing page
en.wikipedia.org	2	Login and account creation, rewritten with an experiment parameter rather than deleted
webflow.com	1	The signed-in dashboard link, present in the server response and not in the rendered page

The twelve distinct same-host URLs present in a raw response and absent from the rendered DOM of the same capture, 15 July 2026. Nine of the twelve are one page's translations of itself.

85 anchors carried no text, and 204 pointed only at a fragment

A crawlable link is not only an href. Google's link best practices documentation, which carries Last updated 2025-12-10 UTC, states that Google can parse links formatted as a elements with href attributes, and separately defines anchor text as the visible text of a link and asks that it be descriptive, reasonably concise and relevant to the page it is on and the page it links to. Both halves are checkable in stored HTML, so both were counted.

The href half came out clean. Of the 1,957 anchor elements in the raw responses, zero were missing an href attribute entirely. That is worth noting because an anchor without href is not a degraded link, it is not a link: MDN's reference for the a element records the implicit ARIA role as link when href is present and generic when it is not. On these five pages nobody had shipped that particular mistake.

The text half came out less clean. 85 of the 1,957 anchor elements, which is 4.34 percent, contained no text at all once whitespace was normalised, and the distribution is lopsided: 33 on gymshark.com, 32 on webflow.com, 14 on Wikipedia, 6 on developer.mozilla.org and none on astro.build. These are overwhelmingly icon links, logos and image tiles, which is why the figure sits alongside the alt attribute count from the same captures rather than replacing it. An icon link with an empty anchor and an image with no alt attribute is a route with no label attached anywhere a text extractor can reach. The gymshark.com capture is described in more detail in what a crawler meets on a real storefront.

The fragment figure is larger and mostly benign. 204 of the 1,957 anchors, 10.42 percent, had an href consisting only of a fragment, and 148 of those 204 are on the single Wikipedia article, where they are the citation and section links that make a long reference page navigable. Those were excluded from the URL sets because they are routes within a document rather than to one, and they are counted here only so the two numbers add up for anyone repeating the measurement. What none of this tests is whether an AI crawler uses anchor text as a signal at all. Google documents that it does for Search. No AI crawler vendor documentation read for this blog says anything on the subject, and the honest position on an undocumented behaviour is the one taken at how AI crawlers are defined: describe what is observable and refuse to model what is not.

gymshark.com 33 anchors with no text Of 334 anchor elements in the raw response
webflow.com 32 anchors with no text Of 258 anchor elements
en.wikipedia.org 14 anchors with no text Of 769 anchor elements, the largest raw count of the five
developer.mozilla.org 6 anchors with no text Of 463 anchor elements
astro.build 0 anchors with no text Of 133 anchor elements, none empty

Anchor elements in the raw HTTP response with no text content after whitespace normalisation, by capture, 15 July 2026. Counted over every a element, including those excluded from the URL sets.

What this count does not establish

Five pages on one day is a description of a small corner of the web, not a survey, and the five were chosen long before this question was asked, as fixtures for a scoring test suite. Nothing here supports a percentage about sites in general, and the figures should be read as an existence check on a mechanism rather than as a rate.

One of the five carries a specific limit that has to be declared. The gymshark.com raw capture is exactly 3,145,728 bytes, and its meta.json records htmlTruncated as true, because maxHtmlBytes in core/src/config.ts caps a raw capture at 3 MiB and the page exceeded it. That cap is a setting somebody chose, not a measurement, and its consequence for this post is direct: the 184 same-host URLs counted in that page's raw response are a floor, and any link past the cap is invisible to this count in the raw column while still being present in the rendered one. That the two columns nonetheless matched exactly at 184 is evidence the truncation fell after the navigation, not evidence there was nothing beyond it. Google's own limit on the same problem, and why a cap like this exists at all, is covered in Googlebot reads the first two megabytes.

Three further boundaries matter more than the sample size. This count reads stored HTML, so it says what was in each document, not whether any crawler followed anything: whether a request is served at all is decided by the layer described in the two layers that decide if AI can read your site, and no figure here touches it. It covers one URL per site, all of them a homepage or a landing page, which is precisely where a site puts its navigation and therefore the most favourable page it has for this test. And it cannot see the failure mode most likely to matter, which is a listing that loads more items as you scroll: those routes exist in neither document, because the request that would produce them is never made, which is the subject of Google does not interact with your page. A rendering fetch that scrolls three viewports is still not a user who scrolls forty.

If you want the equivalent observation for your own URL rather than for these five, the raw side of it is what what GPTBot sees reports, and the aggregate picture this scanner has published from its own scans sits at the crawlability study and on the research page.

Sample Five pages, one day Captured 15 July 2026 as scoring fixtures, chosen before this question was asked. Supports no percentage about the web.
Truncation One capture hit the 3 MiB cap gymshark.com raw is exactly 3,145,728 bytes with htmlTruncated true, so its 184 URLs are a floor.
Page type Homepages and landing pages The most navigation-dense page a site has, and therefore the easiest test it could be given.
Following Not tested Stored HTML shows what a document contained. It cannot show whether any crawler requested what it pointed at.
Scroll-loaded routes Invisible to both columns A link that appears only after a fetch neither client made is in neither the raw nor the rendered document.

The declared limits of this count. Every one of these is a gap in the measurement rather than a finding about any site.

Why none of this reaches the score we publish

The inconvenient part sits in this repository rather than in anybody else's. The StructureChecks interface in core/src/verdict.ts is seven booleans: a title, a meta description, exactly one h1, a canonical, heading coverage, no oversized block, and llms.txt present. Not one of them counts a link, compares link sets between the raw and rendered documents, or looks at anchor text. Neither does the extractor, which walks elements for a small fixed list of attributes and does not collect href except on a link element carrying rel canonical. So every figure in this post was produced by a script written to ask this question, and none of it is in any grade this scanner has issued.

That is a gap worth naming plainly rather than presenting as a roadmap. It is also defensible for a reason worth stating: parity carries 50 of the 100 composite points, as much as access, structure and schema combined, and the evidence from these five captures is that link sets moved by 12 URLs in 991 while the thing parity measures is the thing that actually varies between a plain fetch and a browser. Adding a link parity check would have found nothing on four of these five pages. On the fifth it would have found nine translations, which is a real defect and a narrow one. A check that fires that rarely on real input is a candidate for a report section, not for a weight in a score, and inventing a weight for it would be exactly the move the audit of our own scale criticised.

There is a more general point here about what prose parity does and does not stand in for. Parity is a claim about text, and it has been treated on this site as a proxy for readability in general. This count is a small piece of evidence that the proxy does not extend automatically: on these pages text parity varied from 0.958 to 1.0 while link parity was near total, and the one page with the lowest text parity, gymshark.com, lost no links at all while the page with the second highest lost nine. Two signals that move independently are two measurements, and a scanner that reports one should not imply the other. The same argument applies to markup a text extractor drops, which is why shadow DOM text and the extractor exists as a separate finding.

For a reader trying to act on this, the practical version is short. Fetch your own key templates without JavaScript and check that your primary navigation, your pagination and your localised alternates are in the response, because those three are the routes that decide what gets crawled beyond the page you tested. That is a different check from confirming your text is there, it takes about the same effort, and on the evidence here it will usually pass. Publishing a measurement whose answer is usually "this is fine" is less satisfying than publishing a problem, and it is the reason this site will withhold a grade rather than manufacture one, and part of what AI visibility has to mean if the term is going to carry any weight in generative engine optimization.

titlePresent A title element exists with content.
metaDescriptionPresent A meta description is present.
exactlyOneH1 The document declares one top-level heading.
canonicalPresent A link element with rel canonical is present.
headingCoverage Headings cover the body rather than leaving long unheaded runs.
noOversizedBlock No single block dominates the extracted prose.
llmsTxtPresent An llms.txt file exists and parses, weighted lowest of the seven.
Any check counting a link None. Link sets, link parity and anchor text are absent from the interface, the extractor and the score.

The seven booleans in the StructureChecks interface in core/src/verdict.ts, read on 17 August 2026. Present means the check exists in the code, not that any page passes it.

Written by

Lantad

Published 17 August 2026.

The argument for making a page readable without JavaScript is usually made about text, and this site makes it that way too: fetch the URL once over plain HTTP, load it again in a browser, and compare the prose. That comparison has a second half almost nobody measures. A page is not only something to read, it is also a set of onward routes, and a crawler that cannot see a link cannot queue the page behind it. If hydration is what puts your navigation on the page, then the crawler that reads your prose fine still never learns that the rest of the site exists.

Common questions

Do AI crawlers see links that JavaScript adds to a page?

Nothing in this count answers that, because it reads stored HTML rather than crawler behaviour, and it found almost no such links to begin with. Across five real pages captured on 15 July 2026, three same-host URLs existed only in the rendered DOM and all three were rewrites of links already in the raw response. The crawler pages published by OpenAI, Anthropic and Perplexity do not state whether they execute JavaScript, so the honest answer for those crawlers is that it is undocumented.

Does a client-side framework hide your links from crawlers?

Not on the evidence here. The webflow.com and gymshark.com homepages both ship substantial JavaScript and both had their complete same-host link set in the first HTTP response, 89 and 184 distinct URLs respectively. A framework makes server-rendered navigation a configuration choice rather than a guarantee, so it is worth checking your own templates, but shipping JavaScript is not by itself evidence that your links are missing.

Can rendering a page remove links rather than add them?

Yes, and it did on three of these five captures. Twelve distinct same-host URLs were present in a raw response and absent from the rendered DOM of the same capture, of which nine were developer.mozilla.org's links to nine translations of the same page. Treating the rendered DOM as a superset of the raw response is a modelling assumption, and it was wrong on three of five pages here.

Does Lantad's AI Visibility Score check whether your links survive without JavaScript?

No. The StructureChecks interface in core/src/verdict.ts holds seven booleans covering title, meta description, a single h1, canonical, heading coverage, oversized blocks and llms.txt, and none of them counts a link or compares link sets between documents. Every figure in this post came from a script written for it, and none of it is reflected in any grade this scanner has published.

See what AI can read on your site

Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.

Scan your site free How we measure

In short

What was counted, and how a link was defined

Did JavaScript reveal a crawl path a plain fetch would miss?

Rendering removed twelve links, and nine were translations of the same page

85 anchors carried no text, and 204 pointed only at a fragment

What this count does not establish

Why none of this reaches the score we publish

Common questions

See what AI can read on your site

Related

Prose Parity

AI crawler

173 of 466 images carried alt text a crawler could read

Google does not interact with your page, so scroll-loaded content never loads

Googlebot sets no Accept-Language, so your site picks the locale for it

Two layers decide whether AI can read your site, and robots.txt is the weaker one