Internal links and AI crawlers: 84 of 1,091 home pages offered a crawler no path at all

What counts as an internal link an AI crawler can follow

A link is a thing a crawler can act on, not a thing a user can click. Those two sets overlap and they are not the same set, and the difference is where this measurement lives.

Google's Make your links crawlable documentation, carrying Last updated 2025-12-10 UTC when it was opened on 21 September 2026, is unusually direct about the rule. It says that generally Google can only crawl a link if it is an a HTML element with an href attribute, that most links in other formats will not be parsed and extracted by its crawlers, and that it cannot reliably extract URLs from a elements without an href or from other tags that perform as links because of script events. The same page says Google uses links for page relevancy and discovery, which is the reason any of this matters.

The rule follows from what the element is. The HTML anchor element, as MDN defines it, creates a hyperlink with its href attribute, to web pages, files, email addresses, locations in the same page, or anything else a URL can address. Three of those destinations are not another page on this host, which is why the count below excludes them one by one rather than counting anchors.

That is the rule this scan counted against. For each home page the raw HTML was searched for a elements carrying an href, each value was resolved against the final URL after redirects, and what remained after four exclusions was counted: off-host targets, fragments that point back into the same document, non-http schemes such as mailto and tel, and file extensions that are not pages. A link back to the home page itself was excluded too, since it offers no new path. What is left is the set of distinct further documents on the same host that a crawler reading these bytes could ask for next. The vocabulary underneath all of it is set out under AI crawler.

One limit belongs here rather than at the end, because it shapes every number that follows. No JavaScript was executed, and Google's page says plainly that links inserted dynamically by JavaScript are crawlable as long as they use the same markup. So a count of zero here means zero paths for a client that does not render, which is the condition several of the crawlers in the crawler directory publish for themselves and which this blog measured directly when JavaScript added no new crawl paths on a smaller set of captured pages. It is a real reading of the web for some crawlers and an incomplete one for others.

How many internal links does a home page give an AI crawler?

Across the 1,091 home pages that answered HTTP 200 with HTML, the median count is 48 distinct internal paths and the mean is 75. The gap between those two numbers is the shape of the distribution: a long tail of very large navigations pulls the average up, and the busiest single page in the corpus offers 1,573 distinct internal URLs.

Most of the corpus is comfortable. 535 of the 1,091 home pages offer 50 or more paths, and 255 offer 100 or more. For those sites the question this post asks is not interesting, because a crawler arriving at the home page has more to do than it will get through in one visit, and the useful question moves on to which of those paths are worth following. That is the territory of the sitemap declarations 839 of 1,016 robots.txt files carry, and of what happens when a crawler walks a site properly, which this blog measured when it followed 985 links on one site against 200,712 URLs declared in its sitemaps.

The thin end is where the finding is. 193 of the 1,091 home pages offer nine paths or fewer, and 84 of those offer none. A site with four internal links in its home page HTML is not necessarily a small site; it is a site where a crawler that does not render has four doors, and everything behind the other doors is reachable only from a sitemap, an external link, or a rendering pass the operator has no way to require. This is the same structural problem as prose parity and it is measured the same way: by asking what arrived, not what was authored.

One thing the count deliberately does not measure is quality. A home page offering 300 paths of which 290 are faceted product filters is worse served than one offering 30 substantive pages, and this scan cannot tell those apart. Nor does it weigh anchor text, though the corpus has shown before that anchor text goes missing at scale: 85 links across five captured pages carried no anchor text at all, 34 of them named only by an aria-label.

The 84 home pages that offered no path at all

A count of zero is the kind of result that is usually a bug in the counter, so all 84 were requested again the same day and classified by what their anchors actually did. None of them gained an internal path on the second request. The 84 split into five shapes, and they are not equally interesting.

55 of the 84 contained no anchor element whatsoever in the raw HTML. Not an anchor pointing somewhere unhelpful: none at all. The list includes ryanair.com, southwest.com, aircanada.com, britishairways.com and ethiopianairlines.com; nordstrom.com, lowes.com, jd.com, shopee.sg and takealot.com; schwab.com, rbc.com, zurich.com and bradesco.com.br; and archive.org, jstor.org, orcid.org, openstax.org and khanacademy.org. These are large organisations with large sites, and on 21 September 2026 a crawler that does not execute JavaScript could see exactly one of their pages. Most of them also returned no readable words, which is the same failure this blog counted when 17 of 380 home pages sent a crawler zero words and measured directly when JavaScript supplied all the prose on 11 of 271 pages.

16 had anchors that all pointed off the host. A personal site whose only links go to a newsletter platform, a payment host and two social profiles is complete from a reader's point of view and a dead end from a crawler's. 5 were one-page sites whose navigation is entirely fragment links into the same document, which is a legitimate design and still leaves a crawler with one URL. 4 pointed only back at the home page. The last 4 are the case Google's documentation names explicitly: anchors carrying no href a crawler can resolve to a page, including one site with 11 anchor elements and not a single href among them, and one whose only three anchors are mailto and tel links.

The distinction matters for what a site owner should do about it. A one-page site has nothing to fix and nothing to gain. A site whose navigation is built from elements that respond to script events is losing paths it believes it has, and the fix is markup rather than content, which is the shape of the per-stack advice in the React guide.

Which kinds of site left a crawler with nowhere to go

The corpus is stratified, so the 84 can be attributed rather than merely listed, and the pattern is clean enough to be worth stating plainly: this is a property of how the page is built, not of how large the organisation is.

The no-code stratum is the extreme. 18 of the 42 Bubble-built sites that answered, 42.9 percent, offered a crawler no internal path. The single page application startups follow at 6 of 45. Among the industry strata the rates are lower and the absolute numbers larger, because those strata are larger: travel at 12 of 78, ecommerce at 10 of 66, finance at 10 of 94 and healthcare at 9 of 95.

The other end of the table is the part worth dwelling on, because it is the good news and it is unanimous. Not one of the 54 Wix and Squarespace sites, the 37 WordPress sites, the 34 Shopify storefronts or the 38 SaaS marketing sites offered zero paths. Four platform strata, 163 sites, no failures. Those are the platforms that render navigation on the server as a matter of course, and their users did nothing to earn the result. This is the mirror image of a pattern this blog keeps finding, where a platform default decides the outcome and the site owner never sees the decision: it was true of the 141 of 382 home pages carrying no structured data and it is true here, with the sign reversed.

Two cautions on reading the table. The strata are sampling frames of unequal size and construction, so a rate is a statement about these sites rather than about the platform in general, and the smallest strata carry the least weight. And 214 hostnames refused this crawler outright with HTTP 403 before any of this could be measured, weighted heavily towards news, which is the same refusal pattern reported when 79 of 115 sites refused a crawler their robots.txt allows. Those sites are absent from every figure here.

What happened when the first internal link was requested

For each of the 1,007 home pages that offered a path, the first one in document order was requested the same way. First in document order is not most important, and nothing here claims otherwise: it is simply the link a crawler reading top to bottom meets first, which makes it a fair sample of the quality of what is on offer rather than a judgement about any site's navigation.

1,006 returned a status and 969 of those answered HTTP 200. The 37 that did not divide into refusals and mistakes. 11 answered 403, which is a site treating the second request from a crawler differently from the first. 2 answered 429, 2 answered 503, and one each answered 400, 405, 406 and 500. The remaining 18 answered 404, and that is the number worth explaining, because a broken link is ordinary and most of these are not.

On six of the 18, the href was never a URL. hoshinoresorts.com carries a Vue template expression, locale.texts indexed by hotel id and wrapped in double braces, sitting in an href attribute that was never evaluated. botanicapaper.com and oaklandish.com carry a fragment of JavaScript beginning window.location.href.replace. acc.org offers CTA.url and slate.com offers desktopMastheadUrl, which are variable names that reached the client as literal paths. thetinypod.com has an escaped quote and an image host inside the attribute. Each is a template that did not render, and a crawler has no way to know that: it reads an href, resolves it, and asks for it.

A further five of the 18 are Cloudflare's email protection path, which is a redirect endpoint rather than a page and answers 404 to a client arriving cold. Two more resolve to paths that are not there: wien.gv.at offers a link to nojs, and the first link on destatis.de repeats its own path prefix, which is the signature of a relative URL resolved against the wrong base.

That leaves five ordinary broken links, including one storefront whose first internal link is a collection page that no longer exists. Ordinary is the right word, and five is the honest size of it. The point of separating them is that a 404 rate of 1.8 percent on first links sounds like link rot and mostly is not: 13 of the 18 are markup describing a link the page never actually had, which is the same class of problem as a URL that answers 200 with nothing behind it, counted yesterday in the 125 sites that served a soft 404.

What the second page lost: prose and structured data

969 pairs of home page and first internal page can be compared directly, and on the whole the second page holds up. The median home page carried 1,009 readable words and the median first internal page carried 775, which is the shape you would expect when a home page is a summary of everything and an inner page is one thing.

Two minorities matter more than the median. 126 of the 969 inner pages carried under a quarter of their home page's word count where the home page had at least 100 words, and 13 carried no readable words at all. The 13 include starbucks.com, coinbase.com, flipkart.com, progressive.com and carnival.com, all of which returned a readable home page and then an empty second document. A site can pass a home page check and fail one click later, and nothing in a home page audit would show it. That is an argument for checking more than one URL, which is what multiscan exists to do.

Structured data travels worse than prose. 604 of the 969 home pages carried at least one JSON-LD block, and on 143 of those 604 the first internal page carried none: 23.7 percent of the sites that had it lost it one click in. The reverse happened too, on a smaller scale. 38 of the 365 home pages carrying no JSON-LD led to an inner page that had some, usually a product or article page where the platform emits markup the home page template does not.

This is worth taking seriously because of what structured data is for. It names the entity, and an answer engine that cannot resolve what a page is about has to infer it from prose, which is the difference structured data and entity confidence are there to describe. A site whose home page announces an Organization and whose article pages announce nothing has marked up the one page least likely to be the answer to a question. The scoring this scanner applies to a single page is set out in the methodology, and the honest reading of these figures is that a single-page score, including one of ours, describes a single page.

What to check on your own site, and what this did not measure

The check takes one command and no tools. Request your own home page the way a crawler does, without a browser, and count the a elements carrying an href that point at another page on your host. If that number is zero, nothing else on the page matters for discovery, because a crawler that does not render has arrived at a site with one document in it. If it is small, ask whether the pages you want quoted are among the ones you listed. You can see the same response this scan read on what GPTBot sees, and the declared identity this scanner arrives with is published at the bot page.

Then ask the second question, which the 143 answers above: does the page one click in carry what the home page carries? Structured data, readable prose and a title that names the thing are all properties of a template, and a site usually has several templates. Checking one of them is checking one of them.

Four limits bound everything here. No JavaScript was executed, so for any crawler that does render, a zero in this data may be a number that exists after rendering and not before; Google's own page says dynamically inserted anchors are crawlable, and the nine crawler vendors differ on whether they render at all. 214 hostnames refused this crawler with 403 and 31 never resolved, so 245 sites are simply absent, and they are not absent at random, which the eight reasons a scanner refuses a URL covers from the other direction. Only the first link was followed on each site, one hop deep, so nothing here describes a site's full link graph or how much of it is reachable. And the corpus is an editorial frame of 1,419 large organisations and platform-grouped sites, so every rate supports a statement about these hostnames and nothing wider, the same caveat attached to the crawlability study.

Nothing in these figures says a site with few internal links is doing badly by its readers. A one-page site is a one-page site, and the 55 with no anchor at all are running applications that work perfectly in a browser. What the measurement says is narrower and harder to argue with: on 21 September 2026, for a client that reads HTML and does not run scripts, 84 of 1,091 sites in this corpus were one page long, and half the navigation problem a reader worries about was solved by their platform without them knowing. Where a site sits on that is what AI visibility means in practice, and what a scan is for.