Scan a website for all pages: 985 links found, 200,712 URLs declared

Can you scan a website for all pages from its homepage?

Start with what a link crawl actually yields. Lantad holds six real pages captured and stored on 15 July 2026: the astro.build homepage, the gymshark.com storefront, the developer.mozilla.org HTML reference, the webflow.com homepage, the en.wikipedia.org article on web crawlers, and the www.allbirds.com homepage. On 29 August 2026 every anchor element in each stored document was parsed, its href resolved against the page's own origin, and the result kept only when it stayed on the same registrable host. Fragments, mailto, tel and javascript targets were dropped, and what survived was deduplicated by path plus query string.

Across all six documents that produced 985 unique internal paths from the rendered DOM and 994 from the raw HTTP response body. The per page figures are 52 for astro.build, 184 for gymshark.com, 344 for the MDN reference, 91 for webflow.com, 314 for the Wikipedia article and zero for allbirds.com.

Two of those six are not homepages, and the distinction matters when reading the numbers. The MDN and Wikipedia captures are interior documents, a reference page and an encyclopaedia article, and both are unusually link dense because that is what those page types are. A homepage is the entry point a crawler starting from the domain root would actually get, and the four homepages in the set returned 52, 184, 91 and zero.

The zero is the most informative row and it is not a parsing failure. The stored raw response for allbirds.com is 18 bytes long and contains the string local_rate_limited, and the capture's own confidence note records the reason in its own words: the baseline fetch returned HTTP 429, so crawlers get no content at all. A site that rate limits an unfamiliar client hands a link crawler nothing to walk, which is a complete answer to how many of its pages such a crawler can enumerate.

The gap between the raw and rendered counts is nine paths across six documents, which is small enough to say plainly that client side rendering added almost nothing here. That agrees with the earlier finding on these same captures that JavaScript added no new crawl paths, and it is worth holding alongside the separate problem that crawlers do not scroll, because a path that only appears after an intersection observer fires is a path no counting method here would ever see. If you want to look at a single page through this lens rather than take the aggregate on trust, what GPTBot sees renders the same distinction between the fetched document and the rendered one, and the wider question of whether the two agree is prose parity.

Does fetching the sitemap give you every page in one request?

The obvious answer to an incomplete link crawl is to read the map the site publishes. On 29 August 2026 each of the six hosts was asked for its robots.txt. All six returned HTTP 200 and all six carried at least one Sitemap line. The gymshark.com file carried the identical Sitemap line three times, once in each of three user agent groups, which is harmless and is a fair illustration of how these files are assembled rather than authored.

It is worth being precise about what that line is, because it is often described as part of the robots standard and it is not. RFC 9309, the Robots Exclusion Protocol, is a Standards Track document published in September 2022, and its section 2.2.4 says that crawlers may interpret other records that are not part of the robots.txt protocol, giving Sitemaps as its example. So the Sitemap directive is a convention that the standard explicitly makes optional to read, sitting in a file whose actual subject is exclusion. If you want to see how a given file parses against a specific token rather than reason about it in the abstract, that is what the robots.txt tester does, and the conduct rules this scanner holds itself to are on the bot page.

Five of the six declared addresses returned HTTP 200. Every one of those five was a sitemapindex element containing zero url elements. Not one site in the set answered the question "what pages do you have" in a single request. What each returned instead was a list of other files to fetch: six children for allbirds.com, one for astro.build, nine for gymshark.com, ten for developer.mozilla.org and six for webflow.com, which is 32 child documents in total.

This is the documented design rather than a quirk. Google's guidance on building a sitemap, carrying Last updated 2026-07-08 UTC, states that all formats limit a single sitemap to 50MB uncompressed or 50,000 URLs, that a larger set must be broken into multiple sitemaps, and that an index file can optionally be created and submitted in their place. The same page gives the robots.txt Sitemap line as the way to advertise it.

The two shapes those children take are the practical obstacle. MDN's ten are one per locale and every one is gzipped, so a client that does not decompress gets bytes it cannot parse. Webflow's six are not on webflow.com at all: each loc points at d3e54v103j8qbb.cloudfront.net, a CDN hostname. A crawler restricted to the site's own host, which is a reasonable thing for a scanner to be, follows the index and then stops.

The sixth site is the sharper lesson. The Sitemap line in en.wikipedia.org's robots.txt points at a REST API path, and that address returned HTTP 403 on 29 August 2026 with a Wikimedia error page rather than XML. The site declares a map and the map refuses the request.

How far apart are the linked count and the declared count?

Following all 32 children gave 203,189 loc entries, of which 200,712 were unique. The duplicates are not evenly spread: webflow.com accounted for 2,442 of them, gymshark.com for 33, and allbirds.com and developer.mozilla.org for one each. Deduplication is not a formality when one site repeats a URL two thousand times in its own map.

The per site unique totals are 2,138 for allbirds.com, 325 for astro.build, 7,209 for gymshark.com, 66,771 for developer.mozilla.org and 124,269 for webflow.com. Set against the linked counts from the captures, the share of a site's declared URLs that were reachable as links from the single captured page runs 16 percent for astro.build, 2.6 percent for gymshark.com, 0.52 percent for the MDN reference, 0.073 percent for webflow.com and zero for allbirds.com.

One of those child documents contained exactly 50,000 URLs: webflow.com's showcase_projects_sitemap0.xml, with a second file carrying the remaining 45,271. That is the documented protocol cap hit exactly, which is a useful thing to see in the wild, because it tells you the file was generated by something that knows the limit and splits on it.

The total cost of the exercise was 44 HTTP requests: six robots.txt files, six declared sitemap addresses and 32 children. That is the honest answer to how expensive it is to scan a website for all pages when the site cooperates, and it is a floor rather than an estimate, because it counts only the requests needed to obtain the list. Fetching the 200,712 pages themselves is a different order of work, and the constraints on it are the ones already covered here: Googlebot reads the first two megabytes of a document, and the crawl budget advice written for search crawlers lands differently on AI crawlers.

Two limits on the comparison need stating rather than burying. The captures are from 15 July 2026 and the sitemaps from 29 August 2026, six weeks apart, so a site that published pages in between widens its own gap for reasons that have nothing to do with linking. And the link counts come from one page per site, not from a crawl: a real crawler following those 52 or 184 paths would find more paths on the next hop. What the figures establish is the size of the first step, not the limit of a full traversal. The scoring rules that govern what this scanner does with a single page are on the methodology page, and the aggregate figures it publishes across scans are on the crawlability study.

What does full enumeration find that a link crawl does not?

The reason to do this properly rather than approximately is that the tail of a sitemap index holds things nothing on the page points at. One of allbirds.com's six children is named sitemap_agentic_discovery.xml. Fetched on 29 August 2026 it contained exactly one URL, https://www.allbirds.com/agents.md, with a changefreq of weekly and no lastmod.

That document, read at its own address on the same day, is an instruction file addressed to software rather than to people. Its first line is a heading reading Agent Instructions for Allbirds, and it describes how AI agents can interact with the store. It names two mechanisms. The first is a skill file at https://shop.app/SKILL.md which it says is the recommended way for browser using agents, buy for me agents and personal shopping assistants to transact across Shopify stores, and it asks an agent reading on a user's behalf to recommend that the user allow it to install that skill. The second is the Universal Commerce Protocol, documented at ucp.dev, for which the file gives a discovery endpoint at /.well-known/ucp and an MCP endpoint at /api/ucp/mcp taking a tools/list call, and names 2026-08-25 as its latest stable version. Those hosts are not registered for outbound linking here, so their addresses are written as plain text.

Take the measurement claim carefully, because there is a tempting one that this work does not support. Nothing here establishes that the file is new, that Shopify ships it by default, or how many stores carry it. One store carried it on one day. What is established is narrower and still worth the fetch: a site put a machine readable instruction document in its sitemap, in a child file named for agent discovery, and the only way to find it was to walk the index to the end. A link crawl of that homepage could not have found it, and in this particular case a link crawl of that homepage found nothing at all, because the fetch returned 429. Stores on this platform have their own readability characteristics, which is what the Shopify fix guide covers.

This is the same pattern as llms.txt, a file that exists to tell a model reader something the page does not, and it deserves the same scepticism: publishing a file is not evidence that anything reads it, and this site has published the counter evidence on llms.txt rather than sell around it. What the discovery sitemap adds over a bare well known path is that it uses a mechanism crawlers already follow. If you want to see what your own file looks like to a reader that is not a browser, the llms.txt tool renders it, and the tokens that might come asking are listed under AI crawlers. Whether any of it moves AI visibility is a separate question this measurement does not touch.

What a page count does not tell you

A declared URL is a claim by the publisher and nothing more. It is not evidence that the page exists, that it returns 200, that any crawler fetched it, that a search engine indexed it, or that a model reader could extract a sentence from it. Every one of those is a separate measurement, and running one of them does not license a statement about the others.

That distinction is not pedantry, it is the recurring failure in this subject. A sitemap entry carries a lastmod value that the publisher wrote, and this site has already argued that a sitemap tells a crawler where pages are, not that they changed. The same shape applies to discovery generally: IndexNow speeds discovery, not readability. Knowing about a page faster does nothing for a page whose text a crawler cannot reach. A count of 124,269 declared URLs says a generator emitted 124,269 loc elements. It says nothing about how many of them hold prose an answer engine could quote.

The specific limits on the figures above are worth listing so nobody has to infer them. Six pages is not a sample of the web and five sitemap sets are not a survey of sitemaps. Two of the six captures are interior pages rather than homepages, so their link counts are not comparable with the four that are. The link counts come from anchor elements only, so a path reachable through a form, a script or a JSON blob was never counted. The two halves are six weeks apart. The sitemap fetches were made with curl, following redirects and accepting compression, which is not the same client the product uses and is not a scan: no scoring ran, no grade was produced and nothing in this post reports what any of these sites would score. And the one site that could not be enumerated, Wikipedia, is excluded from the 200,712 rather than estimated into it.

What survives all that is a single structural point, and it is the useful one. If you want to scan a website for all pages, you need the list first, and the list is not something the site's own pages will give you. It comes from the map the site publishes, which takes several requests to assemble, arrives in a shape that varies by platform, may sit on a hostname you did not expect, and may simply refuse you. That is a discovery problem to solve before any readability work begins, and readability is where the actual generative engine optimization question lives. Aggregate figures from the scans this site has run are published on the research page.