Page size limit: 25 of 1,093 home pages passed 2MB, and not one did on the wire

What is the page size limit for a crawler, and what happens at it?

Google's Googlebot documentation carries a last updated date of 2026-02-03 UTC and states the figure in one sentence: when crawling for Google Search, Googlebot crawls the first 2MB of a supported file type, and the first 64MB of a PDF file. Three further statements on the same page decide what that figure means in practice, and all three were read at source on 15 October 2026.

The first is the behaviour at the boundary. The page states that Googlebot stops the fetch and only sends the already downloaded part of the file for indexing consideration. That is specific and comparatively forgiving. It is not an error, not a discard and not a failed URL. What reaches indexing is a truncated document, so the failure mode is not a missing page but a shortened one, and nothing in a server log separates it from a complete fetch.

The second is what the limit is measured on, and it is the sentence this whole post turns on: the file size limit is applied on the uncompressed data. Compression on the wire does not move the ceiling.

The third extends the same limit beyond the document. The page states that each resource referenced in the HTML is fetched separately and that each of those fetches is bound by the same file size limit that applies to other files, excepting PDFs, and it notes that Googlebot Video and Googlebot Image may have different limits.

None of that is an AI crawler. We re-read the three vendors whose crawlers this scanner actually models. OpenAI's crawler documentation documents OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User and states no size limit. Anthropic's crawler support article documents ClaudeBot, Claude-User and Claude-SearchBot, and its only statement about volume is that it supports the non-standard Crawl-delay extension to robots.txt. Perplexity's bots page documents PerplexityBot and Perplexity-User and states no limit either. That is the same silence we recorded in August, unchanged two months later, and it is worth being exact about what it licenses. A reader may not conclude that GPTBot reads an unlimited page. A reader may conclude only that no published figure exists, which is a different and worse position to be in, because it cannot be designed against.

So the 2MB figure is used here as what it is: the one documented cutoff in the field, applied as a measuring stick to real documents, with no claim that any AI crawler enforces it.

Does the page size limit apply before or after compression?

After. Google's page says so in one line, and the consequence is that the number a site owner is most likely to look at is the wrong one. Content-Encoding on MDN, last modified 4 July 2025, lists the encodings applied to a resource and makes the arithmetic explicit: when the header is present, metadata such as Content-Length refers to the encoded data rather than the original resource. The transfer size shown by a browser, a monitoring dashboard or a Content-Length header is therefore the compressed figure, and the documented limit is applied to a number nothing on the wire reports.

How large is the gap? Measuring it cleanly requires care, because a redirect chain makes a single downloaded byte count ambiguous. So the ratio below is computed only on the 290 pages that were reached with no redirect at all and that carried a Content-Encoding header, where the bytes received and the bytes stored both describe one response.

On those 290 pages the median page expanded 5.91 times on decompression. The tenth percentile expanded 3.45 times, the ninetieth 9.74 times, and the largest 21.28 times. 28 of the 290 expanded by a factor of ten or more. Turning the median around gives the practical form of the finding: at 5.91 times, the 2,000,000 byte limit is reached by a response weighing roughly 338,193 bytes on the wire. A page that reads as 330KB in a network panel is at the documented ceiling.

Compression is close to universal in this corpus, which is why the gap applies to nearly everyone in it. 1,020 of the 1,093 pages carried a Content-Encoding header: 598 gzip, 381 br and 41 zstd. 73 sent no encoding at all, and for those the wire figure and the limit figure are the same number.

The effect on the headline count is stark. Measured on the wire, this corpus contains no page size problem whatsoever: not one of the 1,093 pages delivered more than 2,000,000 bytes, and only two delivered more than 1,000,000. Measured as the documentation specifies, 25 passed the cutoff. The same 1,093 documents produce zero or 25 depending on which byte count is used, and only one of those two answers is the one the limit is written against. This is a different failure from the ones prose parity usually catches, where the text exists but a crawler cannot see it, and different again from a page with nothing to revalidate against. Here the bytes were served and counted, and there were simply too many of them.

25 of 1,093 home pages passed the documented cutoff

The capture ran on 15 October 2026 over the 1,419 hostnames in this repository's two committed corpus seed files, 392 sampled by the platform a site is built on and 1,027 by industry. Every hostname was asked for its robots.txt first and the file parsed with this scanner's own parseRobotsTxt, with each root verdict computed by its evaluateRobots. 1,146 answered HTTP 200 for the file, and 1,404 of the 1,419 hostnames carried no rule disallowing LantadBot at the root, so those 1,404 home pages were requested once each. 1,093 answered HTTP 200 with an HTML content type. 188 answered 403, 98 failed before any status arrived, and the remainder spread across 429, 404, 426, 402, 400, 503 and 498. The full capture procedure is on the methodology page, and the user agent is documented on LantadBot's own page.

25 of the 1,093 delivered more than 2,000,000 bytes of uncompressed HTML. Google writes the limit as 2MB without saying whether that is 2,000,000 bytes or 2,097,152, so both readings were counted: 23 of the 25 also pass the larger figure, and two sit between the two readings. The ambiguity moves the count by two and changes nothing else, which is why the strict reading is used throughout.

Every one of the 25 was compressed, and most of them were unremarkable on the wire. 17 of the 25 arrived in under 500,000 bytes and 7 in under 250,000. The smallest was handelsblatt.com at 110,174 bytes received against 2,153,216 uncompressed. The largest wire figure among them, irishtimes.com at 1,047,074 bytes, was one of only two pages in the entire corpus to pass a megabyte on the wire.

Size at this scale turns out to be reasonably stable. All 25 were re-fetched the same day and all 25 were still over the cutoff on the second pass. Most moved by a few hundred bytes. The largest movement was wix.com, from 3,007,583 to 2,509,829, which is a swing of half a megabyte and still comfortably over. That matters for the claim being made: a page at 2,010,000 bytes might cross back and forth, but these pages are not sitting on the line.

The distribution across the corpus strata is lopsided. 13 of the 25 are news home pages, from the 96 news home pages that answered, so roughly one news front page in seven passed the cutoff against 25 in 1,093 overall. The other 12 spread thinly: two each from travel, ecommerce, SaaS, SaaS marketing and Shopify storefronts, and one each from healthcare and education. Earlier corpus work is collected on the research page.

What the cutoff cost: 11 of the 25 lost words and 14 lost none

Passing a size limit is not the same as losing content, and conflating the two would have produced a tidier post and a false one. So the 25 were tested rather than assumed. Each page's raw bytes were cut at exactly 2,000,000, and the truncated document was passed through the same extractFromHtml in core/src/extract.ts that the shipped scanner uses, with the full document extracted as the control.

Across the 25, the full documents yielded 45,554 words and the truncated documents 33,425. The cut removed 12,129 words, 11,441 of which the extractor classes as main content rather than navigation or boilerplate. That total is the honest headline, and the distribution underneath it is the actual finding: 11 of the 25 lost at least one word and 14 lost nothing at all. On those 14, every word, heading and JSON-LD block sat inside the first 2,000,000 bytes and the tail that would have been discarded carried no readable text. irishtimes.com is the clearest case, holding 2,528 words and 203 headings in a 4,517,334 byte document with all of them in the first 2MB.

The eleven that did lose content lost very different amounts. elgiganten.se fell from 6,415 words to 1,368, a loss of 5,047. wix.com fell from 2,447 to 70. mirrormate.com fell from 1,970 to 626 and was the only page in the 25 to lose a structured data block, dropping from three parseable JSON-LD blocks to two. globo.com lost 799 words, klaarhq.com 610, chosun.com 552, vietnamairlines.com 453, sueddeutsche.de 317, fieldguide.io 93 and atlassian.com 87.

cnn.com is the case worth stating precisely, because it is the most extreme result in the measurement and the easiest to misread. Its full document yields 450 words, of which 424 are main content. Truncated at 2,000,000 bytes it yields zero. The reason is not that its prose sits near the end of a long article. It is that the document opens its body element at byte 2,599,949, so a fetch stopping at 2,000,000 bytes never reaches any body content at all. Everything before that offset is head: twelve script elements account for 2,572,281 bytes of the document, and the largest single one is 2,293,195 bytes. A crawler obeying Google's documented cutoff would receive a document with a title and no body.

Two caveats belong next to that figure. 450 words is a small yield for a major news front page to begin with, which is consistent with what we found when hydration JSON scored the same as no text and when client side rendering supplied the prose. And no JavaScript ran in this measurement, so these are the figures for a crawler that reads HTML only.

Why the documents get this big, and what this measurement does not show

The pages that passed the cutoff are not pages with more writing on them. They are pages with more code in them. Counting script element bytes against total document bytes across all 1,093 pages puts the corpus median script share at 13.7 percent. On the 25 over-cutoff pages the median is 62.1 percent. 13 of the 25 spend more than half the document inside script elements, against 184 of the 1,093 overall.

The text share runs the other way and is the more uncomfortable number. Across the 1,093 pages the visible text the extractor recovers is a median 2.8 percent of the delivered bytes. On the 25 over-cutoff pages it is a median 0.46 percent. A page can therefore pass a limit written to bound how much a crawler reads while carrying almost nothing for it to read, which is the same shape as Next.js home pages sending 15 times more payload than prose and of the finding that half the text blocks were navigation. Framework level advice for trimming that payload sits on the Next.js guide.

What this measurement does not show matters as much as what it does. No AI crawler was observed and no request was sent as one. Every byte here was fetched as LantadBot and compared against a figure Google publishes for Googlebot, so the finding is about documents and a documented limit, not about any crawler's real behaviour at a boundary. Whether GPTBot, ClaudeBot or PerplexityBot truncate at all is not knowable from any of the three vendors' pages, as the first section sets out. No JavaScript was executed, so these are HTML-only figures, and a page whose words arrive through script would score differently in a browser, which is the comparison behind whether AI crawlers render JavaScript.

One limit is ours and should be said plainly. This scanner caps its own raw HTML capture at FETCH_LIMITS.maxHtmlBytes in core/src/config.ts, which is 3 MiB, or 3,145,728 bytes. That is a setting we chose, not a measurement of anything. Nine of the 1,093 pages here exceed it, and all nine are among the 25: cnn.com, irishtimes.com, vietnamairlines.com, cancerresearchuk.org, clarin.com, inquirer.com, austria.info, chosun.com and costco.com. Had this study run inside the shipped scanner, those nine would have been truncated by our cap before Google's limit was ever reached, and the 16 remaining over-cutoff pages sit between 2,000,000 and 3,145,728 bytes. The capture for this post was run outside that cap for exactly that reason, and the composite grading we published for the same corpus in the AI visibility score was not. Finally, 1,419 hostnames from two committed sampling frames are not a sample of the web, so no proportion here generalises past them, and a home page is one page: whether interior pages behave the same way is a separate measurement. The vocabulary for the rest of it is on the AI visibility page.