# Googlebot reads the first 2MB, and three AI crawler docs name no limit

> Google's crawler documentation, read on 10 August 2026, states that Googlebot crawls the first 2MB of a supported file type, stops the fetch at the cutoff, and applies the limit to uncompressed data. The crawler documentation published by OpenAI, Anthropic and Perplexity states no size limit at all, which leaves site owners with a documented ceiling from one vendor and silence from three.

- Canonical page: https://lantad.co/blog/googlebot-reads-the-first-two-megabytes
- This file: https://lantad.co/blog/googlebot-reads-the-first-two-megabytes.md
- Last substantive update: 2026-08-10

## Key facts

- **Published:** 2026-08-10
- **Category:** Findings
- **Author:** Lantad
- **Length:** 3440 words
- **Takeaway 1:** Google's Googlebot documentation, carrying a last updated date of 3 February 2026 and read on 10 August 2026, states that when crawling for Google Search Googlebot crawls the first 2MB of a supported file type and the first 64MB of a PDF file, and that once the cutoff limit is reached Googlebot stops the fetch and only sends the already downloaded part of the file for indexing consideration.
- **Takeaway 2:** The same page states that the file size limit is applied on the uncompressed data, so gzip or Brotli on the wire does not move the ceiling, and that each resource referenced in the HTML is fetched separately and bound by the same limit except for PDFs.
- **Takeaway 3:** Google's robots.txt documentation, carrying a last updated date of 8 July 2026, states that Google enforces a robots.txt file size limit of 500 kibibytes and that content after the maximum file size is ignored, while RFC 9309 section 2.5 requires only that a parsing limit be at least 500 kibibytes and says nothing about what happens to the bytes past it.
- **Takeaway 4:** The crawler documentation published by OpenAI, Anthropic and Perplexity, all three read on 10 August 2026, documents no file size limit, byte cap or fetch ceiling for GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot or Perplexity-User.
- **Takeaway 5:** Lantad caps its own raw HTML capture at 3 MiB in FETCH_LIMITS.maxHtmlBytes in core/src/config.ts. That is a setting we chose, not a measurement of any crawler, and the gymshark-shopify golden fixture exceeded it: the stored expectation carries a truncation caveat and marks two entity checks not proven rather than scoring them absent.

## Summary

Almost every question about whether a machine can read your website is asked as though reading were all or nothing. The crawler is allowed in or it is blocked. It runs JavaScript or it does not. The page is in the index or it is missing. That framing survives because it is usually close enough, and because the failures it describes are the loud ones: a 403, a blank shell, a [robots.txt](https://lantad.co/tools/robots-txt-tester) rule that names your whole site.

There is a quieter failure underneath it, and it has a number attached. A crawler does not read your page until it runs out of page. It reads until it runs out of budget, and at least one major vendor publishes what that budget is. This post reads what the crawler documentation actually says about size, compares the vendor that publishes a figure against three that publish none, and then describes what this scanner does at its own cap, which is the one part of the story where we are describing our own code rather than somebody else's documentation. The figures from Google, OpenAI, Anthropic and Perplexity are read from their pages on 10 August 2026 and attributed to them. We have not measured any crawler's real behaviour at a size boundary, and nothing below claims we have.

## What the 2MB figure covers, and what it does not

Google's [Googlebot documentation](https://developers.google.com/search/docs/crawling-indexing/googlebot) carries a last updated date of 3 February 2026 and states the limit in one sentence: when crawling for Google Search, Googlebot crawls the first 2MB of a supported file type, and the first 64MB of a PDF file. Three further sentences on the same page do most of the work of making that figure usable, and they are the ones that tend to go missing when the number is repeated second hand.

The first is about what happens at the boundary. The page states that once the cutoff limit is reached, Googlebot stops the fetch and only sends the already downloaded part of the file for indexing consideration. That is a specific and fairly generous behaviour. It is not an error, it is not a discard, and it does not fail the URL. What reaches indexing is a truncated document, which means the failure mode is not a missing page. It is a page whose last section quietly does not exist, and there is no status code anywhere in your logs that distinguishes that from a complete fetch.

The second sentence is the one worth writing down. The page states that the file size limit is applied on the uncompressed data. Compression is the standard answer to a page weight problem and it does not apply here at all. A 2.4MB HTML document that Brotli squeezes to 180 kilobytes on the wire is still a 2.4MB document against this ceiling, so the transfer figure in a browser network panel, which is the number most people have to hand, is the wrong number to check. The figure that matters is the decompressed byte length of the response body.

The third concerns subresources. The page states that each resource referenced in the HTML, such as CSS and JavaScript, is fetched separately, and that each resource fetch is bound by the same file size limit that applies to other files, excepting PDFs. For a site that renders its content client side that is a second, independent ceiling: a single application bundle over 2MB uncompressed is truncated on fetch, and a truncated script is not a script that runs slowly, it is one that does not parse. Whether that matters to you depends entirely on whether your text is in the served HTML or assembled afterwards, which is the distinction [prose parity](https://lantad.co/glossary/prose-parity) exists to measure and the one a [client rendered stack](https://lantad.co/fix/react) has to get right deliberately.

The page adds that other Google crawlers, for example Googlebot Video and Googlebot Image, may have different limits, and does not say what those are. So even the vendor that publishes a figure publishes one figure, for one crawler, for one purpose.

## robots.txt has a size limit too, and it is far smaller

The file that decides whether a crawler may fetch anything at all has its own ceiling, and it is much lower than the one on your pages. [Google's robots.txt documentation](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt), carrying a last updated date of 8 July 2026, states that Google enforces a robots.txt file size limit of 500 kibibytes, and that content which is after the maximum file size is ignored.

That figure is not Google's invention. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the Standards Track specification for the Robots Exclusion Protocol, states in its section 2.5 that the parsing limit must be at least 500 kibibytes. The two numbers agree, but they are not the same kind of statement, and the difference is the interesting part. The specification sets a floor on what a crawler must be capable of parsing. Google's documentation describes what Google actually enforces as a ceiling. A crawler that parsed 2 megabytes of robots.txt would be conformant with the RFC and would disagree with Google about your file.

The specification also does not say what a crawler should do with the bytes past its limit. Google's page does say, and the answer is that they are ignored. A Disallow line at kibibyte 600 of a 700 kibibyte file is not a rule that partially applies or that produces a warning. It is a rule that, from Google's perspective, was never written. That is a genuinely dangerous asymmetry for anyone generating robots.txt programmatically, because the natural failure of an oversized file is not that everything is blocked. It is that the last rules silently stop existing, and rules at the end of a generated file are usually the most specific ones.

Half a megabyte of robots.txt sounds unreachable until you have seen a file with a per-product-URL Disallow line, which large catalogues do generate. The [robots.txt tester](https://lantad.co/tools/robots-txt-tester) here resolves a rule per named crawler rather than reporting file size, so it will tell you what a given token is allowed to fetch without telling you whether your file is near a parsing ceiling. Those are different questions and it is worth being clear about which one has been answered. The related point that a rules file behaves very differently depending on how the server answers for it is covered separately in [what a 404 and a 503 on robots.txt each mean](https://lantad.co/blog/robots-txt-404-and-503-are-opposites), and the lag between editing the file and a crawler acting on it in [when a robots.txt edit reaches a crawler](https://lantad.co/blog/when-a-robots-txt-edit-reaches-a-crawler).

## Three AI crawler documentation pages name no limit at all

The obvious next question is what the crawlers that feed AI answers do at a size boundary, and the honest answer from their own documentation is that it is not published. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots) names four tokens, OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot, and states no file size limit, byte cap or fetch ceiling for any of them. [Anthropic's crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), which carries a date of 7 April 2026, names ClaudeBot, Claude-User and Claude-SearchBot and states none. [Perplexity's bot documentation](https://docs.perplexity.ai/guides/bots) names PerplexityBot and Perplexity-User and states none. All three were read on 10 August 2026.

It is worth being precise about what that absence is and is not. It is not evidence that these crawlers read unbounded documents. Every production fetcher has a bound somewhere, if only because a client that will read a response of any length is a client that can be made to run out of memory by a single URL. The reasonable inference is that these limits exist and are undocumented, which is a different problem from their not existing, and a worse one for a site owner: an undocumented ceiling cannot be designed around, cannot be tested against, and cannot be distinguished from any other reason a passage did not make it into an answer.

This is a familiar shape for anyone who has read these pages closely. The crawler documentation published by these vendors is consistently strong on the question of control, meaning which token to name and how to block it, and consistently quiet on the mechanics of retrieval. That pattern is why [six of the nine AI vendors we track publish exactly one crawler token](https://lantad.co/blog/six-vendors-publish-one-crawler-token) while operating more than one fetcher, and why [some tokens never appear in server logs at all](https://lantad.co/blog/the-crawler-tokens-that-never-appear-in-your-logs). The [current token list](https://lantad.co/tools/ai-crawlers) is the part these vendors do commit to, and it is the part worth acting on.

What follows for a reader is narrower than a size limit story might suggest, and it is better to say so than to manufacture a recommendation. There is no published AI crawler byte budget to engineer against. There is a published Google one, which is 2MB uncompressed, and it is a sensible order of magnitude to treat as a general ceiling on the grounds that a page comfortably under the one documented limit is unlikely to be near an undocumented one. That is a heuristic, and calling it a heuristic rather than a finding is the whole point of the distinction this site keeps drawing in [our methodology](https://lantad.co/methodology).

## What this scanner does at its own cap, and why it says so

This is the part of the post where the subject is our own code rather than a vendor's page, so the standard of proof is different and so is the voice. Lantad caps raw HTML capture at 3 MiB, written as FETCH_LIMITS.maxHtmlBytes in core/src/config.ts. That is a number somebody chose. It is not a measurement of anything, and it is not a claim about what any crawler does. It sits slightly above Google's documented 2MB so that a page at that boundary is captured whole rather than clipped by our own instrument before the question can be asked.

What matters is not the cap but what happens when a page exceeds it, because that is where a scanner can most easily produce a confident wrong grade. A truncated capture will score badly on structure, on [structured data](https://lantad.co/glossary/structured-data) and on [entity confidence](https://lantad.co/glossary/entity-confidence) for a reason that has nothing to do with the page: the markup that would have satisfied those checks may simply be in the part we did not read. Reporting that as an absence would be inventing a defect.

The gymshark-shopify golden fixture is a stored case of exactly this. Its expectation carries a confidence note reading that raw HTML exceeded 3MB and was truncated, and that parity was scored on the captured portion. It carries a defect with the id html_truncated. Two entity confidence signals, the absolute identity node and the machine readable category, are marked with an explicit qualifier: not proven, your HTML was truncated before we finished reading it. The same stored expectation holds a raw schema sub-score of 1.0 and an entity confidence of 75, so the truncated capture does not drag every number down. It changes what two of them are allowed to claim, and the report says so in the customer's own copy.

That behaviour is the same rule as [withholding a grade we could not measure](https://lantad.co/blog/why-we-withhold-a-grade), applied one level down. A whole scan can be refused when the evidence is not there. A single check inside an otherwise good scan can also be refused, and refusing it is more useful than a confident zero, because a zero and a not proven lead a reader to do completely different things. The wider version of the same argument, that text a crawler never receives cannot be scored as present, runs through [what a crawler meets on a real storefront](https://lantad.co/blog/what-a-crawler-meets-on-a-real-storefront) and [text inside shadow DOM reaching the browser and not the extractor](https://lantad.co/blog/shadow-dom-text-and-the-extractor).

Two things we have not done, since the absence is part of the report. We have not measured how any AI crawler behaves at a size boundary, because doing that honestly needs a controlled page served at graduated sizes and observed per token, which is not something [our current method](https://lantad.co/research) performs. And we hold no distribution of page weight across the sites scanned, so we cannot tell you what share of real pages are anywhere near 2MB. Both are answerable; neither is answered here.

## How to find out whether your own pages are near a limit

The check is cheap and almost nobody runs it, because the number that matters is not the one any of the usual tools puts in front of you. A browser network panel reports transfer size, which is the compressed figure, and Google's documentation is explicit that the limit is applied on the uncompressed data. A content management system reports the length of the article body, which excludes the markup, the inline styles, the serialised application state and the schema blocks that make up most of the weight of a modern template.

Fetch the page the way a crawler would, without compression, and count the bytes. On any machine with curl, request a single URL, send no Accept-Encoding header, and read the size of the downloaded body. Google's page writes 2MB without saying whether it means 2,000,000 bytes or the power of two 2,097,152, a difference of about 5 percent, so treat the smaller of the two as the ceiling and do not build a template that depends on which one it is. Note also that a page anywhere within a factor of two of that figure deserves attention rather than only one already over it, since templates grow and the ceiling does not. Do the same for your largest JavaScript bundle, which is subject to the same limit as a separately fetched resource, and for your robots.txt against the much smaller 500 kibibyte figure.

If a page is over, the fix is almost never compression and almost always the serialised blob. The usual causes on a real template are a full product catalogue inlined as JSON for a client side filter, a hydration payload that repeats the rendered content as data, and a base64 encoded image in the markup. All three are invisible in a word count and dominant in a byte count. The practical test of whether it matters is whether your text sits before or after the blob in the document, because with a truncating fetch, document order becomes a ranking of what survives. That is a strange property for a template to have and worth checking deliberately on a [Shopify theme](https://lantad.co/fix/shopify) or a [Next.js app](https://lantad.co/fix/nextjs), where a large embedded state object is a normal output of the framework rather than a mistake.

The general principle is the one the rest of this site keeps arriving at from different directions. What an [AI crawler](https://lantad.co/glossary/ai-crawler) can use is what it received, not what your page contains, and the gap between those two is made of specific mechanical things: a rule in a file, a script that did not run, a fetch that stopped. A byte ceiling is simply the least discussed member of that set. If you want the fastest read on where your own site sits, [what GPTBot sees](https://lantad.co/tools/what-gptbot-sees) shows the served document rather than the rendered one, which is the view a byte ceiling actually applies to.

## Questions and answers

**What is Googlebot's file size limit?**

Google's Googlebot documentation, carrying a last updated date of 3 February 2026, states that when crawling for Google Search Googlebot crawls the first 2MB of a supported file type and the first 64MB of a PDF file. The same page states that the limit is applied on the uncompressed data, and that once the cutoff is reached Googlebot stops the fetch and only sends the already downloaded part of the file for indexing consideration. It adds that other Google crawlers such as Googlebot Video and Googlebot Image may have different limits, and does not say what those are.

**Do GPTBot, ClaudeBot and PerplexityBot have a page size limit?**

Their published documentation does not say. The crawler documentation from OpenAI, Anthropic and Perplexity, all read on 10 August 2026, names nine tokens between them and states no file size limit, byte cap or fetch ceiling for any of them. That is an absence in documentation rather than evidence that no limit exists, and Lantad has not measured how any of these crawlers behaves at a size boundary.

**Does compressing my pages help me stay under the limit?**

Not for Google's stated limit. The Googlebot documentation states that the file size limit is applied on the uncompressed data, so gzip or Brotli changes the transfer size without changing the figure the limit is measured against. Compression remains worth doing for speed and bandwidth. It is the wrong tool for a size ceiling, and the transfer figure in a browser network panel is the wrong number to check against one.

**How large can a robots.txt file be?**

Google's robots.txt documentation, last updated 8 July 2026, states that Google enforces a limit of 500 kibibytes and that content after the maximum file size is ignored. RFC 9309 section 2.5 separately requires that a crawler's parsing limit be at least 500 kibibytes, which is a floor on parser capability rather than a ceiling on your file, and the specification does not state what happens to the bytes past a parser's limit. The practical risk is that rules at the end of an oversized generated file stop applying without any error appearing anywhere.

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot