Next.js AI crawlers: 28 home pages sent 15 times more payload than prose

How many sites in a 392 host sample run Next.js?

The frame is the same one we have been working through platform by platform for the last week, and it is worth restating what it is and is not. It is a committed list of 392 hostnames grouped into ten strata by the kind of site they are, kept in the repository so that any figure published from it can be recomputed against the same list. It is not a random sample of the web. It over-represents hosted site builders and small software marketing sites, because that is what it was assembled to cover, so a proportion measured here describes this frame and not the internet.

We asked each hostname once for its home page over HTTPS, identifying ourselves as LantadBot, following redirects, with a twenty five second timeout. Three hundred and eighty three returned HTTP 200. Of the nine that did not, one returned 404, two returned 403, one returned 503, and five returned HTTP 202. Four of those five, rechecked the same day, answered with a 169 byte captcha challenge rather than a page, which is the same bot defence we looked at from another angle when five of 391 home pages sent an X-Robots-Tag and all five were a captcha.

Platform detection ran over the raw bytes of the 383 that answered, using the ordered signal list the product ships rather than anything written for this post. Next.js is tested first in that list, so a page carrying both a Next.js signal and, say, a WordPress one is counted as Next.js. Twenty eight matched: 7.3 percent of the pages that answered. That places it sixth by count, behind Webflow and WordPress at 65 each, Squarespace at 40, and Framer and Shopify at 35 each. Two of the 28 turned out to be the same site published on two domains, returning documents of 205,752 bytes that hash to the same MD5 digest, which is worth naming because it means the 28 pages cover 27 distinct sites.

The 28 do not sit where a reader might guess. Eight came from the software marketing stratum and seven from the startup stratum, which is expected. Six came from the stratum assembled for Framer sites and four from the stratum assembled for no-code tools, which is not: those are sites that have moved off a builder, or never were on one, and still sit in a list that says otherwise. That is a limitation of a fixed frame rather than a finding about Next.js, and it is why the stratum labels do not appear in any figure below. Every count in this post is over the 28 pages that carried the signal, not over a stratum.

What do Next.js AI crawlers receive from an App Router page?

We took each of the 28 documents exactly as delivered and split every character into four groups that cannot overlap. Script element bodies came out first, since an HTML parser never reads them as text. Within those, we measured separately the characters sitting inside self.__next_f.push calls and inside a script with the id __NEXT_DATA__, which are the two shapes Next.js uses to hand the serialised tree to the browser. Visible prose came from the scanner's own extractor, the same one that produces the text a report is graded on, which drops script, style, noscript, template, svg and iframe content before counting. Everything left over is markup: tags, attribute values, inline style rules, inline SVG path data.

The four groups sum to 11,321,162, which is every character delivered. Markup is the largest at 6,426,905, and nobody should be surprised by that: attributes and inline styles are how a modern page is built. The payload is second at 3,897,015, made up of 3,805,887 characters of App Router flight data and 91,128 characters of __NEXT_DATA__. All other script bodies together came to 744,063, which is less than a fifth of the payload. Prose is last at 253,179 characters, or 2.2 percent.

Per site the picture is consistent rather than driven by outliers. Twenty six of the 28 shipped more payload than prose. Eighteen shipped more than ten times as much. The median page delivered 208,193 characters and 5,209 characters of prose, and the median payload was 77,164 characters. Among the pages carrying more than 500 words, the widest ratio was a direct to consumer site that sent 732,923 bytes: 506,015 characters of payload against 3,802 characters of prose, or 133 to one, for 534 words of copy.

Two pages went the other way, and both are instructive. One is an API reference that ships 14,642 words of tabular content and only 17,947 characters of payload, because there is very little interactivity to serialise. The other is a Pages Router site whose __NEXT_DATA__ block is 199 characters, because it passes almost no props through it. Neither is doing anything exotic. They are pages whose content is mostly text, and the ratio follows the content rather than the framework. That is the useful version of this finding: the framework sets a floor on the overhead, and what the page is made of decides everything above it. Our Next.js fix guide covers the levers, and prose parity is the measure the report grades this on.

Is the RSC payload a second copy of the text?

Next.js documents what the payload is, and the definition matters here because it decides whether the 3,897,015 characters are waste or work. The App Router documentation page on Server and Client Components, carrying a last updated date of 25 August 2026 and a version label of 16.3.5, defines it directly: the RSC Payload is a compact, serialised representation of the rendered React Server Components tree, used by React on the client to update the browser's DOM, and it contains the rendered result of Server Components, placeholders for where Client Components should be rendered with references to their JavaScript files, and any props passed from a Server Component to a Client Component. The same page says the HTML is used to immediately show a fast non-interactive preview of the route. The page is at nextjs.org/docs/app/getting-started/server-and-client-components.

Read plainly, that means the rendered result of the server tree is serialised twice on first load: once as HTML for the preview, once as payload for the reconciliation. So we tested the overlap. For each page we took the twenty longest visible text blocks of six words or more, normalised whitespace, and asked whether the first sixty characters of each appeared anywhere inside the payload region. Twenty five of the 28 pages had both a payload and enough text blocks to sample.

Thirteen of those 25 had at least half their sampled blocks present in the payload, and the median across the 25 was exactly 50 percent. Five pages hit 85 percent and one hit 100 percent. Five returned zero, which is what a route with little interactive content looks like: almost nothing to serialise, so the payload carries structure rather than copy.

The method has real limits and they cut both ways. A sixty character prefix match misses text the payload holds in split chunks or with different escaping, so the true overlap is probably higher than 50 percent rather than lower. It also only samples the longest blocks, which are the ones most likely to be genuine content. What it establishes is narrower than a percentage: on the median Next.js page in this sample, a meaningful share of the payload is text the crawler has already read in the markup, delivered a second time in a form it cannot read. That is not a bug in Next.js. It is the cost of hydration, and it is the same shape as the case where text in hydration JSON scored the same as no text at all, with one important difference: there the text was only in the JSON, and here it is in both places. A site in the first situation has a problem. A site in the second is paying a tax, and the React prose parity guide is about the first.

Does a Next.js page hit a crawler's byte limit?

There is exactly one published number to check this against. Google's Googlebot documentation, last updated 3 February 2026, states that Googlebot crawls the first 2MB of a supported file type, and the first 64MB of a PDF. No AI crawler vendor publishes an equivalent figure, which we went through token by token when we found that Googlebot reads the first two megabytes and three AI crawler vendors name no limit at all. So the honest framing is that 2MB is the only cutoff anybody outside these companies can plan against, and it belongs to a crawler that is not an AI crawler.

Against that line, one of the 28 pages is over. It delivered 2,827,930 bytes, so a client applying Google's rule would stop at roughly 71 percent of the document. What sits in the last 29 percent of a Next.js page is not random: flight data is streamed in chunks appended through the body, so the tail of a large App Router document is disproportionately payload rather than prose. On this particular page the prose is 9,634 characters in a document of 2.8 million, so the content is almost certainly inside the first 2MB. Being truncated and losing the content are different events, and we did not measure which happened.

The next largest were 1,315,648 bytes, 838,518 and 732,923. Twenty four of the 28 were under 600,000 bytes and the smallest was 4,938. So this is not a widespread problem in this sample, and a post claiming Next.js pages routinely blow past crawler limits would be overstating one page into a trend. What the distribution does show is a long tail: the largest page in the sample is roughly 570 times the size of the smallest, and the four largest together account for 5,715,019 of the 11,333,470 bytes delivered by all 28.

The reason to watch it at all is that the number nobody publishes is the one that matters. Crawl budget is spent in bytes as well as requests, and across these 28 pages the payload raises the cost from 219 characters per readable word to 333, which is half as much again for nothing a parser can use. That is the same argument that applies to the archive crawlers: Common Crawl archived 2.14 billion pages in July without running JavaScript, and every byte of payload in those captures is a byte that carried nothing into the archive.

Do Next.js sites name AI crawlers in robots.txt?

Rendering is only half of what decides whether a crawler reads a page, so we asked each of the 28 hostnames for its robots.txt on the same day. Twenty five answered HTTP 200 with something that was not an HTML document. Three returned 404, which under RFC 9309 means unrestricted access rather than no access. All 25 files carried at least one User-agent line, which already separates this group from the hosted builders, where we found 20 of 40 Webflow files holding no rule of any kind.

Seven of the 25 name at least one of the 15 AI crawler tokens the scanner evaluates. Eighteen name none. The split is visible in the file sizes: the seven that name tokens run from 487 to 6,603 bytes, and the eighteen that do not run from 66 to 722 bytes. That is the difference between a file somebody wrote and a file somebody accepted. Seven of 25 is a better rate than we found on Wix, where none of 69 files named a single token, and a worse one than Squarespace, where 64 of 66 carried a platform generated file naming 26 tokens that nobody at the site chose either way.

The token distribution inside those seven is tight. GPTBot, ClaudeBot, PerplexityBot and Google-Extended each appear in all seven. OAI-SearchBot and ChatGPT-User appear in six. Claude-SearchBot and Claude-User appear in two. That last gap is the one worth acting on, because it is the same asymmetry we found at scale when GPTBot turned out to be the most blocked AI crawler, disallowed by 82 of 718 robots.txt files against 24 for its search twin. A file that names ClaudeBot and stops has addressed the training crawler and said nothing to the two that fetch pages for answers.

One file in the 25 does something specific to this framework: it carries a Disallow for /_next, the path Next.js serves its build assets from. That is the only appearance of the string in any of the 25 files. It is a defensible rule for a crawler that would otherwise walk a directory of hashed JavaScript chunks, and it changes nothing about whether a page is readable, because the payload is inline in the HTML rather than fetched from that path. If you want to see how your own file resolves per token rather than as prose, the robots.txt tester evaluates each one separately.

What to check on your own Next.js site

Four checks come out of this scan, in the order they change a result. Each one can be run from a terminal against your own domain in under a minute, and none of them needs a scanner.

First, count the prose. Fetch your home page without a browser and strip script, style and noscript before counting words. Four of the 28 pages in this sample came back under 50 words, and two returned none at all. A site in that state is not paying a hydration tax, it is failing to render, and the fix is the standard one: move the fetch out of a client effect and into a Server Component or a data loading function. Only two of nine crawler vendors state whether they execute JavaScript at all, which we established when we read all nine and found two of nine operators say either way, so a page that needs a renderer is gambling on an undocumented behaviour.

Second, check the structure the text sits in. Seventeen of the 28 carried at least one JSON-LD block and eleven carried none, which is 61 percent and is no better than the frame as a whole: the day before, across the same 392 hostnames, 141 of 382 home pages carried no structured data in the raw HTML, leaving 63 percent that did. Server rendering gets the text out. It does not get the schema out, because nothing emits schema unless somebody writes it. Sixteen of the 28 carried exactly one h1, four carried none, and eight carried more than one, the highest being ten on a single page. An extractor reading structure rather than styling can be stricter than you expect about that, as we found when an h1 inside a header element counted as none. The vocabulary itself is covered in our note on structured data.

Third, weigh the page. Divide the characters your home page delivers by the words of prose it carries. Across the 26 pages here that held at least one word, the median was 383 characters delivered per word. If your figure is several times that, the payload is the first place to look, and the lever is the client boundary: every component marked with the client directive pulls its props into the serialised tree. Moving a boundary deeper is the change that shrinks it.

Fourth, read your robots.txt per token rather than as a document. Eighteen of the 25 files here name no AI crawler at all, which is a decision by default rather than a decision. Whether you want the crawlers is your call, and AI visibility is worth wanting only if you also want the crawl. What is not a call is not knowing. What GPTBot sees renders the crawler side of a single URL, and the methodology page sets out what a grade counts and what it refuses to guess.

What this post did not measure should be stated plainly. We read HTML and robots.txt on one day, 12 September 2026, from one network location, using the user agent documented on our bot page. We did not observe any AI crawler requesting any of these sites, we read no access log, and no figure here reports what a crawler did with any of these bytes. We rendered nothing in a browser, so every count describes what arrived rather than what a person eventually sees. Twenty eight pages is a small sample and one unusual site moves a percentage by nearly four points. And the framework attribution is a signal match on delivered bytes, not a statement about anybody's build configuration.