Common Crawl archived 2.14 billion pages in July without running JavaScript

What Common Crawl published on 28 July 2026

The announcement is written by Sebastian Nagel, described on the page as a Distinguished Engineer at the Common Crawl Foundation, and it opens with the figures rather than burying them. The crawl archive for July 2026 is available, the data was crawled between July 7th and July 25th, and it contains 2.14 billion web pages, or 364.01 TiB of uncompressed content. Page captures come from 40.5 million hosts or 33.2 million registered domains and include 603 million new URLs not visited in any prior crawl. The identifier is CC-MAIN-2026-30, and Common Crawl's latest crawl page names the same identifier as current.

Read those four numbers next to each other and the shape of the thing becomes clear. 2.14 billion pages against 33.2 million registered domains is an average of roughly 64 pages per domain, which is arithmetic on the two published figures rather than a distribution Common Crawl reported, and the real distribution is certainly nothing like flat. The point of the ratio is not precision. It is that this is a sample of the web rather than a copy of it, which Common Crawl says outright in its FAQ: the dataset is a sample, and it does not generally archive an entire website but a randomly selected subset of it. A site owner asking whether their pages are in there is asking a question with a probabilistic answer, and no external tool, ours included, can turn that into a yes or a no.

The archive itself sits at a fixed and public location, crawl-data/CC-MAIN-2026-30/ in the commoncrawl bucket, reachable over S3 or over HTTPS at data.commoncrawl.org. That permanence is the property that makes this different from a live fetcher. An answer engine's retrieval of your page is an event that happens and is over. A Common Crawl capture is an object with an address. Every one of the fifteen crawler tokens in our public crawler directory behaves the first way except this one, and treating them all as the same kind of risk is the most common mistake in AI visibility advice about blocking.

Common Crawl states that its crawler executes no JavaScript

Here is the sentence that decides what your site contributed. In the technical section of Common Crawl's FAQ, under the question of how CCBot fetches a web page, the answer ends: currently, JavaScript is not executed and cookies are not used. That is the operator of the crawler describing its own behaviour in the present tense, on the page it points its user agent string at, and it is about as unambiguous as vendor documentation gets.

The consequence is mechanical and it is not a matter of opinion. If your page ships an HTML document that already contains your prose, that prose is what went into the WET text extract for whichever of your URLs were sampled during those nineteen days. If your page ships a nearly empty document and assembles its content in the browser, then what went into the extract is the nearly empty document. There is no queue, no second pass, no rendering service that catches up later. Google's JavaScript SEO basics guide states that Googlebot queues pages for rendering and that a headless Chromium executes the JavaScript once resources allow, which is a real difference between the two and one worth keeping straight; Common Crawl publishes no such step, and the absence is stated rather than inferred.

This is exactly the quantity prose parity exists to measure, and it is why parity carries 50 of the 100 composite points in our score, more than access, structure and schema put together. We fetch the URL once over plain HTTP with our own honest user agent, load the same URL in a real browser, and score what share of the rendered visible text was already present in the raw response. A parity near 1.0 means a crawler that never runs a line of JavaScript receives what a visitor receives. A parity of 0.3 means most of your page exists only after a bundle executes. What GPTBot sees renders that comparison for a single URL if you want to look at your own before reading further.

The reason to raise it in a post about an archive rather than about an answer engine is that the failure is not recoverable here. A live fetcher that got an empty shell from you this morning can get a full page from you tomorrow, because it will come back. A crawl archive published on 28 July 2026 is finished. If your React application returned a shell during that window, the shell is what is in CC-MAIN-2026-30, and the React specific causes and fixes matter for the next crawl rather than for that one.

The archive preserves your robots.txt and your refusals

The file list in the announcement contains two entries most readers skip, and both are more interesting than the WARC line. One is a robots.txt dataset: 100,000 files, 0.15 TiB compressed. The other is non-200 responses: 100,000 files, 2.71 TiB compressed. Common Crawl is not only archiving the pages it was allowed to read. It is archiving the instructions it was given and the answers it was refused.

That is unusual and it is worth pausing on, because it inverts the usual assumption about what a block accomplishes. A Disallow rule is not an absence of data. It is a fact about your site on a date, recorded in a public dataset with a permanent address, next to whatever status codes your server returned during the same window. If you changed your policy on 12 July 2026, both sides of that change are somewhere in this archive, and the archive is not going anywhere. Nothing about that is sinister; a web archive that discarded its own robots.txt fetches would be a worse archive and a less reproducible one. It is simply a different mental model from the one most site owners have.

It also means the specification governs more of this than people expect. Under RFC 9309, the Robots Exclusion Protocol, the status code your robots.txt returns is not a detail, it is the ruling: a 4xx means the file is unavailable and a crawler may access anything, a 5xx means unreachable and a crawler is obliged to assume complete disallow. We have written that up in full in why a 404 and a 503 on robots.txt are opposites, and this archive is the reason it is not academic. A robots.txt that answered 503 during a crawl window is a robots.txt that told every conforming crawler to stay away, and the non-200 dataset is where that answer now lives. If you have never checked what your file actually returns, rather than what it says, our robots.txt tester resolves it per crawler token, which is the level at which the question has an answer.

A Disallow written today applies to the next crawl, not the last one

Common Crawl documents the block clearly and without friction. The CCBot page gives the two lines, a User-agent of CCBot and a Disallow of slash, and the FAQ adds that Common Crawl will periodically continue to check whether the robots.txt file has been updated. Read that carefully. It is a statement about future crawls. It is not, and does not claim to be, a statement about archives already published to a public bucket with a stable path.

There is a second route, and Common Crawl publishes it rather than hiding it. On 17 September 2025 the foundation announced its opt-out registry, a published list of every legal opt-out request it has received, on the stated reasoning that downstream users of the data are not otherwise aware of the notices. The same post is where Common Crawl describes its own scale in its own words: a free, non-commercial archive of the public web that has operated for 18 years, cited in over 10,000 research papers, crawling several billion new pages each month from a frontier of over 1 trillion pages. It directs opt-out questions to info@commoncrawl.org, and it says plainly that it encourages publishers to use standard methods like robots.txt to control crawling. We have tested neither route and are reporting what the page says.

None of this makes blocking pointless. It makes the timing of a block the thing that matters, and timing is exactly what gets lost when AI crawler policy is discussed as a permanent setting rather than as a dated act. The same problem shows up at the CDN layer, where Cloudflare's managed robots.txt writes a Content-signal line and eight Disallow rules that take effect the moment they are generated and say nothing about the period before. It shows up again in the gap between the file you edit and the edge that answers, which is the subject of why two layers decide whether AI can read your site. A robots.txt is an instruction to the next request. Everything before it has already happened.

How CCBot fetches, in its operator's own words

The rest of the FAQ is a straightforwardly useful description of a crawler, and it is more specific than most vendors manage. CCBot is Nutch based and processes crawl candidates with Apache Hadoop and Map-Reduce. It checks robots.txt first and, if crawling is allowed, fetches with HTTP GET. It supports HTTP/1.1 and HTTP/2, the latter only over TLS, over both IPv4 and IPv6. It follows up to four consecutive HTTP redirects, or up to five when fetching robots.txt, which it attributes to RFC 9309. It supports conditional GET, and gzip, Brotli and ZStandard encodings. It uses an adaptive back-off that slows down when a server answers with 429 or 5xx. It honours the nofollow attribute, with the caveat that if it ever stopped doing so it would use those links only for discovery and would create no association with the source document.

It identifies itself as CCBot/2.0 with a link to the FAQ, and that string is the one recorded in our own registry. Common Crawl also warns on its documentation that it is aware of crawlers falsely identifying themselves as CCBot and recommends verifying the user agent string. That warning is the whole of the argument we have made before at length: a user agent is a claim, not an identity, and any per-crawler traffic count that rests on the header alone is counting assertions rather than crawlers.

The verification path Common Crawl offers is reverse DNS. Its documentation gives a worked example resolving 18.97.14.84 to a hostname under crawl.commoncrawl.org and back again, and it publishes CIDR blocks. The FAQ's list is labelled as of 2024-11-29 and holds one IPv6 block with no reverse DNS plus five IPv4 blocks: two /29s, one /30, one /32 and one further /29, which is 29 IPv4 addresses in total by arithmetic on the prefix lengths. The live list is published as JSON at index.commoncrawl.org/ccbot.json, written here as plain text because that host is not on this site's registered outbound list, and we did not fetch it: the endpoint refused our connection from this environment, so the only figure we are willing to state is the dated one on the FAQ page. For contrast with a fleet built for on-demand answering rather than scheduled archiving, the four OpenAI crawler IP lists we opened and counted on 29 July 2026 carried 37,328 IPv4 addresses for ChatGPT-User alone.

What a scan of your site can and cannot tell you about this

The honest boundary needs stating plainly, because it would be easy to write this post as though our product answered the question it raises. It does not. Lantad fetches a URL you give it, fetches the same URL in a browser, compares the two, evaluates your robots.txt against fifteen crawler tokens and reads your structured data. Every one of those is a measurement of your site as it is right now. None of them is a measurement of a corpus, and we cannot tell you whether any of your URLs appear in CC-MAIN-2026-30 or in any of the crawls before it. Anyone who tells you they can should be asked how, and the answer had better involve querying Common Crawl's own index rather than scanning your site.

There is a second boundary inside our own scoring, and it is a decision rather than a finding. Our registry classes CCBot's purpose as training, and only the search and user-agent purposes cost points in the Access component of the score. So a site that blocks CCBot and nothing else loses nothing on our report. That is deliberate: blocking a training crawler is a legitimate owner choice and a scanner that penalised it would be scoring a preference as a defect. It is a judgement we made, written next to the data in the registry rather than defended afterwards, and our methodology page is where the rest of those calls are set out. It is not evidence that blocking training crawlers is free of consequence, and we have no measurement either way.

What the scan does answer is the question this archive makes concrete: whether the words you want machines to have are in the response your server sends, or only in what a browser assembles afterwards. That is measurable from outside, on any URL, today, and it is the same answer for a crawler that archives on a schedule and one that fetches on demand. Six real storefront captures showed two stores on the same platform producing opposite outcomes on exactly that measurement, which is the clearest evidence we have that this is a build decision and not a platform verdict. If the answer comes back thin, the cause is stack specific rather than general, and the Next.js notes name a different mechanism from the React ones.

And if the measurement itself fails, we say so instead of printing a number, which is the policy behind why we withhold a grade. That policy is the reason this post ends where it does. We can tell you what Common Crawl published and what Common Crawl says its crawler does. We can tell you what your own pages return to something that runs no JavaScript. We cannot tell you what is in the archive, and we are not going to imply otherwise to make a cleaner ending.