# CCBot robots.txt: 140 of 1,013 files named it, and 15 that blocked GPTBot left it allowed

> Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 17 October 2026, as LantadBot with no JavaScript executed. 1,013 parsed into at least one user-agent group. 140 named CCBot and 81 disallowed it at the root. 15 files that closed the root to GPTBot left CCBot allowed, and not one of those 15 named CCBot anywhere in the file.

- Canonical page: https://lantad.co/blog/ccbot-robots-txt-140-of-1013-named-it
- This file: https://lantad.co/blog/ccbot-robots-txt-140-of-1013-named-it.md
- Last substantive update: 2026-10-17

## Key facts

- **Published:** 2026-10-17
- **Category:** Findings
- **Author:** Lantad
- **Length:** 3410 words
- **Takeaway 1:** A CCBot robots.txt rule reaches the archive rather than an answer engine: 140 of the 1,013 robots.txt files Lantad parsed on 17 October 2026 named the CCBot token and 81 disallowed it at the root, against 173 naming GPTBot and 84 disallowing it.
- **Takeaway 2:** 15 of the 1,013 files disallowed GPTBot at the root and left CCBot allowed, and none of those 15 named CCBot anywhere in the file. Among them are aljazeera.com, cbsnews.com, airbnb.com and alibaba.com.
- **Takeaway 3:** Common Crawl's own FAQ, read on 17 October 2026, states that it obeys the Crawl-delay parameter. Zero of the 1,013 files set a Crawl-delay inside a group naming CCBot, although 152 files set one somewhere and 305 Crawl-delay lines appear across the corpus.
- **Takeaway 4:** Naming CCBot is not refusing it. 35 of the 140 files naming CCBot are Squarespace-shaped files that list it in one group alongside 28 other named tokens and the wildcard, and none of those 35 disallowed it at the root.
- **Takeaway 5:** Blocking is concentrated in news, as with every citation crawler this blog has counted: 37 of the 65 news hostnames whose file parsed disallowed CCBot at the root, against 44 across the other 948 files.

## Summary

CCBot is the oldest name in this argument and the one furthest from the chatbot. Common Crawl is a nonprofit archive that publishes monthly web crawls for anyone to download, so a CCBot robots.txt rule does not govern an answer engine the way a GPTBot rule does. It governs the dataset upstream of several of them. That distance is what makes the token worth counting rather than assuming: a site deciding how it feels about [AI crawlers](https://lantad.co/glossary/ai-crawler) can refuse the engines it has heard of and leave the archive that feeds them wide open, and nothing in the file will point that out.

On 17 October 2026 this scanner requested the robots.txt of all 1,419 hostnames in the two committed corpus seed files in this repository, as LantadBot/1.0 with redirects followed and no JavaScript executed, and parsed the 1,013 that resolved into at least one user-agent group. Separately, [Common Crawl's CCBot page](https://commoncrawl.org/ccbot) and its [FAQ](https://commoncrawl.org/faq) were opened and read the same day. What follows is how many of those files name CCBot, how many refuse it, which sites refused one crawler and not the other, and the one control Common Crawl documents honouring that not a single file in this corpus pointed at it. The token list behind the measurement is the one published on the [AI crawler reference](https://lantad.co/tools/ai-crawlers).

## What does a CCBot robots.txt rule control?

Common Crawl documents CCBot on two pages, and together they describe a crawler that is unusually well behaved and unusually far from the product a reader is probably worried about. The FAQ states that CCBot is an automated crawler, checking first the robots.txt, and if crawling a page is allowed, fetches pages using HTTP GET requests. It states that CCBot follows up to four consecutive HTTP redirects, or up to five when fetching robots.txt in line with RFC 9309. It states that Common Crawl obeys the Crawl-delay parameter, and that it currently honors the nofollow attribute as it applies to links embedded on a site. The CCBot page gives the blocking recipe directly, a User-agent line naming CCBot followed by Disallow slash, and the FAQ adds that Common Crawl will periodically continue to check if the robots.txt file has been updated, and that a site may also ask to be added to its opt-out registry.

That list is longer than any answer engine publishes, and it is worth saying plainly that Common Crawl documents more cooperation than the vendors this blog usually counts. The catch is what a rule reaches. Common Crawl does not answer questions. It publishes an archive, and what that archive is used for is decided by whoever downloads it. The best documented example is still the clearest: Table 2.2 of [the GPT-3 paper](https://arxiv.org/abs/2005.14165), submitted on 28 May 2020, lists Common Crawl (filtered) at 410 billion tokens and 60 percent of that model's training mix, the largest single source in the table. That is one model from 2020 and not a claim about anything trained since, because the training mixes of current models are not published. It is enough to establish the shape: the archive sits upstream, and a rule written for it is a rule about supply rather than about any one engine.

The second consequence of being an archive is that a rule written today is a rule about future crawls only. Common Crawl's FAQ describes blocking as stopping the crawler, and says nothing about data already collected. This blog has read both halves of that: Common Crawl [published CC-MAIN-2026-30 on 28 July 2026 with 2.14 billion pages](https://lantad.co/blog/common-crawl-july-archive-no-javascript) collected over about three weeks, and a paper by three Common Crawl Foundation authors [put the persistent core of the crawl near 40 percent of domains](https://lantad.co/blog/common-crawl-core-and-shell-persistence) across 51 monthly crawls. A Disallow added in October 2026 governs the November crawl. It does not reach back into the ones already published, which is the difference between an opt out and a deletion, and it is the part most likely to be misread by somebody writing the rule for the first time.

## What was measured on 17 October 2026, and what was not

The population is the 1,419 hostnames in the two committed corpus seed files in this repository: worker/seeds/corpus-seeds-platform.json with 392 hostnames across ten platform strata, and worker/seeds/corpus-seeds-industry.json with 1,027 across eight industry strata. Each hostname was asked once for https://hostname/robots.txt, as LantadBot/1.0, from one network location, with redirects followed and a fifteen second timeout. The user agent is the one described on the [bot page](https://lantad.co/bot), and it was never altered to impersonate a crawler belonging to anybody else.

The funnel decides every proportion below, so here it is in full. 1,092 hostnames answered HTTP 200 and 8 answered 202, giving 1,100 in the 2xx range. 39 of those 1,100 served an HTML page at the robots.txt address, a defect this blog has [counted across the same corpus before](https://lantad.co/blog/robots-txt-validator-232-of-1059-carried-a-defect), and they were excluded. 48 more parsed into no user-agent group at all, usually a file holding nothing but a Sitemap line or a comment, and a file with no group cannot name a token, so those went too. That leaves 1,013, the denominator for every file count in this post. Of the remainder, 178 hostnames answered 403, 51 answered 404, 36 answered 503, 4 answered 429, 3 answered 401, three more returned a 406, a 418 and a 498, and 44 requests failed before any status arrived, 22 of them by timeout. A 403 on the robots file is a firmer refusal than anything the file could have contained, the same asymmetry behind [92 of 1,056 sites refusing GPTBot the file](https://lantad.co/blog/can-gptbot-read-your-robots-txt-92-of-1056).

Parsing and evaluation used parseRobotsTxt and evaluateRobots from core/src/robots.ts, the code that ships in the product rather than a script written for this post. A file counts as naming a token when a user-agent line lowercases to that exact string. A root verdict is the answer evaluateRobots returns for the path slash, which is the strongest form of the question, since a file can leave the root open and still close most of what matters. Group selection follows [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), whose section 2.2.2 requires that the most specific match be used and defines it as the match with the most octets. The approach is the one set out on the [methodology page](https://lantad.co/methodology).

Three limits apply to all of it. No Common Crawl request was observed: there are no server logs here, so every statement about CCBot's behaviour restates Common Crawl's documentation and nothing else. One URL per hostname was requested, at the apex, so none of this covers subdomains, where [73 of 277 gave an AI crawler a different answer from their apex](https://lantad.co/blog/robots-txt-subdomains-73-of-277-differ). And these 1,419 front doors come from two committed sampling frames rather than from the web, so no proportion here generalises beyond them.

## 140 files named CCBot, and 81 closed the root to it

140 of the 1,013 files named CCBot and 81 disallowed it at the root. 74 of those 81 were refused through a group naming CCBot directly, and the other 7 through a wildcard group that never mentioned it, the quiet failure mode this blog measured when [559 of 581 pages were lost to a rule that never named the crawler](https://lantad.co/blog/blocked-by-robots-txt-559-of-581-by-a-wildcard). Three spellings appeared across 142 user-agent lines: CCBot on 138, CCbot on 2 and a lowercase ccbot on 2. All three bind, because RFC 9309 matches the token case insensitively, which this corpus confirmed from the other direction three days ago when [84 mis-cased crawler tokens all bound anyway](https://lantad.co/blog/is-robots-txt-case-sensitive-84-tokens-bound-9-pages-open). Two files wrote the name twice: bloomberg.com has two CCBot groups and forbes.com has one CCBot and one CCbot.

Set against the other tokens in the same files, CCBot sits in the middle of the pack on naming and near the top on refusal. 173 files named GPTBot and 84 disallowed it, 161 named ClaudeBot with 76 refusals, 145 named Google-Extended with 62, 129 named Bytespider with 77, 120 named Meta-ExternalAgent with 63, 119 named the retired anthropic-ai token with 62, 114 named Applebot-Extended with 59, 111 named Amazonbot with 57, and 102 named PerplexityBot with 45. CCBot is named by fewer files than GPTBot and refused by almost as many, which is the ratio worth noticing: of the files that bother to name it, a far larger share say no. That is consistent with the two training crawlers counted most recently here, [130 of 1,077 files naming Bytespider](https://lantad.co/blog/bytespider-robots-txt-130-of-1077-named-it) and [108 of 1,016 naming Amazonbot](https://lantad.co/blog/amazonbot-robots-txt-108-of-1016-named-it).

Where the 81 sit is the familiar story. 37 of them are news: of the 65 hostnames in the news stratum whose file parsed, 38 named CCBot and 37 disallowed it at the root, a refusal rate no other stratum approaches. The other 44 are spread thinly, 9 across 102 education files, 9 across 81 ecommerce, 8 across 86 healthcare, 6 across 75 government and 3 across 120 SaaS. One finance file of 90 refused it and one travel file of 76 did. The concentration matches what this blog found counting blocking by category, where [42 of the 77 files blocking a citation crawler were news sites](https://lantad.co/blog/ai-crawler-statistics-77-of-706-blocked-42-were-news). A refusal of Common Crawl in this corpus is close to a synonym for a newsroom, and a reader can run the same per-token evaluation against their own file with the [robots.txt tester](https://lantad.co/tools/robots-txt-tester).

## 15 files blocked GPTBot and left CCBot allowed

Treating the two tokens as one question splits the 1,013 files four ways. 917 refuse neither CCBot nor GPTBot at the root. 69 refuse both. 12 refuse CCBot and allow GPTBot. 15 refuse GPTBot and allow CCBot. The last group is the one worth reading slowly, because not one of those 15 files names CCBot anywhere: aljazeera.com, cbsnews.com, motherjones.com, irishtimes.com, semafor.com and 404media.co among the news sites, alibaba.com and mercadolivre.com.br in ecommerce, airbnb.com in travel, bancolombia.com in finance, dukehealth.org and rivm.nl in healthcare, uct.ac.za in education, and the developer sites ryanmulligan.dev and sia.codes. Every one of them wrote Disallow slash for GPTBot. None of them considered CCBot and allowed it. They wrote a rule for the name they had heard of.

On Common Crawl's own account that gap is not symmetrical with the one it resembles. A GPTBot rule, by OpenAI's documentation, governs OpenAI's own collection. It does not reach an archive that anybody can download, and the GPT-3 paper is the published evidence that an archive of this kind has supplied the majority of at least one model's training mix. So a site in this group has declined the crawler whose operator it can name and left open the supply route whose operator it cannot. Whether that matters depends entirely on what the site was trying to achieve, which is the honest end of this finding rather than a hedge: a publisher protecting its traffic from a chatbot has done exactly what it meant to, while a publisher trying to keep its words out of model training has not, and the file looks identical in both cases.

The 12 that run the comparison the other way are the more deliberate group, and most of them are older rules. washingtonpost.com, axios.com, sfchronicle.com and aftenposten.no refuse CCBot by name while leaving GPTBot allowed at the root; so do who.int and illinois.gov, the healthcare sites bhf.org.uk and scripps.org, and the retailers warbyparker.com, jumia.com.ng, mediamarkt.de and gmarket.co.kr. Ten of the twelve do not name GPTBot at all, which fits a CCBot stanza written before the current crawler names existed and never revisited, the same ageing this blog measured in [2,209 tokens across 1,004 files](https://lantad.co/blog/crawler-user-agent-2209-tokens-in-1004-files). Across all four groups the practical point is the one the product exists to make: the question is not whether a site blocks AI crawlers but which of them, and a file that answers for one token has not answered for the rest. That is also why [seven of 22 sites that set out to block AI training blocked nobody](https://lantad.co/blog/block-ai-training-seven-of-22-blocked-nobody).

## Naming CCBot is not refusing it, and nobody used the limit Common Crawl documents

The gap between 140 files naming CCBot and 81 refusing it is not 59 sites that weighed the archive and waved it through. 35 of the 140 are Squarespace-shaped files carrying an identical generated stanza, and that stanza is a single group whose user-agent lines list CCBot alongside 28 other named tokens including GPTBot, ClaudeBot, Bytespider, Google-Extended and Applebot-Extended, and then the wildcard as well. Because the wildcard heads the same group, the rules beneath it are the ordinary housekeeping a platform writes for every crawler: Disallow /config, Disallow /search, Disallow /account, a handful of query-string patterns. There is no root Disallow, so none of the 35 refuses CCBot, and naming it carries no AI-specific intent whatsoever. This blog has watched the same platform from the other side, where [61 of 66 Squarespace sites named an AI crawler and blocked none of them](https://lantad.co/blog/squarespace-ai-crawlers-61-of-66-named-not-blocked). Strip that stratum out and 105 files name CCBot by a decision somebody made.

The reverse shape appears too, and it is where the refusals get commercial. 12 of the files that close the root to CCBot reopen specific paths to it, 124 Allow rules between them. canva.com carries 91 of those on its own, in a group heading GPTBot, ClaudeBot, CCBot, Applebot-Extended, anthropic-ai and Claude-Web, carrying 91 Allow rules and a Disallow slash at line 146; the root still evaluates to blocked, because under the longest-match rule a bare slash loses only to a longer matching pattern. washingtonpost.com writes the pattern in four lines at 167 to 170, User-agent CCBot, Disallow slash, then Allow /creativegroup/ and Allow /advertising/. ebay.com adds 11 Allow rules and nbcnews.com 9. The archive is refused the journalism and offered the marketing, which is the same split this corpus showed for the citation crawlers.

Then there is the control nobody used. Common Crawl documents that it obeys Crawl-delay, and Crawl-delay is the setting a site reaches for when the objection is load rather than principle. Across the 1,013 files there are 305 Crawl-delay lines in 152 files. Not one of them sits in a group naming CCBot. 59 files set a wildcard Crawl-delay and have no CCBot group, so the delay does reach CCBot, at values from 1 to 30 seconds with a median of 10. The sharper case is the 8 files that name CCBot in its own group and set a Crawl-delay under the wildcard instead: villagedentaldtc.com, congress.gov, bhf.org.uk, healthline.com, scripps.org, chronicle.com, scientificamerican.com and warbyparker.com. Group selection sends CCBot to its own group, so the rate limit those eight wrote is the one crawler-specific instruction CCBot will never see, and the reason [157 of 1,014 files setting a Crawl-delay](https://lantad.co/blog/crawl-delay-robots-txt-157-of-1014-set-one) is a weaker number than it sounds. A site that wanted to throttle the archive rather than refuse it wrote the instruction into the one group the archive will not read.

## Questions and answers

**What is CCBot in robots.txt?**

CCBot is the crawler operated by Common Crawl, a nonprofit that publishes monthly web archives. Its CCBot page gives the full agent string as CCBot/2.0 (https://commoncrawl.org/faq/) and says a site refuses it with a User-agent line naming CCBot followed by Disallow slash. 140 of the 1,013 robots.txt files Lantad parsed on 17 October 2026 named the token and 81 disallowed it at the root.

**Does blocking CCBot remove my site from existing Common Crawl archives?**

No. Common Crawl's FAQ describes a robots.txt rule as stopping the crawler and says nothing about data already collected, so a rule added today governs future crawls only. Common Crawl publishes its archives monthly and they stay published, which makes a CCBot rule an opt out rather than a deletion.

**Does blocking GPTBot also block Common Crawl?**

No, they are separate tokens and a rule for one does not bind the other. 15 of the 1,013 files Lantad parsed on 17 October 2026 disallowed GPTBot at the root and left CCBot allowed, and not one of those 15 named CCBot anywhere in the file.

**Can I slow CCBot down instead of blocking it?**

Common Crawl's FAQ states that it obeys the Crawl-delay parameter for robots.txt, which few AI crawler operators document. No file in this corpus used it on CCBot: zero of the 1,013 files set a Crawl-delay inside a group naming CCBot, although 152 files set one somewhere for somebody.

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot