Block AI training: 7 of 22 sites declaring ai-train=no blocked no crawler

How do you block AI training in robots.txt?

There is one mechanism that crawlers are specified to obey and one that asks. The mechanism is a group: a user-agent line naming a product token, followed by rules. RFC 9309, which defines the protocol, gives crawlers the matching and precedence rules for those groups and says nothing about anything else in the file. Writing Disallow: / under GPTBot is therefore a rule in the protocol's own terms, and it is the only edit in a robots.txt file that a compliant training crawler is specified to act on.

The asking half is the Content Signals Policy, a set of fields written into the same file. Its three categories are search, ai-input and ai-train, and the vocabulary is a permission rather than an access control: yes means the operator grants that use, no means they do not, and an absent signal means neither granted nor restricted. It sits in the file as a line reading Content-Signal followed by a comma separated list of those fields. Section 2.2.4 of RFC 9309 is what governs how a crawler treats it: crawlers MAY interpret other records that are not part of the robots.txt protocol, and may be lenient when doing so. May, not must. Our own parser records each of these lines as an unknown field and ignores it, which is the conservative reading and the same one the specification permits any crawler to take.

That leaves a site with two edits available and a choice about which to make. The rule half needs the token, and the tokens are published by the vendors: OpenAI's crawler documentation names GPTBot for training and OAI-SearchBot for the ChatGPT search index at its bots page, and the equivalent pages exist for the other operators, though not all of them are as complete, which is why we found that six of nine vendors publish exactly one crawler token. The declaration half needs no token at all, which is part of why it is easier to adopt and part of why it can end up alone. If you want to see which of the two your own file is relying on, the robots.txt tester evaluates a path against each token separately rather than reading the file as a statement of intent.

How many robots.txt files carry a content signal at all?

The frame is a committed file in this repository rather than a list assembled for this post, so every figure below can be recomputed against the same hostnames. It holds eight sectors of roughly equal size: government at 129 hosts, education at 130, healthcare at 123, news at 128, software at 130, ecommerce at 130, travel at 130 and finance at 127. It is weighted toward large organisations and it is not a random sample of the web, so a proportion measured here describes this frame and nothing wider. The scanner identifies itself the way it always does, which is documented on the bot page.

Fifty eight of the 1,027 requests never reached the site: the network path this scan ran from refused them before the connection was made, and they are excluded from every count rather than being recorded as a site response. Of the 969 that did reach an origin or its bot defence, 736 answered HTTP 200, 110 answered 403, 39 answered 503, 25 answered 404, and 44 returned nothing at all, 24 of those timing out at twelve seconds and 20 failing to connect. The 404 and the 503 are opposites under the protocol and worth keeping apart, which we set out in full when we established that a 404 on robots.txt allows every crawler and a 503 blocks them all. Thirty of the 736 sent a Content-Type of text/html, which is a 200 that a strict parser is entitled to discard.

Thirty eight of the 736 carry at least one Content-Signal line. That is 5.2 percent, and it is a larger number than we expected for a field that did not exist eighteen months ago, though the shape of the adoption explains it: 23 of the 38 are software company sites, five are education, four healthcare, two news, two travel, one government and one finance. Every one of the 38 declares both search and ai-train. Twenty four also declare ai-input, and only two of those declare ai-input=no, so the population is far more willing to refuse training than to refuse the retrieval that puts a page inside an answer. The same corpus produced the count behind GPTBot being disallowed at the root by 82 of 718 files, which is the rule-side comparison: the declaration is rarer than the block by a factor of about four. What this scan counts and what it refuses to infer is set out on the methodology page.

What do the 38 content signals actually declare?

Twenty two of the 38 declare ai-train=no and 16 declare ai-train=yes. The split is not evenly spread across the frame: all 16 of the yes files are software company sites, and they include several whose product is developer infrastructure, which is a coherent position for a business whose documentation being inside a model is a distribution channel rather than a loss.

Fifteen of the 38 carry a fourth field, written as use=reference. That field is not one of the three categories in the policy proper. Cloudflare describes it on the managed robots.txt documentation page, carrying Last updated Aug 3, 2026, as an optional extension it is testing, with the values immediate, reference and full, and the same page gives the managed default it writes as search=yes, ai-train=no, use=reference. Fourteen of those 15 files also carry the policy's full comment block, the one that opens by saying that as a condition of accessing this website you agree to abide by the following content signals. So most of the use field in this sample is one vendor's default rather than fifteen independent decisions, and the earlier finding that Cloudflare's managed robots.txt writes one signal line and eight Disallow rules is what these files look like in the wild.

Two of the 38 are written in a shape the documented format does not use. freshworks.com writes three separate Content-Signal lines, one each for ai-train, search and ai-input, where the format puts the fields in one comma separated list. aiven.io writes search=yes, ai-train=yes ai-input=yes, with no comma between the last two fields. Neither is a rule, so neither changes what any crawler may fetch, and that is the point worth taking from them: a syntax error in a field that nothing is required to parse produces no error anywhere. Cloudflare's own page notes that Google Search Console may report Syntax not understood for Content Signals, and says it has observed no impact on crawling rates as a result. The signal's reach depends entirely on adoption by the people it addresses, which is the same dependency that limits the RSL licence directive and, from the other direction, is why we checked whether Cloudflare's own crawl endpoints read the signals and found that one of ten did. None of this makes the declaration pointless. It makes it a legal and commercial artefact rather than a technical one, which is a reasonable thing for a site to publish as long as nobody in the building thinks it is doing the work of a Disallow line. Advice about generative engine optimization rarely draws that line clearly.

The seven files that declared ai-train=no and disallowed nobody

For each of the 22 files declaring ai-train=no we evaluated the nine training tokens in this scanner's registry against the site root, using the matcher that ships in the product rather than by reading the file. Those nine are GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot. Amazonbot is classed as training in our registry by our own judgment, since Amazon documents it as improving Alexa answers, which is a mixed role; the other eight are training crawlers on their vendors' own descriptions.

Fifteen of the 22 disallow eight or nine of the nine at the root. Three of the fifteen disallow all nine: unesco.org, acog.org and sendbird.com, each of which has added anthropic-ai to a list that does not otherwise include it. Seven disallow none of the nine: circleci.com, freshworks.com, snyk.io, twilio.com, vercel.com, visma.com and geico.com. Six of those seven name no training token anywhere in the file, so the Content-Signal line is the only thing in the file that addresses training at all, and it is the line with no force.

geico.com is the seventh and it is a different case, because it names them. The file writes Content-Signal: search=yes, ai-input=yes, ai-train=no inside the group headed by an asterisk, and then, further down the same file, writes a group for GPTBot whose only rule is Allow: /, and another for ClaudeBot, and for Google-Extended, CCBot, Anthropic-AI, Applebot-Extended, Bytespider, Amazonbot and meta-externalagent, each with Allow: / under it. Twenty five user-agent lines, no Disallow rule inside any of the training groups, and a declaration at the top saying training is not permitted. Read by a crawler that implements only the protocol, that file grants nine training crawlers the entire site. Read by a person, it says the opposite.

We are not calling that a mistake, because we cannot see the intent behind it and the two lines may be doing deliberately different jobs, one legal and one technical. What we can say is what a crawler receives, and this is the same gap in a different place from the one we measured yesterday, when 89 of 145 files naming an AI crawler ruled only on the whole site. A robots.txt file is a poor summary of a site's position on AI and a precise statement of its access rules, and reading it as the first is how AI visibility gets estimated wrongly in both directions. Seeing what a crawler actually receives from a page is a separate question again, which is what the GPTBot view renders.

What does Cloudflare's managed robots.txt actually write?

The fifteen files that pair ai-train=no with real Disallow rules are almost all one product's output, so it is worth reading what that product writes. The example on Cloudflare's managed robots.txt page, read on 13 September 2026, puts Allow: / in the wildcard group next to the Content-Signal line, then writes Disallow: / under eight named tokens: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. That is eight of the nine training tokens in our registry, the missing one being the legacy anthropic-ai, which is why fifteen of these files score eight rather than nine and why the three that reach nine had to add a line themselves.

The files in the wild carry one token the documented example does not. Fourteen of the 38, all of them managed files, also write a group for CloudflareBrowserRenderingCrawler, and 15 of the 736 files in the whole scan name it. That is not a discrepancy anybody should be alarmed by, since a managed list is meant to move, but it is the behaviour to plan around: the list is maintained by the vendor and a site that turns the feature on is delegating which crawlers it refuses, which we set out when we found that Cloudflare generates your robots.txt and its token list changes without you. Cloudflare is explicit about the limit of the whole mechanism on that page: robots.txt compliance is voluntary, the file expresses your preferences, and it does not prevent crawlers from accessing your content at a technical level. If you want to enforce blocking rather than request it, the same page points at AI Crawl Control, a product rather than a file.

That distinction is about to matter to more sites than have chosen it, because the defaults are moving: from 15 September 2026 new domains onboarding to Cloudflare get bot settings under which crawlers classified as Training or as Agent are blocked on pages that carry ads, which we read from the changelog and wrote up as a default that depends on whether the page carries ads. A site that has been relying on a declaration will find its actual access position decided somewhere else, and neither the declaration nor the robots file will record it. The tokens themselves, and which vendor documents which, are collected in the AI crawler reference.

What to check in your own robots.txt this week

Four checks, in the order that tells you the most for the least work, and none of them needs a tool to start.

First, read the file for the two halves separately. If a Content-Signal line is the only place training appears, then whatever the line says, every training crawler currently has whatever access your wildcard group gives. Second, evaluate each token rather than the file. A group headed by a token replaces the wildcard group for that crawler instead of adding to it, so a restrictive wildcard group and a permissive named group is a common and easily missed combination, and it is the geico pattern above in a milder form. Third, decide which of the two halves you actually want, because they answer different questions: the rule decides access, the declaration reserves a right and may matter in a jurisdiction where a reservation has legal weight, which is the ground the EU AI Act and its Code of Practice sit on. Wanting both is normal. Assuming one implies the other is the failure.

Fourth, check what you have said to the crawlers that are not training crawlers, because that is where most sites are accidentally strict. Of the 38 files carrying a signal, only two declare ai-input=no, so almost everybody in this sample is content to be read for a live answer. If your rules are stricter than your declaration on that point, you have blocked a citation without meaning to, and the guides for being cited by ChatGPT are written for exactly that gap. Our running measurements against this frame, and what they do and do not support, are collected on the research page.