Cloudflare's Content-signal line asks, and the Disallow lines under it block

What Cloudflare's managed robots.txt actually writes into your file

Cloudflare published the Content Signals Policy on 24 September 2025 and released it under a CC0 licence, which means anyone can put the same text in their own file without asking. The announcement states that Cloudflare customers had already turned on the managed robots.txt feature for over 3.8 million domains at the time it went out, so the deployment surface existed before the policy did.

The policy defines three signals, and the definitions are narrower than the names suggest. The signal called search covers building a search index and providing search results, given as returning hyperlinks and short excerpts, and the text adds explicitly that search does not include providing AI generated search summaries. The signal called ai-input covers inputting content into one or more AI models, with retrieval augmented generation and grounding as the worked examples. The signal called ai-train covers training or fine tuning AI models. Each takes yes or no, and the policy states that where an operator includes no signal for a use, the operator neither grants nor restricts permission via content signal for that use. Absence is not a no.

Cloudflare's managed robots.txt documentation, last updated 1 July 2026, prints the file the feature generates, and that file carries a fourth field that is not among the three signals the policy defines. The line reads Content-signal: search=yes, ai-train=no, use=reference, and the documentation describes use=reference as added in line with the existing default of search=yes and ai-train=no. The full policy text also lives at contentsignals.org, which is not on this site's registered outbound host list, so it is named here rather than linked. Note the spelling too: the September 2025 announcement writes Content-Signal with a capital S and the generated file writes Content-signal with a lowercase one. That is harmless, because robots.txt field names match case insensitively, but it is worth knowing before you search your own file for the string.

Above the rules sits a preamble in capitals asserting that any restrictions expressed via content signals are express reservations of rights under Article 4 of Directive (EU) 2019/790 on copyright and related rights in the digital single market. That is a legal claim made inside a comment, and it is Cloudflare's claim rather than a court's or the Commission's. Article 4(3) is the provision that makes the text and data mining exception conditional on a rightholder reserving the use in an appropriate manner, such as machine readable means, and we have written separately about what that Article obliges and what the AI Act does with it. Whether a comment block in robots.txt satisfies it is an open question, and this post does not answer it.

Below the preamble sit the rules, and the rules are ordinary robots.txt.

What a robots.txt parser is obliged to do with a Content-signal line

RFC 9309, the Robots Exclusion Protocol, published in September 2022, defines the records a crawler must handle and then says what to do with everything else. The operative sentence is permissive in both directions: crawlers MAY interpret other records that are not part of the robots.txt protocol, with Sitemaps given as the example, and crawlers MAY be lenient when interpreting other records. Then comes the constraint, which is the only obligation in the area: parsing of other records MUST NOT interfere with the parsing of the explicitly defined records.

A Content-signal line is an other record. Nothing in the protocol forbids a crawler from reading it, and nothing in the protocol obliges one to. The same sentence that lets Sitemap work is the sentence that lets Content-signal be skipped, and it moves the whole question off the specification and onto what each individual crawler operator has chosen to implement. That is a familiar shape for anyone who has looked closely at this file before: RFC 9309 also decides, for instance, that a 404 on robots.txt allows every crawler while a 503 blocks them all, which is a rule almost nobody expects and every compliant crawler follows.

So look at what a crawler operator documents. Google's robots.txt documentation, last updated 8 July 2026, lists the fields Google supports as user-agent, allow, disallow and sitemap, and names crawl-delay on the same page as an example of a field that is not supported. On unrecognised content it is explicit, stating that rules other than allow, disallow and user-agent are ignored by the robots.txt parser. Content-signal is not on the supported list and is not one of those three rules. That is Google describing its own parser in the present tense, on a page it dated three weeks before this post.

None of this is a criticism of Cloudflare, because Cloudflare says the same thing in the announcement itself: content signals express preferences, they are not technical countermeasures against scraping, and some companies might simply ignore them. A preference labelled honestly as a preference is not a broken feature. It is a different kind of thing from a Disallow rule, and the failure mode worth naming is a site owner who reads two adjacent lines in one file and assumes they carry the same force.

There is a useful contrast with the standards track attempt at the same problem. The IETF AI preferences working group has a live vocabulary and no live mechanism for attaching it to anything, which is what we found reading the state of its drafts in July. Content Signals is the mirror image of that: a deployed syntax sitting in a large number of files with no standards status behind it. Neither shape produces a crawler obligation, and site owners tend to hear about the deployed one and assume it must therefore be the settled one.

What happened when we ran Cloudflare's file through our own parser

We copied the example file out of Cloudflare's managed robots.txt documentation as published, and on 30 July 2026 ran it through parseRobotsTxt in core/src/robots.ts, which is the same function every Lantad scan uses. That is the whole experiment, and its limits are worth stating before its result. It measures our parser against a documented file. It does not measure a live site, it does not measure how many domains carry the file, and it does not measure what any crawler does on receiving one.

The parser returned one parse note, and this is it: line 29: unknown field content-signal, ignored. Line 29 is where the Content-signal line sits in that file. It found nine groups and zero sitemap directives, which is the correct reading of a file holding one wildcard group, eight named groups and no Sitemap line.

The word ignored in that note is doing exactly what RFC 9309 asks and nothing beyond it. Our parser carries a case for user-agent, allow, disallow, sitemap and crawl-delay, and a default branch that records the unrecognised field name and moves on. Content-signal lands in the default branch. Every rule verdict the parser then produced is identical to the verdict it would have produced had the line been deleted from the file first.

That is the inconvenient half, so it is better said plainly than buried: Lantad does not read content signals, does not score them, and a site that adds one will see no change in its report. We have not built support for the field, and the reason is the section above. A line that no crawler operator documents parsing is not evidence about whether a crawler can read your site, which is the only question our methodology is built to answer.

There is a second reading of that parse note worth having. It is a note rather than an error, and it is surfaced rather than swallowed. Our robots.txt tester parses your live file through the same code path, so if you have a Content-signal line it will tell you that it saw the line and did not use it. That is a more useful answer than silence in either direction. The failure we care most about avoiding is a tool that meets a line it does not understand and prints a confident verdict anyway, which is the same instinct behind refusing to grade a page we could not measure.

Which crawlers the file actually stops, and which it lets through

The Disallow rules are the half that works, and they are worth counting, because what a file blocks is a measurement anybody can reproduce from published inputs.

Lantad's registry holds 15 robots.txt product tokens across nine vendors, and it is published as a public crawler directory rather than kept internal. Evaluated against Cloudflare's example file on 30 July 2026, eight of those fifteen came back blocked at the site root and seven came back allowed.

The eight blocked are the eight names Cloudflare writes: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. Each matched its own group and its own Disallow rule rather than falling through to the wildcard, which is what you want to confirm rather than assume, because robots.txt group selection is a most specific match and not a union of everything that could apply.

Six of the seven allowed share a property. OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User are the tokens our registry classes as search or user fetch rather than training, and they fall to the wildcard group's Allow rule. That is the file behaving exactly as its own Content-signal line describes it. The pair search=yes and ai-train=no is what the rule block implements, in the one syntax a crawler is obliged to read, which makes the signal a readable summary of the file rather than an instruction the file depends on. OpenAI's crawler documentation draws the same training against search line across its own tokens, so a file that stops GPTBot and leaves OAI-SearchBot alone is a coherent position rather than an oversight, and it is roughly the position most sites pursuing AI visibility say they want.

The seventh allowed token is the one worth flagging. Our registry classes anthropic-ai as a training token, and it is not named in Cloudflare's block list, so it came back allowed under the wildcard. In practice this matters little, because anthropic-ai is a deprecated robots.txt token that sends no request user agent of its own, one of the tokens that will never appear in your logs. It is a difference between two maintained lists rather than a hole in anyone's defence. It is also a fair illustration of why per token evaluation earns its keep: two organisations keeping lists of AI crawler tokens produced lists that do not match, and the only way to discover that is to evaluate a real file token by token instead of reading it as a mood.

Anyone weighing this file up should read it as a visibility decision and not only as a licensing one. Blocking the training crawlers is defensible and it is the default here, but it is a choice with a consequence, and if the goal is to be quoted in ChatGPT then what the search side of that fleet is allowed to fetch is the thing that decides it.

What to check in your own robots.txt

The check is short and you can do most of it without a tool. Fetch your own robots.txt, see whether the Cloudflare managed block is present, and then read the two halves separately rather than as one statement.

First, the rules. Count the groups and read them per token rather than as a whole, because a file that reads as restrictive at a glance can be permissive to the specific crawler you care about, and the reverse happens just as often. Our robots.txt tester prints a row per token for exactly that reason. The classification problem underneath it, turning a token or a user agent string into a crawler identity, is harder than it looks, and we wrote up why a user agent is a claim rather than an identity on its own.

Second, the signal. If a Content-signal line is present, treat it as a statement of your position rather than as enforcement. It costs nothing to carry, it may matter legally, and today no major crawler operator documents acting on it. Adding one is reasonable. Assuming it does the work of a Disallow rule is not, and that assumption is the specific error this post exists to prevent.

Third, the layer above both. The file is not Cloudflare's only lever, and an edge rule can allow or block a request regardless of what the file says, which is what we found when the file and the edge disagreed. Cloudflare's own AI Crawl Control documentation, last updated 23 April 2026, describes tracking whether a robots.txt file contains content signals as a reporting feature, which is a fair description of what the field is for at present: something to observe rather than something to enforce.

The general shape here is one this blog keeps meeting. A file gains a new line, adoption follows quickly because a platform can write that line at scale, and the measured effect on crawler behaviour stays at zero until a crawler operator publishes that it reads the line. That was the finding on llms.txt, where Ahrefs measured 137,210 domains in May 2026 and found 97 percent of valid llms.txt files were never fetched at all, and it is the honest state of Content-signal today. The one real difference is that Content-signal ships alongside eight Disallow rules that do work. Unlike llms.txt, the file it lives in is not doing nothing. It is just not the signal doing it.