Content signals robots.txt: one of ten Cloudflare endpoints reads it

What Cloudflare changed on 31 August 2026

The crawl endpoint is not new. Cloudflare's Browser Run changelog records it arriving in beta on 10 March 2026, described there as scraping content from a starting URL and following links across the site. The product it belongs to was called Browser Rendering until 15 April 2026, when Cloudflare's changelog announced the rename to Browser Run; the REST path still carries the old name, which is worth knowing if you go looking for it in your own code.

What changed on 31 August 2026 is a rule about who may start a job. Cloudflare's changelog entry for that date states that the crawl endpoint now respects the use directive of the Content Signals standard, that a caller declares its intended level through a new contentUse parameter whose allowed values are reference and full from least to most permissive, that the default is full, and that a crawl request is rejected with a 400 error if the target site's robots.txt sets a more restrictive level. The crawl endpoint documentation, which carries a last updated date of 31 August 2026, puts the rule in one sentence: a crawl is rejected when your declared contentUse level is more permissive than the site's declared use level.

There is a second rule alongside it, covering the three purpose signals rather than the use level. That documentation states that if a target site sets any of those content signals to no, the crawl request will be rejected at initiation with a 400 Bad Request error unless you explicitly narrow your declared purposes. Rejected at initiation is the part to notice. This is not a per-URL skip buried in a results file. The job does not start.

The same page records that the endpoint respects the directives of robots.txt files, including crawl-delay, and that URLs blocked by robots.txt come back in the results with a status of disallowed. Those two behaviours are ordinary and have been available since the endpoint shipped. The Content Signals rules are the new part, and they are different in kind: a Disallow line stops a fetch, while a Content-signal line now stops a job on the basis of what the caller said it was going to do afterwards.

What can a content signals robots.txt line make a crawler do?

Six values, on two axes. Cloudflare's crawl documentation sets them out and gives each a definition, which is worth quoting because the definitions are narrower than the labels suggest.

The three purpose signals are search, ai-input and ai-train. Cloudflare defines search as building a search index and providing search results with links and excerpts, ai-input as inputting content into AI models at query time, giving retrieval-augmented generation and grounding as the examples, and ai-train as training or fine-tuning AI models. That split is the same distinction this blog has kept returning to, because it is the one that decides whether blocking a bot costs you the citations an answer engine hands out or only the training corpus. The ai-input row is the one most site owners set wrongly, since it covers the retrieval that puts a page inside an answer today rather than a model weight next year.

The use directive is the second axis and carries three levels, given least to most permissive. Cloudflare defines immediate as ephemeral, single-response use where content is not retained, reference as content may be retained, indexed, or cited, and full as unrestricted use, including AI training. A site expresses a ceiling on that scale, and the caller declares where it intends to sit.

None of these are new fields in robots.txt. They are values carried on a Content-signal record, and RFC 9309, published in September 2022, is explicit about the status of such a record. The robots exclusion protocol says at section 2.2.4 that crawlers MAY interpret other records that are not part of the protocol, and that parsing them MUST NOT interfere with the parsing of the records section 2 defines. Google's own robots.txt reference lists four supported fields and states that anything else is ignored by its parser. So a Content-signal line remains, at the level of the standard, a comment that happens to be machine-shaped. What changed is that one commercial crawling service has chosen to read it, which is exactly the permission the RFC grants and nothing more.

That distinction matters for anyone about to edit a file. Adding a use directive does not make your robots.txt stricter in any way a general crawler is obliged to notice. It adds a line that a specific set of clients have volunteered to honour, which is a real thing and a much smaller thing. If you want the part that binds, that is still the Disallow line, and it is still worth checking what your file actually says per crawler rather than assuming.

The defaults declare the maximum on both axes

Read the two parameter definitions next to each other and one property falls out. Cloudflare documents crawlPurposes as defaulting to all three purposes, and contentUse as defaulting to full, which its own scale calls the most permissive level. A developer who calls the endpoint with nothing but a URL has therefore declared the widest possible intent on both axes, without typing a word to that effect.

That inverts the usual shape of a default. The untouched call is the one most likely to be refused, because it collides with any restriction a site has expressed. A caller who wants jobs to succeed has an incentive to narrow, and the documentation says so directly: the purpose rejection applies unless you explicitly narrow your declared purposes. Narrowing is free, takes one array, and is checked against nothing.

There is a second asymmetry, and it is arithmetic on the two published lists rather than anything Cloudflare states. The use directive defines three levels and contentUse accepts two of them, reference and full. immediate is not among the values a caller can send. Since the rejection rule fires whenever the declared level is more permissive than the site's, and both declarable values sit above immediate on Cloudflare's own ordering, a site publishing use=immediate cannot be crawled by this endpoint under any parameter the endpoint accepts. Whether that is intended or simply a gap between a specification with three levels and a parameter with two, the published lists do not say, and we are not going to guess at it.

None of this makes the feature pointless. It makes it a declaration mechanism rather than an access control, which is a distinction this blog has had to draw before: a licence directive in robots.txt changes no access by its own admission, and a pay per crawl price is carried on a status code the specification leaves undefined. The pattern repeats because the layer where these controls live was designed for one question, whether to fetch, and is being asked a different one, what you may do with what you fetched.

Nine of the ten Browser Run endpoints name none of it

The rule applies to one endpoint. That is not a criticism, it is scope, and scope is the thing a reader most needs to know before deciding whether any of this touches them. So we checked it rather than assuming.

Cloudflare documents ten quick actions in Browser Run: content, screenshot, pdf, markdown, snapshot, accessibilityTree, scrape, json, links and crawl. On 3 September 2026 we opened all ten documentation pages and searched each for the phrases Content Signals, contentUse, crawlPurposes and robots.txt. The crawl page carries all four. The other nine carry none of them, and none of the nine mentions robots.txt at all. Their last updated dates run from 28 May 2026 for content, screenshot, pdf and markdown, through 7 July 2026 for links, json, snapshot and accessibilityTree, to 17 July 2026 for scrape. The crawl page is dated 31 August 2026, the day of the change.

The nine are single-page actions rather than site crawlers, which explains part of it: a robots.txt file governs automated traversal, and fetching one page a human named is a different act. But several of them do the thing the Content Signals vocabulary is written about. The markdown endpoint extracts prose in the format AI crawlers were observed fetching by URL rather than by content negotiation. The json endpoint runs a model over the page. Those are ai-input shaped operations by Cloudflare's own definition, performed by a documented product, with no declaration asked for and no signal read.

This is also the point at which we have to say what it means for this scanner, because the honest answer is nothing. Read out of this repository on 3 September 2026, Lantad's production renderer calls the content and screenshot endpoints of exactly this API and never the crawl endpoint, which is what you would expect of a tool that fetches the single URL a person pasted. So the 31 August change alters no scan we run, and our own crawler conduct policy is unaffected by it. Saying so is cheaper than letting a reader assume the product they are reading about was in scope.

Cloudflare calls it trust-based, and that is the whole mechanism

The documentation does not oversell this, and the sentence it uses is the most important one on the page. Content Signals are trust-based, it says, and by setting crawlPurposes and contentUse you are declaring to the site owner how you intend to use the crawled content.

Nothing verifies the declaration. A caller that intends to train a model may send a crawlPurposes array containing only search, pass every check, and receive the pages. The check is on the string, not on the use. That is not a flaw Cloudflare has hidden; it is the property the word trust-based names, and it is the same property that runs through every control in this area. A bot declares three things about itself and can prove one, which is identity. Intent has never been provable from the outside and is not provable here.

What the mechanism does achieve is worth stating plainly rather than dismissing. It moves the moment of refusal from the site to the crawler's own vendor. Before 31 August 2026 a site's Content-signal line reached a general purpose crawling API and was ignored, in the way our 30 July 2026 parse found. Now one such API reads it and refuses its own paying customer at job initiation. The site did nothing differently. The change happened entirely on the other side, and that is the only place a change of this kind can happen, because a request that arrives is a request that arrived.

It also means the population this binds is not the population most site owners worry about. It binds developers who chose a documented, hosted, invoiced product. It does not touch anyone running their own headless browser, and it does not touch the crawlers whose behaviour we keep having to measure rather than read: the ones where a Disallow line was published and a 200 was served anyway, or where a robots.txt block did not stop 12 of 18 chatbots. Compliance improving among the compliant is a real improvement with a narrow base.

What a site owner can check today

The practical consequence is small, cheap and worth doing, so here it is without inflation.

Your Content-signal line, if you have one, now has one documented consumer where it previously had none. That is a reason to check the line says what you meant, not a reason to expect traffic to change. If Cloudflare generates the file for you, remember that the token list inside it is refreshed by Cloudflare rather than by you, so the file you approved is not necessarily the file being served this week.

Check the file is served at all, and served consistently. When we fetched robots.txt across the 14 hostnames belonging to six real sites on 2 September 2026, three of the six served a different file on a second hostname and one returned nothing. A signal on the wrong hostname is not a signal. So is the status code: a 404 on robots.txt allows every crawler and a 503 blocks them all, and neither outcome is visible from a browser tab that renders your homepage fine.

Do not spend the effort on the Content-signal line that belongs on the Disallow lines under it. Those are the records the standard defines and every compliant AI crawler evaluates, and their behaviour is less obvious than it looks: we found that appending a trailing asterisk changed the verdict on none of 168 path comparisons while a trailing dollar sign changed 36 of 112. If you want to see which named agents your current file admits, the crawler reference lists the tokens and a single page fetch shows what one of them receives.

Finally, keep the layers separate in your head. Cloudflare is also changing defaults at the edge, where a new domain's AI bot policy will depend on whether the page carries ads from 15 September 2026. That is enforcement against arriving requests. The crawl endpoint change is enforcement against departing ones, applied to Cloudflare's own customers. They are different mechanisms with different populations, and a site owner who conflates them will over-estimate what either has done. What we grade, and how we grade it, stays where it has always been: whether a crawler that does arrive can fetch, parse and read the page.