# Content signals robots.txt: one of ten Cloudflare endpoints reads it > Cloudflare's changelog of 31 August 2026 records that the Browser Run crawl endpoint now respects the Content Signals use directive and rejects a job with HTTP 400 when the caller declares a more permissive level than the site does. We opened all ten Browser Run quick action documentation pages on 3 September 2026: nine of them name neither Content Signals nor robots.txt. - Canonical page: https://lantad.co/blog/content-signals-robots-txt-one-of-ten-endpoints - This file: https://lantad.co/blog/content-signals-robots-txt-one-of-ten-endpoints.md - Last substantive update: 2026-09-03 ## Key facts - **Published:** 2026-09-03 - **Category:** Findings - **Author:** Lantad - **Length:** 3543 words - **Takeaway 1:** Cloudflare's changelog of 31 August 2026 records that the Browser Run crawl endpoint now respects the Content Signals use directive, adding a contentUse parameter whose allowed values are reference and full and whose default is full. - **Takeaway 2:** A content signals robots.txt line now has one documented consumer: Cloudflare's crawl endpoint documentation, carrying a last updated date of 31 August 2026, states that a crawl is rejected when the declared contentUse level is more permissive than the site's declared use level. - **Takeaway 3:** Cloudflare's crawl endpoint documentation states in that same section that Content Signals are trust-based, and that by setting crawlPurposes and contentUse an operator is declaring to the site owner how it intends to use the crawled content. - **Takeaway 4:** Lantad opened all ten Browser Run quick action documentation pages on 3 September 2026: only the crawl page names Content Signals or robots.txt, and the other nine carry last updated dates between 28 May 2026 and 17 July 2026. - **Takeaway 5:** Read out of this repository on 3 September 2026, Lantad's production renderer calls the content and screenshot endpoints of the same Cloudflare API and never the crawl endpoint, so none of this enforcement applies to any scan this scanner runs. ## Summary A content signals robots.txt line has spent most of a year being a sentence addressed to nobody in particular. Cloudflare published the Content Signals Policy on 24 September 2025, [as recorded here on 30 July 2026](https://lantad.co/blog/content-signal-asks-disallow-blocks), and the line it writes into a file sits outside the four fields the robots exclusion protocol defines. When we ran Cloudflare's own published example through this scanner's parser on that date, the Content-signal line came back as an unknown field and changed no verdict for any of the 15 crawler tokens in our registry. That was not a defect in the parser. It was the correct handling of a record the standard does not define. On 31 August 2026 one piece of software started acting on it. Cloudflare's changelog records that the crawl endpoint of Browser Run, its hosted browser product, now reads the Content Signals use directive out of a target site's robots.txt and refuses to start a job when the caller has declared an intent the site did not permit. This post reports what Cloudflare published about that change, reads the endpoint documentation it applies to, and then does the one piece of first-hand work available: opening every sibling endpoint in the same product and writing down which of them mentions any of this. Lantad has sent no request to the crawl endpoint, holds no log of a rejection, and has not observed any site being refused. What follows is a reading of published documentation and of this repository, with the dates on it. ## What Cloudflare changed on 31 August 2026 The crawl endpoint is not new. Cloudflare's [Browser Run changelog](https://developers.cloudflare.com/browser-run/changelog/) records it arriving in beta on 10 March 2026, described there as scraping content from a starting URL and following links across the site. The product it belongs to was called Browser Rendering until 15 April 2026, when Cloudflare's changelog announced the rename to Browser Run; the REST path still carries the old name, which is worth knowing if you go looking for it in your own code. What changed on 31 August 2026 is a rule about who may start a job. Cloudflare's changelog entry for that date states that the crawl endpoint now respects the use directive of the Content Signals standard, that a caller declares its intended level through a new contentUse parameter whose allowed values are reference and full from least to most permissive, that the default is full, and that a crawl request is rejected with a 400 error if the target site's robots.txt sets a more restrictive level. The [crawl endpoint documentation](https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/), which carries a last updated date of 31 August 2026, puts the rule in one sentence: a crawl is rejected when your declared contentUse level is more permissive than the site's declared use level. There is a second rule alongside it, covering the three purpose signals rather than the use level. That documentation states that if a target site sets any of those content signals to no, the crawl request will be rejected at initiation with a 400 Bad Request error unless you explicitly narrow your declared purposes. Rejected at initiation is the part to notice. This is not a per-URL skip buried in a results file. The job does not start. The same page records that the endpoint respects the directives of robots.txt files, including crawl-delay, and that URLs blocked by robots.txt come back in the results with a status of disallowed. Those two behaviours are ordinary and have been available since the endpoint shipped. The Content Signals rules are the new part, and they are different in kind: a Disallow line stops a fetch, while a Content-signal line now stops a job on the basis of what the caller said it was going to do afterwards. ## What can a content signals robots.txt line make a crawler do? Six values, on two axes. Cloudflare's crawl documentation sets them out and gives each a definition, which is worth quoting because the definitions are narrower than the labels suggest. The three purpose signals are search, ai-input and ai-train. Cloudflare defines search as building a search index and providing search results with links and excerpts, ai-input as inputting content into AI models at query time, giving retrieval-augmented generation and grounding as the examples, and ai-train as training or fine-tuning AI models. That split is the same distinction this blog has kept returning to, because it is the one that decides whether blocking a bot costs you [the citations an answer engine hands out](https://lantad.co/glossary/geo) or only the training corpus. The ai-input row is the one most site owners set wrongly, since it covers the retrieval that puts a page inside an answer today rather than a model weight next year. The use directive is the second axis and carries three levels, given least to most permissive. Cloudflare defines immediate as ephemeral, single-response use where content is not retained, reference as content may be retained, indexed, or cited, and full as unrestricted use, including AI training. A site expresses a ceiling on that scale, and the caller declares where it intends to sit. None of these are new fields in robots.txt. They are values carried on a Content-signal record, and RFC 9309, published in September 2022, is explicit about the status of such a record. [The robots exclusion protocol](https://www.rfc-editor.org/rfc/rfc9309.html) says at section 2.2.4 that crawlers MAY interpret other records that are not part of the protocol, and that parsing them MUST NOT interfere with the parsing of the records section 2 defines. Google's own [robots.txt reference](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) lists four supported fields and states that anything else is ignored by its parser. So a Content-signal line remains, at the level of the standard, a comment that happens to be machine-shaped. What changed is that one commercial crawling service has chosen to read it, which is exactly the permission the RFC grants and nothing more. That distinction matters for anyone about to edit a file. Adding a use directive does not make your robots.txt stricter in any way a general crawler is obliged to notice. It adds a line that a specific set of clients have volunteered to honour, which is a real thing and a much smaller thing. If you want the part that binds, that is still the Disallow line, and it is still worth [checking what your file actually says per crawler](https://lantad.co/tools/robots-txt-tester) rather than assuming. ## The defaults declare the maximum on both axes Read the two parameter definitions next to each other and one property falls out. Cloudflare documents crawlPurposes as defaulting to all three purposes, and contentUse as defaulting to full, which its own scale calls the most permissive level. A developer who calls the endpoint with nothing but a URL has therefore declared the widest possible intent on both axes, without typing a word to that effect. That inverts the usual shape of a default. The untouched call is the one most likely to be refused, because it collides with any restriction a site has expressed. A caller who wants jobs to succeed has an incentive to narrow, and the documentation says so directly: the purpose rejection applies unless you explicitly narrow your declared purposes. Narrowing is free, takes one array, and is checked against nothing. There is a second asymmetry, and it is arithmetic on the two published lists rather than anything Cloudflare states. The use directive defines three levels and contentUse accepts two of them, reference and full. immediate is not among the values a caller can send. Since the rejection rule fires whenever the declared level is more permissive than the site's, and both declarable values sit above immediate on Cloudflare's own ordering, a site publishing use=immediate cannot be crawled by this endpoint under any parameter the endpoint accepts. Whether that is intended or simply a gap between a specification with three levels and a parameter with two, the published lists do not say, and we are not going to guess at it. None of this makes the feature pointless. It makes it a declaration mechanism rather than an access control, which is a distinction this blog has had to draw before: a licence directive in robots.txt [changes no access by its own admission](https://lantad.co/blog/rsl-license-directive-changes-no-access), and a [pay per crawl price is carried on a status code the specification leaves undefined](https://lantad.co/blog/pay-per-crawl-runs-on-an-undefined-status-code). The pattern repeats because the layer where these controls live was designed for one question, whether to fetch, and is being asked a different one, what you may do with what you fetched. ## Nine of the ten Browser Run endpoints name none of it The rule applies to one endpoint. That is not a criticism, it is scope, and scope is the thing a reader most needs to know before deciding whether any of this touches them. So we checked it rather than assuming. Cloudflare documents ten quick actions in Browser Run: content, screenshot, pdf, markdown, snapshot, accessibilityTree, scrape, json, links and crawl. On 3 September 2026 we opened all ten documentation pages and searched each for the phrases Content Signals, contentUse, crawlPurposes and robots.txt. The crawl page carries all four. The other nine carry none of them, and none of the nine mentions robots.txt at all. Their last updated dates run from 28 May 2026 for content, screenshot, pdf and markdown, through 7 July 2026 for links, json, snapshot and accessibilityTree, to 17 July 2026 for scrape. The crawl page is dated 31 August 2026, the day of the change. The nine are single-page actions rather than site crawlers, which explains part of it: a robots.txt file governs automated traversal, and fetching one page a human named is a different act. But several of them do the thing the Content Signals vocabulary is written about. The markdown endpoint extracts prose in the format [AI crawlers were observed fetching by URL rather than by content negotiation](https://lantad.co/blog/no-ai-crawler-used-content-negotiation). The json endpoint runs a model over the page. Those are ai-input shaped operations by Cloudflare's own definition, performed by a documented product, with no declaration asked for and no signal read. This is also the point at which we have to say what it means for this scanner, because the honest answer is nothing. Read out of this repository on 3 September 2026, Lantad's production renderer calls the content and screenshot endpoints of exactly this API and never the crawl endpoint, which is what you would expect of a tool that fetches the single URL a person pasted. So the 31 August change alters no scan we run, and [our own crawler conduct policy](https://lantad.co/bot) is unaffected by it. Saying so is cheaper than letting a reader assume the product they are reading about was in scope. ## Cloudflare calls it trust-based, and that is the whole mechanism The documentation does not oversell this, and the sentence it uses is the most important one on the page. Content Signals are trust-based, it says, and by setting crawlPurposes and contentUse you are declaring to the site owner how you intend to use the crawled content. Nothing verifies the declaration. A caller that intends to train a model may send a crawlPurposes array containing only search, pass every check, and receive the pages. The check is on the string, not on the use. That is not a flaw Cloudflare has hidden; it is the property the word trust-based names, and it is the same property that runs through every control in this area. A bot [declares three things about itself and can prove one](https://lantad.co/blog/llm-crawler-detection-declared-three-verified-one), which is identity. Intent has never been provable from the outside and is not provable here. What the mechanism does achieve is worth stating plainly rather than dismissing. It moves the moment of refusal from the site to the crawler's own vendor. Before 31 August 2026 a site's Content-signal line reached a general purpose crawling API and was ignored, in the way our 30 July 2026 parse found. Now one such API reads it and refuses its own paying customer at job initiation. The site did nothing differently. The change happened entirely on the other side, and that is the only place a change of this kind can happen, because a request that arrives is a request that arrived. It also means the population this binds is not the population most site owners worry about. It binds developers who chose a documented, hosted, invoiced product. It does not touch anyone running their own headless browser, and it does not touch the crawlers whose behaviour we keep having to measure rather than read: the ones where [a Disallow line was published and a 200 was served anyway](https://lantad.co/blog/gptbot-bans-served-a-200-anyway), or where [a robots.txt block did not stop 12 of 18 chatbots](https://lantad.co/blog/robots-txt-block-did-not-stop-twelve-chatbots). Compliance improving among the compliant is a real improvement with a narrow base. ## What a site owner can check today The practical consequence is small, cheap and worth doing, so here it is without inflation. Your Content-signal line, if you have one, now has one documented consumer where it previously had none. That is a reason to check the line says what you meant, not a reason to expect traffic to change. If Cloudflare generates the file for you, remember that [the token list inside it is refreshed by Cloudflare rather than by you](https://lantad.co/blog/cloudflare-generates-robots-txt-and-the-token-list-moves), so the file you approved is not necessarily the file being served this week. Check the file is served at all, and served consistently. When we fetched robots.txt across the 14 hostnames belonging to six real sites on 2 September 2026, [three of the six served a different file on a second hostname](https://lantad.co/blog/check-for-robots-txt-fourteen-hostnames-eight-files) and one returned nothing. A signal on the wrong hostname is not a signal. So is the status code: [a 404 on robots.txt allows every crawler and a 503 blocks them all](https://lantad.co/blog/robots-txt-404-and-503-are-opposites), and neither outcome is visible from a browser tab that renders your homepage fine. Do not spend the effort on the Content-signal line that belongs on the Disallow lines under it. Those are the records the standard defines and every compliant [AI crawler](https://lantad.co/glossary/ai-crawler) evaluates, and their behaviour is less obvious than it looks: we found that [appending a trailing asterisk changed the verdict on none of 168 path comparisons](https://lantad.co/blog/wildcard-robots-txt-trailing-star-changed-nothing) while a trailing dollar sign changed 36 of 112. If you want to see which named agents your current file admits, [the crawler reference](https://lantad.co/tools/ai-crawlers) lists the tokens and [a single page fetch shows what one of them receives](https://lantad.co/tools/what-gptbot-sees). Finally, keep the layers separate in your head. Cloudflare is also changing defaults at the edge, where [a new domain's AI bot policy will depend on whether the page carries ads](https://lantad.co/blog/block-ai-crawlers-cloudflare-default-turns-on-ads) from 15 September 2026. That is enforcement against arriving requests. The crawl endpoint change is enforcement against departing ones, applied to Cloudflare's own customers. They are different mechanisms with different populations, and a site owner who conflates them will over-estimate what either has done. What we grade, and [how we grade it](https://lantad.co/methodology), stays where it has always been: whether a crawler that does arrive can fetch, parse and read the page. ## Questions and answers **What is a content signals robots.txt line?** A record beginning Content-signal, carrying values on two axes. Cloudflare's crawl endpoint documentation, read on 3 September 2026, names three purposes, being search, ai-input and ai-train, and three use levels, being immediate, reference and full. RFC 9309 classes such a record as an other record: crawlers MAY interpret it, and parsing it MUST NOT interfere with the allow, disallow and user-agent records the protocol defines. **Does adding a use directive block AI crawlers?** No. It expresses a ceiling that a client may choose to honour. As of 3 September 2026 the one documented consumer we can point at is Cloudflare's Browser Run crawl endpoint, which rejects a job at initiation with a 400 Bad Request when the caller's declared contentUse is more permissive than the site's declared use level. Every other client is free to ignore the line, and the records that actually stop a fetch are still the Disallow lines. **Can a crawler lie about its declared purpose?** Cloudflare's own documentation answers this. It states that Content Signals are trust-based, and that by setting crawlPurposes and contentUse you are declaring to the site owner how you intend to use the crawled content. Nothing in the published rules verifies the declaration against the eventual use, and narrowing a declaration to pass the check costs one array element. **Does this change what Lantad does?** No. Read out of this repository on 3 September 2026, Lantad's production renderer calls the content and screenshot endpoints of Cloudflare's browser API and never the crawl endpoint, because it fetches the single URL a person pasted rather than traversing a site. Our conduct policy and user agent are published at /bot and are unchanged by the 31 August 2026 update. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot