# Block AI crawlers: Cloudflare's new default depends on whether the page carries ads

> Cloudflare's bots changelog entry of 1 July 2026 states that from 15 September 2026 new domains onboarding to Cloudflare receive defaults under which bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed. That is a rule about a page, and robots.txt has no way to write one.

- Canonical page: https://lantad.co/blog/block-ai-crawlers-cloudflare-default-turns-on-ads
- This file: https://lantad.co/blog/block-ai-crawlers-cloudflare-default-turns-on-ads.md
- Last substantive update: 2026-09-01

## Key facts

- **Published:** 2026-09-01
- **Category:** Findings
- **Author:** Lantad
- **Length:** 3564 words
- **Takeaway 1:** Cloudflare's bots changelog entry of 1 July 2026 states that on 15 September 2026 new domains onboarding to Cloudflare receive updated defaults under which bots classified as Training or as Agent are blocked on pages that display ads, while bots classified as Search remain allowed.
- **Takeaway 2:** The ordinary way to block AI crawlers is a path rule in robots.txt, and RFC 9309, published September 2022, matches rules against the URI path alone, so no robots.txt file can express a rule that turns on whether the page it names displays advertising.
- **Takeaway 3:** Cloudflare's Block AI bots documentation, read on 1 September 2026, describes the ads option as using Cloudflare automated detection for pages that display ads on your zone, and does not publish how that detection decides.
- **Takeaway 4:** Lantad probed 12 AI crawler user agent strings against five pages it captured on 15 July 2026: robots.txt permitted the crawler in all 60 probes, and 11 of the responses still differed from the baseline, 4 of them returning HTTP 403.
- **Takeaway 5:** Cloudflare's documentation states that before 15 September all customers can opt out of these new defaults, and that multi-purpose crawlers combining Search and Training are blocked by configurations that block AI training.

## Summary

For two years the answer to whether a site can refuse an AI crawler has been a single mechanism. You write a group in robots.txt naming the crawler's product token, you put a Disallow under it, and the crawler either honours the request or it does not. The control lives in a file at a known path, anybody can read it, and what it says is the same for every page on the domain that matches the rule. From 15 September 2026 a large part of the web gets a second control that works nothing like that one, sits in front of the origin, and makes its decision per page rather than per path. [What an AI crawler is](https://lantad.co/glossary/ai-crawler) has not changed. What decides whether it gets a page has.

This is a report of Cloudflare's own announcement rather than a test of it. Lantad has not run the new defaults, has no access to the classification Cloudflare applies, and cannot say what any particular zone will serve on the day. Every statement about the change below is attributed to a Cloudflare page with the date that page carries, read on 1 September 2026. What this site can add sits beside the announcement rather than inside it: what a condition on advertising does to a file that can only talk about paths, why a published rule and a served response were already capable of disagreeing before any of this, and what somebody should go and check on their own domain in the two weeks that remain. [How a capture is taken and scored](https://lantad.co/methodology) here measures the second of those and not the first.

## How do you block AI crawlers, and what changes on 15 September 2026?

The change is published in two places and they agree. [Cloudflare's bots changelog entry titled New options to manage AI traffic](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/) carries the date 1 July 2026 and states that all customers can now manage AI crawlers by behaviour, Search, Agent and Training, instead of a single Block AI bots toggle. The same entry states that beginning 15 September 2026, new domains onboarding to Cloudflare receive updated defaults under which bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed. [The Block AI bots documentation page](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) carries a last updated date of 1 July 2026 and repeats it in the same terms, adding that before 15 September all customers can opt out of these new defaults.

Three configuration options exist for each of the three behaviours, and the documentation page words them as block on all pages, which issues the block across the entire zone, block on pages with ads, which uses Cloudflare automated detection for pages that display ads on your zone to block only on those pages, and allow, which does not add any blocking. The defaults arriving on 15 September are therefore not new machinery. They are the middle option applied to two of the three behaviours, for domains that arrive after the date, without the domain owner choosing it.

Two limits on what that paragraph establishes are worth stating before anybody acts on it. The first is that the announcement is scoped to new domains onboarding to Cloudflare, which is Cloudflare's phrasing, and neither page read on 1 September 2026 states that an existing zone's configuration changes on the date. The second is that the documentation describes the ads condition as Cloudflare automated detection for pages that display ads on your zone and does not publish how that detection reaches its answer. There is no list of ad networks on the page, no description of a signal, and no statement of what happens to a page that carries an ad slot which fails to fill. Anybody who needs to know which of their URLs the condition will catch cannot work it out from the published documentation, and this post is not going to guess on their behalf.

What can be said without guessing is the shape of the thing. A site's AI crawler policy has until now been one document that a person can read, and it is becoming a decision made at request time by a network, informed by a classification of the crawler and a property of the page. [The token list this scanner works from](https://lantad.co/tools/ai-crawlers) is a list of crawler names, because names were the only thing the old control could talk about.

## Search, Agent and Training are three behaviours, not three crawlers

The classification is the part most likely to be misread, because it is not a list of bots. Cloudflare's documentation defines Search as crawlers that index content so they can answer questions about it later, Agent as automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents, and Training as crawlers that take content to train or fine-tune a model. Those are descriptions of purpose. A crawler is sorted into them by what Cloudflare judges it to be doing, and the sorting is Cloudflare's, not the vendor's and not the site's.

That matters because a single product token can carry more than one purpose, and the documentation says what happens then. Multi-purpose crawlers combining Search and Training are, in the documentation's terms, blocked by configurations that block AI training. The consequence runs one way only: the stricter classification wins. A site that wanted to keep the search behaviour of a crawler because search behaviour is where citations come from, and to refuse the training behaviour because training returns nothing, does not get to split them when the vendor ships both under one name. The vendor's packaging decides.

This is the same structural problem that shows up whenever an AI crawler policy is written against companies instead of tokens, and it was covered here from the other direction when [a network level ratio names a company while robots.txt names a token](https://lantad.co/blog/crawl-to-refer-ratio-excludes-native-app-referrals). Vendors do not divide their crawling the same way as each other. Some ship separate tokens per purpose and some do not, and a policy expressed in purposes inherits whatever division each vendor happened to choose. The registry in this repository, published here on 31 August 2026, holds 15 tokens across 9 vendors, split 9 training, 3 search and 3 user agent by purpose, and that split is a record maintained by hand from vendor documentation rather than a discovery about the world.

There is a further gap underneath all of it, which is that a classification has to be applied to something a request actually carries. A request carries a user agent string, and a user agent string is a claim. Only [6 of the 15 crawler tokens catalogued here publish a User-Agent a site can match on](https://lantad.co/blog/gptbot-detection-six-of-fifteen-tokens-publish-a-ua), which is a measurement this site published on 27 August 2026 about vendor documentation rather than about Cloudflare. Cloudflare operates a bot database and verification machinery well beyond string matching, and nothing here should be read as a claim about its accuracy. The point is narrower: a site owner reading the word Training in a settings panel is reading the end of a chain of judgements, and cannot see any link in it. [Evaluating robots.txt against a specific crawler token](https://lantad.co/tools/robots-txt-tester) answers a question about the file, and after 15 September the file is no longer the only thing answering.

## robots.txt has no way to express a rule about advertising

[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the Robots Exclusion Protocol, was published in September 2022 and it is short about what a rule is. Allow and Disallow lines indicate whether accessing a URI that matches the corresponding path is allowed or disallowed, and to evaluate whether access to a URI is allowed a crawler matches the paths in allow and disallow rules against the URI. Matching starts with the first octet of the path and the most specific match is used. Every part of that is about the path. There is no syntax in the standard for a rule that depends on what a page contains, what it renders, or whether a slot on it fills with an advertisement.

So the two controls are now speaking different languages about the same site. One names a token and a path prefix, is published at a fixed location, and says the same thing to everybody who reads it. The other names a behaviour and a property of the page, is evaluated at request time, and is visible to nobody except the party making the request and the party serving it. Neither is wrong. They are not translatable into each other, and the first one is the one every published guide, every audit tool and every compliance conversation about AI crawling is currently built on.

The awkward case is the one where both are Cloudflare's. Bot Preference Sync, announced on 21 August 2026, is a setting that writes a site's AI bot policy for Search, Agent and Training into robots.txt and prepends it above whatever file already exists, which was covered here in [Cloudflare will generate your robots.txt, and its token list changes without you](https://lantad.co/blog/cloudflare-generates-robots-txt-and-the-token-list-moves). A policy of block on pages with ads has no representation in the grammar RFC 9309 defines, and [the Bot Preference Sync announcement](https://blog.cloudflare.com/bot-preference-sync/) does not state on the page read on 1 September 2026 how such a policy is rendered into a file that can only write path rules. That question is open here rather than answered, and it is open because nothing published resolves it, not because it was too hard to check.

Whatever the answer turns out to be, only two outcomes are available to a file that can only talk about paths. Either it writes a rule broader than the policy, in which case the file overstates the restriction and a crawler obeying it withholds itself from pages it would have been served, or it writes nothing for that behaviour, in which case the file understates and a crawler obeying it requests pages it will be refused. Both are ordinary consequences of encoding a conditional decision in an unconditional format, and both mean that reading a site's robots.txt tells you less than it did. That file was already an unreliable narrator in ways this site has measured: [the trailing star in a wildcard rule changed nothing across 168 comparisons](https://lantad.co/blog/wildcard-robots-txt-trailing-star-changed-nothing), and [a file that names a crawler at all can grant it more than the wildcard group does](https://lantad.co/blog/robots-txt-names-the-wildcard-not-the-ai-crawler).

## A published rule and a served response were already two different things

None of this introduces the gap between what a site says and what it does. It widens one that this scanner has already measured. Six of the golden fixtures in this repository are captures of live pages taken on 15 July 2026, each storing a bot matrix: one row per crawler token, holding what robots.txt said about that token and what the server actually did when a request carrying that token's user agent string arrived. Read out of those fixtures, [robots.txt permitted the crawler in all 60 of the user agent probes that ran against five real pages, and 11 of the responses came back unlike the baseline](https://lantad.co/blog/robots-txt-allowed-sixty-probes-eleven-differed). Four returned HTTP 403. Seven returned HTTP 200 carrying visible text that matched the honest baseline only 87 percent.

Those 11 divergences had nothing to do with Cloudflare's classification, which did not exist in its current form when the capture was taken, and this post makes no claim about their cause. What they establish is narrower and more useful: a site's published permission and a site's served response are independent facts, they were observed to disagree on a fifth of the probes in a real sample, and the only way to learn that they disagree is to send a request and read what comes back. An audit that reads robots.txt and stops has been reporting a document, not a behaviour, for as long as anyone has been running one.

From 15 September a systematic reason to diverge arrives on top of the incidental ones, and it lands on a specific and awkward subset of pages: the ones carrying advertising, which for a publisher are the ones that earn. A page that monetises would, under the default, be served to a Search crawler and refused to an Agent or Training crawler, while a page on the same domain that carries no ad is served to all three. The domain's robots.txt cannot describe that, and a crawler cannot discover it except by asking for each page and comparing the answers. Content withheld from a crawler is not a new subject either: a canary study found [a robots.txt block did not stop 12 of 18 AI chatbots](https://lantad.co/blog/robots-txt-block-did-not-stop-twelve-chatbots), which is the same disagreement running in the other direction.

The measurement gap this leaves is worth naming plainly, because it is the sort of thing that gets asserted rather than checked. Lantad has not observed the new defaults in operation, has captured no page under them, and has no fixture dated after the change because the change has not happened. Nothing in this post is a finding about Cloudflare's behaviour. It is a reading of Cloudflare's published intent, set against measurements this site took of something else. When the date passes, the question of whether a page is [readable in the initial HTTP response](https://lantad.co/glossary/prose-parity) will still be answered the same way it is answered now, by fetching the page as the crawler and looking at what arrived, which is also how this site established that [text delivered only in hydration JSON scored the same as no text at all](https://lantad.co/blog/hydration-json-scored-the-same-as-no-text).

## What to check on your own domain before 15 September 2026

Four checks, none of which needs a vendor and all of which can be done in an afternoon. They are ordered by how often the answer turns out to be different from what the person expected.

First, establish whether the change applies to you at all. Cloudflare's wording is that new domains onboarding to Cloudflare receive the updated defaults, and its documentation states that before 15 September all customers can opt out of these new defaults, with the AI bot policy configuration reached from Security settings. A zone that already exists and an opt-out recorded before the date are two different protections and it is worth knowing which one you are relying on, because only one of them is a choice you made. Second, fetch your own pages the way a crawler would and compare them with what a browser shows, on an ad-carrying page and on a page without ads, so that you have a before to compare against afterwards. [Seeing what GPTBot receives for a URL](https://lantad.co/tools/what-gptbot-sees) is that comparison, and taking it now is the only way to have a baseline on the fifteenth.

Third, decide the question the default decides for you, rather than inheriting it. Blocking Training on monetised pages is a defensible policy and there is a real argument for it, since a training crawler refers nothing by design. Blocking Agent is a different bet, because an agent fetch usually happens because a person asked for that page in that moment, and refusing it removes the page from an answer somebody was actively waiting for. The measured cost of blocking is not zero: a study of news publishers found that [blocking AI crawlers cost large publishers about 7 percent of traffic](https://lantad.co/blog/blocking-ai-crawlers-cost-7-percent-of-traffic) within roughly six weeks. That figure is somebody else's measurement of somebody else's sites, and it belongs in the decision as evidence rather than as a verdict on yours.

Fourth, if you conclude that being read is what you want, the work after access is ordinary and unchanged. Pages have to state what they are about in text that survives the fetch, which is the whole of [generative engine optimization](https://lantad.co/glossary/geo) once the crawler is through the door, and the platform specific version of it is written up for [getting cited in ChatGPT](https://lantad.co/how-to-get-cited/chatgpt) among others. [Our own robots.txt and crawler identity](https://lantad.co/bot) is published for the same reason this post names its sources: a control nobody can inspect is a control nobody can argue with. The limits of what a crawlability scan can establish are set out in [what these scans can and cannot show](https://lantad.co/research/crawlability-study), and the honest position on 1 September 2026 is that neither this site nor anybody outside Cloudflare knows which pages the ads condition will catch.

## Questions and answers

**What changes for AI crawlers on 15 September 2026?**

Cloudflare's bots changelog entry of 1 July 2026 states that from that date, new domains onboarding to Cloudflare receive updated defaults under which bots classified as Training or as Agent are blocked on pages that display ads, while bots classified as Search remain allowed. Its Block AI bots documentation, last updated 1 July 2026, repeats this and states that before 15 September all customers can opt out of these new defaults.

**Can robots.txt block AI crawlers only on pages that show ads?**

No. RFC 9309, published September 2022, defines allow and disallow rules that are matched against the URI path, with the most specific path match winning, and it provides no syntax for a rule conditioned on what a page contains or displays. A policy that turns on whether a page carries advertising can only be enforced by whatever serves the request, not expressed in the file.

**How does Cloudflare decide that a page displays ads?**

Its Block AI bots documentation, read on 1 September 2026, describes the option as using Cloudflare automated detection for pages that display ads on your zone, and does not publish the mechanism. No ad network list, signal or edge case is described on that page, so which specific URLs the condition catches cannot be determined from the published documentation.

**Does a robots.txt that allows a crawler mean the crawler gets the page?**

Not necessarily, and this was measurable before any of this change. Lantad probed 12 AI crawler user agent strings against five pages it captured on 15 July 2026: robots.txt permitted the crawler in all 60 probes, yet 4 responses came back HTTP 403 and 7 came back HTTP 200 carrying visible text matching the baseline only 87 percent. A published permission and a served response are independent facts.

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot