Cloudflare will generate your robots.txt, and its token list changes without you

What does Cloudflare's Bot Preference Sync actually do?

Cloudflare's announcement, published on 21 August 2026 under the byline Jin-Hee Lee, describes a setting that keeps robots.txt reflecting the AI bot preferences a customer has already set on the zone level dashboard. Three categories are covered: Search, Agent and Training. For Search and Agent the three options Cloudflare announced on 1 July 2026 remain, which the post gives as Allow, Block on pages that serve ads, and Block everywhere. For Training the post describes refining the option with Disallow, which writes a no training preference into the file.

The mechanics are stated plainly enough to quote. Where a site already has a robots.txt file, the announcement says the contents added by Bot Preference Sync will be prepended to the existing material, so any existing Disallow directives are maintained. The generated block is bounded by two comment lines, BEGIN Cloudflare Bot Preference Sync and END Cloudflare Bot Preference Sync, and the worked example in the post shows four User-agent lines followed by a single Disallow slash inside those markers. The names in that example are placeholders which the post says have been shortened and anonymised.

Availability and defaults matter as much as the mechanism. The post says the feature is available to all customers from the Free tier to Enterprise, that it will be on by default for all new customers, and that existing customers using the legacy managed robots.txt feature will be prompted to review and confirm their preferences during the transition. New customers get no blocks or disallows added on their behalf, with one exception: a site that selects the option about monetising from pages with ads at onboarding gets Training set to Disallow as its default, which the post frames as staying in search while keeping content out of model training.

The feature this replaces is already documented. Cloudflare's page for the managed robots.txt setting, carrying Last updated 3 August 2026 and read on 28 August 2026, says that when the setting is turned on, Cloudflare generates and maintains a robots.txt file that instructs known AI crawlers to stay away from your content. The new setting is that idea widened from one category to three, and the announcement says the sync can be turned on or off at any time.

Why does a prepended group change which rules a crawler reads?

Prepending sounds safe, and for the file's own text it is: nothing is deleted, and the lines a site wrote stay exactly where they were. The consequence is not in the text. It is in how a crawler decides which lines to obey.

RFC 9309, the Robots Exclusion Protocol, published on the Standards Track in September 2022, is specific in section 2.2.1. A crawler must use case insensitive matching to find the group that matches its product token and then obey the rules of that group. If there is more than one group matching the user agent, the matching groups' rules must be combined into one group. Then comes the sentence that does the work here: if no matching group exists, crawlers must obey the group with a user-agent line with the star value, if present. The wildcard group is a fallback for crawlers nobody named, not a base layer that applies underneath everything else.

So consider the common shape of a real file. A site has one group, headed by the wildcard, carrying whatever it wants kept out of every crawler's reach. No AI crawler is named in it, because most files do not name one: the 2025 Web Almanac count written up on this site found the catch-all user agent in 77.04 percent of desktop robots.txt files and gptbot in 4.5 percent. Bot Preference Sync then prepends an explicit group naming a set of crawler tokens. Every crawler named in that block now has a group matching its own token where it previously had none, so under section 2.2.1 it stops reading the wildcard group and obeys the generated block instead.

Where the generated rule is a blanket Disallow the outcome is the same either way, because everything is blocked and there is nothing left for the wildcard group to add. Where the generated rule is narrower, a crawler that used to be governed by the site's own wildcard rules is governed by Cloudflare's rules instead, and the site's rules for that crawler have quietly stopped applying to it. Cloudflare has not published the output for the option that blocks only on pages serving ads, and nobody here has seen it, so whether that option produces a narrow rule is not established. The mechanism is, and it is the same failure mode as a group headed by a token no crawler answers to: a file that reads as though it is in force and is not. The behaviour is documented rather than exotic, and fourteen named Google agents already ignore the wildcard group for a related reason.

The longest rule wins, not the one written first

The second half of the matching story runs the other way, and it is the half that keeps prepending from being as destructive as it might sound.

RFC 9309 section 2.2.2 says the matching must start with the first octet of the path, that the most specific match found must be used, and that the most specific match is the match that has the most octets. Where an allow rule and a disallow rule are equivalent, it says the allow rule should be used. Position in the file decides nothing at all. A rule written at the top has no precedence over a rule written at the bottom, and the only thing separating two conflicting rules is how many characters of the path each one matched.

That has a direct consequence for a prepended block. Suppose a site's existing file already carries a group for a named crawler with an Allow rule for a documentation directory inside it, and the generated block prepends a blanket Disallow for the same token. Section 2.2.1 merges the two groups into one, and section 2.2.2 then resolves the conflict by length rather than by order. For a request below that directory the Allow rule matched several octets and the blanket Disallow matched one, so the Allow wins and the page stays crawlable. A site owner reading the top of the file sees a blanket Disallow and reasonably concludes the crawler is shut out. It is not.

This is the same arithmetic behind a result published here on 27 August 2026, that a trailing asterisk changed no verdict across 168 pattern comparisons while still making a rule one character longer and therefore one character more specific. Octet count is the currency, and a character that matches nothing extra can still win a conflict. Anyone auditing a file that two parties now write into has to read it as a merged rule set resolved by length rather than as a document read from the top down, which is exactly why the robots.txt tester on this site resolves a path against the whole file rather than showing you the lines.

A maintained token list is the point, and also the cost

The most consequential sentence in the announcement is a short one. Cloudflare says it will use bots that it tracks in BotBase to periodically update the list of bots added to robots.txt when a customer chooses to block or disallow a category.

Read as a fix, that is the right design. Crawler tokens change, and a hand written file goes stale in silence. Google replaced Google-NotebookLM with Google-GeminiNotebook on 16 July 2026, a rename written up here in the post on stale robots rules, and nothing on a site reports a group that has stopped matching anything at all. Lantad's own crawler registry in core/src/bots.ts holds 15 robots.txt product tokens, and keeping even that many current is continuing work rather than a job that finishes. A vendor that maintains the list removes a genuine maintenance burden, and it removes it from thousands of sites at once.

Read as a change of control, it is a different thing. The file stops being a reviewed artefact. Its contents on any given day are whatever the vendor's bot database held when it last regenerated, which means the set of crawlers a site is blocking can change without a commit, without a deploy, and without anyone at the site deciding to block them. Cloudflare is explicit that the sync handles policy decisions made category wide rather than case by case, and that it will not read from individual custom rules with more complex logic, so a site with a specific arrangement with a specific company is told to turn the sync off and tailor the file itself. That is honest advice and it is also the whole tradeoff in one sentence.

Timing compounds it. RFC 9309 section 2.4 says crawlers may cache a fetched robots.txt and should not use a cached version for more than 24 hours unless the file is unreachable, which is a recommendation rather than a requirement and is made independently by each crawler. A regenerated file therefore does not take effect when it is written: there is a window, up to about a day wide and different per crawler, in which crawlers are acting on different versions of a file nobody at the site edited. The lag between an edit and its arrival is the subject of its own post here, and it applies to a machine's edits exactly as it applies to yours. It is worth remembering too that some of the names on any such list never appear in a server log at all, because they are usage controls rather than fetchers.

What robots.txt says and what the server does are two measurements

Cloudflare's stated motivation is worth reporting carefully, because it names the same problem this scanner measures. The announcement says there are cases in which your robots.txt states that a crawler is disallowed from accessing your website while your enforcement rules do not actually block that crawler, and that when stated preferences and enforced rules disagree, some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules.

That gap is measurable, and it has been measured, including here. Lantad probed 12 AI crawler user agent strings against five real pages it captured and stored on 15 July 2026, and robots.txt permitted the crawler in all 60 of those probes while 11 came back unlike the baseline fetch: four received HTTP 403 and seven received HTTP 200 carrying visible text that matched the baseline only 87 percent. The file said yes and several servers said something else. The disagreement runs in the other direction too, and at a larger scale: HasData found that 234 of the 592 domains disallowing GPTBot in robots.txt still returned it a live 200 when a request carrying that name actually knocked, measured in July 2026 across a 2,096 domain enforcement subset.

Both directions of that disagreement are what a sync between the stated preference and the enforced rule is trying to close, and closing either one is useful. What no sync can close is the part that was never robots.txt's job. RFC 9309 says in its first section that the rules are not a form of access authorization, so a site that writes a Disallow line and still serves the bytes is following the specification rather than breaking it. A synchronised preference remains a preference, it is read by clients that choose to read it, and a user agent string is a claim rather than an identity whichever side of the file you are standing on.

None of the figures in this section describe Cloudflare's feature. Lantad has run no test against Bot Preference Sync, and both measurements above predate the announcement by more than a month. They describe the condition the feature is aimed at, observed on sites that had nothing like it in place.

What should you check on your own site?

None of this requires a verdict on whether the feature is good. It requires a habit change, because the file is now something two parties write into.

Fetch your own robots.txt and look for the marker comments. If the BEGIN Cloudflare Bot Preference Sync line appears, everything below it and above the END line is generated, and only what follows is yours. Record what the generated block names today, because that is precisely the part designed to change later, and a diff against a copy you kept is the only way you will notice when it does.

Then resolve a real path against the merged file rather than reading the file. If a crawler token appears both in the generated block and in a group you wrote, section 2.2.1 merges them and section 2.2.2 settles any conflict by octet count, which is not something anyone does reliably by eye on a long file. Our methodology page sets out how this scanner grades a robots.txt result and what it will not claim from one, and our own crawler's conduct rules are published under the same standard we are describing here.

Check the enforcement layer separately, because it is the half no file can report on. A 404 and a 503 on robots.txt mean opposite things to a crawler, and a server that returns 200 to a browser and 403 to a named agent leaves no trace in the file at all. The list of AI crawler tokens this scanner tests is a reasonable starting point for which names to send when you check what your own origin answers.

Finally, decide who you want writing the file. A maintained list is the right answer for a site with no special arrangements and no capacity to track token renames, which is most sites. It is the wrong answer for a publisher with a signed deal, and Cloudflare says as much: its guidance for anyone with a more fine tuned policy is to turn the sync off. That decision now sits alongside the other content controls a site can state, including the Content Signals Policy and the per request toll that pay per crawl proposes, neither of which is access control either.