Two layers decide whether AI can read your site, and robots.txt is the weaker one

What Cloudflare announced on 1 July 2026

Cloudflare's post, published on 1 July 2026 under the title "Your site, your rules: new AI traffic options for all customers", does two separable things, and it is worth keeping them apart because only one of them is a change of default.

The first is taxonomy. The post sorts bot traffic into eleven classifications: Search, Agent, Training, Transact, Data Collection, Security Testing, SEO, Ads Verification, Social and Link Preview, Feed Fetching, and Monitoring and Operations. Three of those carry the new defaults. The organising idea is purpose rather than identity, which is a genuine improvement on treating every automated request as one undifferentiated population, and it mirrors how the crawler vendors already describe their own fleets.

The second is the default itself. The announcement states that on 15 September 2026 new defaults will be set for those three classifications, and that Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default. Every qualifier in that sentence carries weight. It applies to newly onboarded domains rather than to the existing base. It applies to pages that display ads rather than to a whole zone. Existing customers are not moved silently: the post says an owner who wants to opt out of the new default configurations can mark that in their Security settings at any time leading up to 15 September.

Reporting this as a ban on AI crawling would be wrong, and much of the coverage has done exactly that. Search remains allowed by default, and Search is the category that produces citations and referral clicks, which is the outcome most people asking about AI visibility actually want. What moves is not permission but authorship. Today, a site that wants to be readable by an AI crawler expresses that in a file it wrote and can read back. From 15 September a growing number of sites will hold a position on the question that nobody at that site ever typed, because it arrived as a default.

Scale is why a default is a larger event than a setting. Cloudflare's own post describes more than 20 percent of web domains as sitting behind Cloudflare, in a passage arguing that losing trusted status across a network that size is a deterrent with teeth. That is a company characterising its own footprint rather than an independent census, and it deserves to be read with that discount applied. Even discounted, this is not a niche configuration detail affecting a handful of zones. It is a shift in the base rate of what a crawler meets when it makes a request, and base rates are the kind of thing the AI Visibility Index exists to track.

Why your robots.txt cannot answer this question

Robots.txt was never an access control, and the specification says so in plain words. RFC 9309, the Robots Exclusion Protocol, published on the Standards Track in September 2022, states that the rules in a robots.txt file "are not a form of access authorization" and that the protocol "is not a substitute for valid content security measures". That is not a caveat buried in an appendix. It is the design.

What a robots.txt does is publish a request. A crawler that reads it and complies is choosing to comply. A crawler that ignores it has broken nothing technical, only a convention, which is why the useful question about any crawler is never what your file says but what the crawler received. Lantad's capture of six real storefronts on 15 July 2026, written up in what a crawler actually meets on a real storefront, is a record of responses rather than of intentions for exactly that reason.

An edge rule is the opposite kind of object. It does not ask. It returns a status code, and it does so before the request reaches anything you wrote. The two layers are not in conversation. Nothing in the protocol requires your CDN configuration and your robots.txt to agree, nothing warns you when they diverge, and no field in either one references the other.

The gap itself is not new, and it is not specific to any one vendor. Anyone who has put a site behind a web application firewall has met it already: a rule written to stop credential stuffing catches an archiver, a rate limit tuned for human browsing catches a well-behaved crawler making a hundred requests in a minute, a geographic block written for compliance catches a data centre range a search engine happens to use. What is new is that the gap is about to be produced deliberately, at scale, by a default rather than by an accident, and along a dimension that has nothing to do with abuse. A crawler blocked under the new defaults will not have done anything wrong. It will simply have been sorted into a category.

Cloudflare's post is unusually clear about which layer it is operating in, and the distinction is worth quoting because it is the whole argument of this piece. Describing its managed robots.txt feature, which prepends a preference stating that crawling for search is acceptable but crawling for training is not, the post says such signals "signal a website owner's preference, rather than issuing blocks directly". The preference and the block are separate products doing separate jobs. You can have either without the other, and after 15 September some sites will have the block without ever having written the preference.

That is the failure mode worth naming. A site can serve a robots.txt that welcomes GPTBot by name, pass a robots.txt check cleanly, and still return a 403 to that same crawler at the edge. Nothing in the file is wrong. The file was simply never the thing answering the request. Any tool that reads your robots.txt and reports "allowed" without fetching the page as that crawler is reporting a preference and calling it an outcome, which is the same category error we refuse to make when a measurement fails.

The multi-purpose crawler problem

The hardest part of a purpose-based taxonomy is that not every crawler has one purpose.

Cloudflare's post is explicit about how it resolves this: multi-purpose crawlers such as Googlebot, Applebot and BingBot will be blocked by customers who have selected to block Training. A crawler serving several functions is treated according to the most restrictive category it falls into. That is a defensible rule. If one user agent fetches content feeding both a search index and a training corpus, and you have said no to training, there is no way to honour that with a single allow or deny except by denying.

The trouble is that the crawler vendors have spent two years building the alternative, and it works differently. Google's crawler documentation, last updated on 14 July 2026, describes Google-Extended as a standalone product token that publishers use to manage whether crawled content may be used for training future generations of Gemini models, and states directly that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". The same page says Googlebot's crawling preferences affect Google Search, including Discover and all Search features, as well as Images, Video and News.

So Google's design offers a scalpel: disallow Google-Extended, keep Googlebot, opt out of training without touching search. A category system that maps Googlebot to Training because Googlebot is multi-purpose reaches a different answer to the same question. Neither is dishonest. They are two coherent models of one problem, and a site can end up governed by both at once, in which case the more restrictive one wins by default and the owner may never learn which layer made the call.

Google-Extended has a further property that makes this hard to audit, and it is one we have already written about: it is a robots.txt product token that sends no user agent of its own. You cannot look for it in your logs, because it never arrives. It is a name you write in a file to control a behaviour you cannot observe. Layer an edge rule on top of that and you have two invisible controls stacked, with the only observable being whatever the stack produces at the end.

If you care specifically about being surfaced in Google's AI answers, the practical consequence is that the Google AI Overviews path now depends on a CDN setting as well as a robots.txt line. The directory of AI crawler user agents is still the right place to start, but it describes what each token is for, not what your infrastructure will do when one of them shows up.

The vendors already split their own crawlers by purpose

Cloudflare's three headline classifications are not an invention. They describe a split the largest AI vendors already publish.

OpenAI's crawler overview documents four separate tokens with four separate jobs. GPTBot is "used to crawl content that may be used in training our generative AI foundation models". OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features". ChatGPT-User is "used for certain user actions in ChatGPT and Custom GPTs", visiting a page when a user's question requires it. OAI-AdsBot is "used to validate the safety of web pages submitted as ads on ChatGPT". Training, search, agent, and a fourth thing that is none of the three.

Map that onto the category names and the correspondence is close enough to be useful and loose enough to be dangerous. GPTBot is Training. OAI-SearchBot is Search. ChatGPT-User is an Agent, which is the category blocked by default on ad-carrying pages for newly onboarded domains from 15 September. That last one deserves a second look, because ChatGPT-User is the token that fetches a page because a person asked a question that needed it. Blocking Agent traffic is a legitimate choice, and a publisher whose costs are driven by agent fetches may well want it. It is simply not the same choice as blocking training, and anyone who reads "AI crawlers blocked by default" as being about training alone has misread which categories are in scope.

The same structure exists across the other engines, which is why the how-to-get-cited section is organised per platform rather than as one set of universal advice. What gets you into ChatGPT's answers is not identical to what gets you into Claude's, and Perplexity differs again in both its crawler fleet and its citation behaviour. A single site-wide allow or block cannot express those differences, and a per-category one expresses them only if your categories and the vendor's tokens agree about which bot is which.

There is a verification problem underneath all of this that no taxonomy solves. A category rule keyed on a user agent string inherits every weakness of user agent strings, and as a user agent is a claim, not an identity sets out, that claim is trivially forgeable and confirming it means checking the requesting address against the vendor's published ranges. OpenAI publishes those ranges as JSON endpoints per bot, and its documentation notes that a robots.txt change can take around 24 hours to propagate through its systems, which is a useful reminder that none of these controls take effect the instant you save a file.

What an external scan can and cannot see

This is the point where a scanner has to be careful about what it claims, so here is the boundary drawn explicitly.

What a fetch from outside can establish: the status code your server returned to a request carrying a named crawler's user agent, the bytes that came back, whether your robots.txt names that crawler and what it says about it, and how the text a crawler receives compares with the text a browser renders. That last comparison is prose parity, and it is the one that most often explains a page which is technically reachable and still unreadable. All of it is observable, repeatable and dated, which is what makes it worth reporting at all, and how the score works sets out how those observations combine.

What a fetch from outside cannot establish, and must not be dressed up as establishing: which layer produced the response. A 403 to GPTBot looks identical whether it came from a CDN category default, a hand-written firewall rule, an origin deny, a rate limiter that happened to trip, or a WAF rule somebody added for an unrelated reason four years ago. The response is evidence. The cause is inference, and inference is not measurement.

Nor can an external scan see the settings themselves. Whether a page displays ads within the meaning of Cloudflare's new default is a determination made inside Cloudflare, not something derivable from the markup. Whether a zone has opted out before 15 September is visible only in that zone's Security settings. Whether a block is deliberate is a question only the owner can answer, and a great many blocks are entirely deliberate.

That last point matters more than it might seem. A scanner that treats every block as a fault is not measuring, it is selling anxiety. Plenty of sites should block Training. Plenty should block Agent traffic and keep Search. A publisher who has decided that model developers may not train on their archive, and has configured that correctly, deserves a clean report on the thing they configured rather than a red flag for having succeeded. Lantad publishes its own crawler's conduct policy and opt-out at our bot page precisely because the right to refuse a crawler is not a defect to be scored against.

What changes after 15 September is not the measurement but its interpretation. Before the change, an unexplained block on a site whose robots.txt said otherwise was most likely a misconfiguration. After it, the same observation has a second plausible cause involving nobody having made a mistake at all. That is a real reduction in what can be concluded from a single fetch, and it is better stated out loud than quietly absorbed.

What to check on your own site before 15 September

Five checks, in the order that resolves the most uncertainty per minute spent. None need a paid tool and none need access to anything but your own site.

Start with what your robots.txt actually says, per crawler rather than in aggregate. A file can allow one AI crawler and block another through a pattern nobody remembers writing, and the per-agent answer is the only one that means anything. A robots.txt tester that evaluates per named crawler gives you that from one paste.

Second, fetch a page as each crawler and look at the status code rather than the file. This is the check that distinguishes the two layers, and it is the only one that can. If the file says allow and the fetch says 403, you have found the gap this post is about. What GPTBot sees returns the exact text a crawler receives, which settles reachability and readability in the same request.

Third, compare the crawler view against the rendered view. A page that returns 200 to a crawler and hands it an empty shell is not readable, and this is by a wide margin the most common finding on JavaScript-heavy stacks. The per-stack notes exist because the cause differs by framework: Next.js fails differently from Shopify, and the full set of stack guides names the specific mechanism in each.

Fourth, find out who actually terminates your TLS. A surprising number of teams cannot say from memory whether their apex domain sits behind a CDN, which vendor it is, or who holds the dashboard login. If the answer is Cloudflare, the questions worth asking internally are whether the zone counts as newly onboarded, whether anyone has looked at the bot category settings, and whether the pages carrying advertising are the pages you most want cited.

Fifth, check more than one page. Access rules are rarely uniform: a marketing homepage, a product page and a documentation page can sit behind different rules, different origins and, in this case, different ad configurations. Scanning several pages at once is the cheap way to learn whether your access story is consistent, and comparing against another site is the cheap way to learn whether it is normal.

What none of these five will tell you is what your competitors did, or what the aggregate looks like across the web. That would need a survey, and the research page says plainly that there are not yet enough scans through Lantad to publish an honest one. Reporting a survey nobody ran is the specific failure this blog is written to avoid.

What this does not change

Access is a gate, not a destination. Getting through it is necessary and it is not sufficient, and that distinction survives every change to how the gate is operated.

A page any crawler may fetch, which returns 200 to all of them, and which ships its content only after a JavaScript bundle executes, is still unreadable to a crawler that does not execute JavaScript. A page that renders its text server side and says nothing about who published it, what the page is, or how it relates to anything else, is readable and still hard to cite, which is what structured data and the entity signals behind it are for. None of that is affected by a CDN category default in either direction.

It is worth being blunt about the limits of the news itself. This is one CDN, with one set of defaults, taking effect on one date, applying to newly onboarded domains, on pages that display advertising. Written out in full it is a narrower change than the headlines suggest. The reason it is worth writing up at length is not its immediate blast radius but what it signals: the decision about whether an AI crawler may read a page is migrating out of a text file the site owner authored and into infrastructure defaults the site owner inherited. That direction of travel is the story, and no single vendor's settings page will reverse it.

There is also a reasonable case that the change is good for the people reading this. A purpose-based split is more expressive than a single allow or deny, and a publisher who wants to be cited without feeding a training corpus has never had a clean way to say so at the infrastructure layer. Cloudflare's managed robots.txt and its category controls are, between them, an attempt to give that preference somewhere to live and something to enforce it with. The criticism here is not that the controls are wrong or unwelcome. It is narrower than that: two layers now answer the same question, they can disagree, and only one of them is visible in a file anyone can open and read back. Nothing about the controls being useful makes that gap smaller, and a site owner who assumes the file still speaks for the site will be wrong more often after 15 September than before it.

The honest position on the broader field is unchanged too. Most of what circulates as generative engine optimization advice remains unmeasured. Some of it is actively contradicted by the evidence, which is why what the evidence actually says about llms.txt exists on a site that ships an llms.txt tool. The checks in the previous section are worth doing because their effects are observable, not because they are fashionable, and the glossary is written to define the terms rather than to promote them.

Finally, the thing this post is not: a measurement. Lantad has not scanned Cloudflare's rollout, has not sampled how many domains will be affected, and will have nothing to report on 15 September beyond what any other outside observer could fetch for themselves. When there is something measured to publish, it will carry a date, a sample size and a description of what was not measured, in the same form as everything else here. Until then this is a report of an announcement and an argument about which layer answers the question, and it should be weighed as exactly that.