A robots.txt block did not stop 12 of 18 AI chatbots

How a canary token maps a chatbot to the crawler that fed it

The method is simple enough to describe in a paragraph, which is part of why it is interesting. The authors bought 20 .com domains with no recent ICANN history, hosted one hand-written website on each using Google Cloud, and built each site from a template carrying 10 placeholders for canary tokens. A canary token here is an ordinary-looking fact, a name or a place or a phone number, generated uniquely for each visiting scraper and dropped into otherwise identical prose. Every visitor therefore reads a slightly different version of the same page. If a chatbot later repeats one of those unique values back, the value identifies which visitor's copy the answer was built from.

Identity is defined in the study as the pair of a User-Agent string and an autonomous system number, which is a coarser notion than a full verification but a good deal stronger than the string alone. That distinction matters and this blog has argued it before, in a user agent is a claim, not an identity: a header is something a client asserts about itself, and pairing it with the network the request came from is the cheapest way to make the assertion cost something.

The sites were made findable rather than left to luck. The authors submitted them to Google, Bing and Brave Search through the respective webmaster tools and forms, and embedded hidden links to all 20 sites on other properties they controlled. Table 2 of the paper records what arrived: 2,765 unique User-Agent strings and 549 unique autonomous system numbers across all sites, 4,042 unique visitors in total, with a minimum of 313 unique visitors on the least popular site and 674 on the most popular. The average was 592.2 per site, which is the number the authors use to argue that token collisions are improbable, since the smallest of their token sample spaces holds 4,761 possibilities.

Then they asked. Twenty-two production chatbots were queried with a prompt tailored to each site, asking the system to search the internet and relay what the subject self-reports, followed by a second prompt in the same session asking whether any variant versions of the site existed. Eighteen of the 22 returned at least one measurable canary token. Deepseek, Hunyuan, GLM and Liquid returned nothing the authors could match and are excluded from every count that follows. That exclusion is worth holding onto: the study's denominator is 18, not 22, and a system that reveals nothing is not a system that took nothing.

Ten of eighteen chatbots returned content a search engine had fetched

The finding the authors describe as surprising is the one with the most direct consequence for anyone writing robots.txt rules. Content associated with Googlebot, bingbot or Bravebot turned up in responses from 10 of the 18 systems. In many of those cases the chatbot's vendor does not operate the search engine whose crawler fetched the page.

Some of those relationships are public. The paper treats Claude's use of Brave as publicly known, citing a news report rather than vendor documentation. Others were not: the authors name Qwen and Perplexity returning content associated with Googlebot as relationships that were not publicly documented. Perplexity is the sharper case, because Table 3 records both PerplexityBot, its own declared agent, and Googlebot against the same system. A site that disallows PerplexityBot by name has addressed one of the two paths the study observed into that product.

The authors considered whether they were simply watching User-Agent spoofing and rejected it on evidence. They checked the autonomous system number behind each third-party search agent and found the traffic originating from the network of the company the string named, which points at the first of their two explanations: the chatbot is ingesting search results about the site rather than fetching the site itself. That is a supply chain with two hops, and robots.txt only ever addressed the first one.

A separate group of six systems arrived without a crawler name at all. The paper reports generic browser User-Agent strings for six of the 18, naming ERNIE, Grok, Solar, Qwen and Kimi in the prose while Table 3 records a generic Chrome agent for a sixth system, Reka. Kimi is the extreme case, and Table 4 lists nine distinct strings measured in its responses, including Chrome, Edge, Googlebot, Obsidian, QQBrowser and QuarkPC. The authors infer that it rotates through a list, possibly to avoid bot detection. No robots.txt group can address a request that presents itself as Chrome, because there is no name to write in the User-agent line.

That is a different failure from the one this blog has covered in three AI crawler tokens that never appear in your logs, and the two are worth holding apart. Those tokens are robots.txt product tokens that deliberately send no User-Agent of their own, so a log search returns zero by design. What the canary study describes is the reverse direction: a real request does arrive, it does carry a name, and the name belongs to somebody else. Both leave a site owner writing rules against a set of names that is not the set of fetchers, which is the same problem underneath six of nine vendors publishing exactly one crawler token.

What happened when the sites blocked everything

The second half of the experiment is the part that answers the question most site owners are really asking, which is not whether a crawler can reach them but whether an edit undoes what already happened. The authors ran three conditions over three stages. Stage one left all 20 sites fully accessible for two months, long enough for the content to propagate, and ended with one round of queries to establish the baseline mapping. Stage two split the sites in half: 10 were taken offline entirely, and the other 10 stayed up but served a robots.txt file disallowing all bots. Chatbots were queried again one week and two weeks after that change. Stage three restored everything and queried again at the same intervals.

The result is stated plainly in section 5.3. Of the 18 chatbots for which the authors obtained User-Agent information, 12 continued to return content associated with the sites in both blocking conditions. Duck.ai was the only system that stopped in both, which the authors read as evidence that it evaluates availability and robots.txt directives when deciding whether to answer from a site.

Caching explains much of it and the authors say so. In section 5.2 they report that content fetched through a search engine crawler was still returned after a week offline for seven of the eight chatbots they count as ingesting data from Googlebot, bingbot or Bravebot. That is the expected behaviour of a retrieval pipeline sitting on a search index: the index holds a copy, and the origin going dark does not empty it. Persistence was not limited to that route, though, which is the less obvious half. Grok, ERNIE and Solar continued to return content associated with a generic Chrome agent after the sites went down, so material fetched by a browser-like client persisted too.

Two counts in the paper do not line up, and it is better to say so than to pick one. Section 5.1 counts 10 of the 18 systems returning content associated with those three search crawlers. Section 5.2, discussing the offline condition, refers to eight such chatbots and reports that seven of them still answered after a week. The direction of the finding is unaffected and the smaller figure is the conservative one, but a reader checking the paper will meet both numbers.

None of this is a timing artefact of the kind this blog has written about in when a robots.txt edit reaches a crawler, where the gap between saving a file and a crawler re-reading it is measured in days. Two weeks is well past the 24 hour ceiling RFC 9309 places on a cached robots.txt, and past the roughly 24 hours OpenAI's crawler documentation says its search systems take to adjust to a robots.txt update. The content persisted because it had already been taken, not because the instruction had not arrived.

What the study does not establish, in the authors' own words

The most useful sentence in the paper is a refusal. In the robots.txt condition the authors measured slightly more User-Agent strings than in the offline condition, which is exactly the shape of result that gets written up elsewhere as proof that AI crawlers ignore robots.txt. They decline to say it. Their wording is that the observation alone does not establish that a particular scraper violated robots.txt, because there are many possible confounding factors that could explain the difference between the two conditions. A site that is offline cannot serve anything to anyone, so the two conditions are not equivalent tests of compliance, and the paper does not pretend otherwise.

That restraint is the right way round, and it separates this study from the class of finding this blog covered in 234 of 592 sites that ban GPTBot served it a 200 anyway, where the measurement was of sites failing to enforce their own stated rule rather than of crawlers breaching one. Nothing in the canary study demonstrates a crawler fetching a disallowed page. What it demonstrates is that answers kept containing the content, which is a claim about product behaviour rather than about crawler compliance.

The stated limitations narrow it further. The analysis rests on 20 websites, queried from a single vantage point, the campus network of a United States university. The authors note that chatbots may use different scraping infrastructure depending on the target site, its content type or its geography, so their measurements may not capture the full range. The method can only observe scrapers that generate enough evidence to be matched, so a scraper that left no token in any answer is invisible to it rather than absent. And the particular prompts, sites and observation window all shape which scrapers become observable at all.

One more gap is worth naming because it affects how the figures should be quoted. The paper gives its measurement timeline in stages and intervals, two months here and one or two weeks there, but no calendar dates for any of them. The submission date, 13 May 2026, is the only fixed point available, so the honest form of every figure above is that it was measured in the months before that date rather than on any stated day. The paper also discards matches conservatively, dropping any token that was accidentally served twice, any token that is a substring of another, and all numeric tokens, on the grounds that a chatbot may produce a matching number by coincidence. Those discards push the counts down rather than up.

Naming what a measurement cannot support is the same discipline our methodology applies to a scan, and a similar controlled study of web agents reached a comparably narrow conclusion in robots.txt stopped one of twelve web agents. Neither result licenses the sentence people want to write, which is that robots.txt does not work.

Robots.txt is an access control, and the specification says so

It is tempting to read all of this as robots.txt failing. It is more accurate, and more useful, to read it as robots.txt being asked to do a job it was never specified to do.

RFC 9309 opens by describing itself as a method for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Access is the whole of the subject. The document sets out how a crawler matches a group, how allow and disallow rules are evaluated, and how long a fetched robots.txt may be cached, which it caps at 24 hours in section 2.4. It says nothing about content that has already been retrieved, because retention was never in scope. There is no removal verb in the protocol. A disallow rule is a request about future requests, and a study measuring what a chatbot says two weeks later is measuring something the file never claimed to govern.

The vendor documentation is consistent with that reading, and it is worth checking what each page actually commits to. OpenAI documents four tokens, OAI-SearchBot for surfacing sites in ChatGPT search, GPTBot for its foundation models, ChatGPT-User for user actions and OAI-AdsBot for validating ad landing pages, and the commitment attached to robots.txt is about how quickly the systems adjust. Anthropic's crawler article documents ClaudeBot, Claude-User and Claude-SearchBot and states that its crawlers honour robots.txt directives, including the non-standard crawl-delay extension. Perplexity's bots guide documents PerplexityBot and Perplexity-User and spends most of its length on firewall rules and IP ranges. Every one of those is a promise about fetching. None of them is a promise about forgetting.

Google's robots.txt documentation, read on 9 August 2026 and carrying the line Last updated 2026-07-08 UTC, makes the same point from the other end. It states that Google cannot index the content of pages which are disallowed for crawling, but that it may still index the URL and show it in search results without a snippet, and it points at a separate instruction for blocking indexing. Keeping a page out of an index and keeping a crawler off it have been two different controls for as long as both have existed, and the canary study is a reminder that the separation now runs through more parties than it used to.

So the practical position is not that robots.txt is useless. It is that it is a valve on one pipe, at a moment when the study found content reaching 10 of 18 products through a pipe that valve does not sit on. The pieces of infrastructure being drafted to close that gap, from crawler authentication to an AI preferences vocabulary, exist precisely because a per-crawler disallow line cannot express a preference that survives being passed along a supply chain.

What a robots.txt test can tell you, and what to check this week

Since this post is about the limits of a control, the consistent thing is to state the limits of the tool. Lantad answers a question with a definite scope: for a given URL and a given named crawler, is that crawler permitted to fetch this page, and if it does fetch it, is the text there. The robots.txt tester resolves a path and an agent against the rules as written, the AI crawlers reference lists the tokens tested, and what a crawler receives shows the served HTML against what a browser assembles. We publish what our own crawler sends so the request can be recognised in your logs.

What that does not answer is now clearer than it was, and the study is the reason. It cannot tell you which of the systems in the paper is currently holding a copy of your page. It cannot tell you whether removing access will remove the content from an answer, and on the evidence above the expectation should be that it will not, at least not quickly. It cannot see a fetch that arrived as Chrome from a residential network. And it says nothing about content that reached a chatbot through a search index rather than through a direct crawl, which was the majority route in the study.

Three things follow that are worth an afternoon. First, decide whether your concern is future access or existing copies, because they need different actions and only the first is a robots.txt job. If it is existing copies, the lever is the vendor's own removal or opt-out process, not your file. Second, if you do write per-crawler rules, write them against the search engine crawlers as well when the goal is genuinely to stay out of AI answers, and understand the cost: opting out of search indexing to close the second hop is a large price, and the study says as much in section 5.1. Third, check the ordinary readability question before the exotic one, because a page an AI crawler cannot parse is not visible whatever your rules say. That is prose parity, it is the largest weighted part of our score, and the archive evidence for how common the gap is sits in what a Common Crawl archive holds without JavaScript.

The broader point for anyone working on AI visibility is that the unit of control and the unit of measurement have come apart. Rules are written per crawler name. Content moves per supply chain. Until those two line up, a robots.txt file is an honest statement of intent whose enforcement depends on parties it cannot name, and the right response to a study like this one is to narrow the claims you make about your own setup rather than to abandon the file. That is why we withhold a grade when a scan could not establish enough to justify one, and why this post reports someone else's numbers as theirs.