Do AI crawlers respect robots.txt: 6 of 10 assistants never asked for the file

How a controlled domain catches an assistant fetching a page

The design is worth describing because it separates two things that public debate keeps welded together: what an assistant fetched, and what it told the user. The authors ran the work in two phases on two domains they registered for the purpose, genaimonitor.org and aicrawlerlab.org.

Phase one existed to establish that an assistant could browse at all. For each product the authors worked down a fixed priority order of configurations, preferring the free default model in a temporary chat without signing in, then signed in, then alternative models, and tested each candidate five times. A configuration counted as working only when the server log recorded a request for the target page and the assistant returned the secret code embedded in it. That double condition matters, and the paper reports it caught real cases: Copilot fetched the requested page in four of five trials under its Default Smart mode while telling the user it could not reach the server.

Phase one was not uniformly easy. ChatGPT required the most calibration of any product in the study, with 22 configurations tested and none succeeding in all five trials; the best returned the correct code in two of five. Copilot took nine configurations. Claude, Diffy Chat and Qwen produced correct answers in the first configuration tried, and Gemini, Grok, Mistral and Perplexity in the second. The authors draw the obvious methodological conclusion, which is that an advertised browsing mode is a product label rather than an observed capability, and they treat browsing as an empirical property of a specific configuration instead.

Phase two then tested compliance. The robots.txt was written to RFC 9309, the 2022 standard that defines the Robots Exclusion Protocol, and it carved the site into paired folders: one allowed and one disallowed for all user agents, and one allowed and one disallowed for each assistant that had exposed a distinguishable name in phase one. Assistants that had not exposed one shared a generic pair. Each of the four conditions was tested five times with a different target page each time to defeat caching, giving 20 trials per assistant and 200 overall. Each target page carried a secret number derived by HMAC over the page version, with the key held on the authors' server, and each assistant was assigned its own page-version range so a log line could be traced back to the product that caused it. Lantad's own scanner announces itself in a comparable way and publishes what it sends on the bot page, and the reasoning behind that kind of instrumented design is the same one set out in our methodology.

Do AI crawlers respect robots.txt when the page is fetched live?

The study's answer is that the question has no single answer, and that the interesting split is not between compliant and non-compliant products but between products that consult the file and products for which the file is not part of the retrieval path at all.

Four assistants requested robots.txt during the active trials: ChatGPT, Claude, Mistral and Perplexity, one request each. Six did not request it in any of their 20 trials: Copilot, DeepSeek, Diffy Chat, Gemini, Grok and Qwen. The authors also checked their historical logs for the server and found no earlier requests from those six that would have supplied the rules in advance.

Two assistants behaved as a site owner writing a Disallow line would expect. Claude accessed all five pages in the allow-all condition and all five in its own allow condition, accessed nothing in either disallow condition, and returned an explicit message saying retrieval was blocked by the site's robots.txt rules. Mistral likewise accessed nothing under either disallow condition, though its answers were less stable when access was permitted: the paper attributes two wrong answers to confusion between similar page-version identifiers rather than to an access failure. Perplexity mostly followed the pattern, avoiding its own disallowed folder entirely, but accessed one page in the disallow-all condition and surfaced that page's secret number.

Four assistants returned correct answers from pages the file disallowed: DeepSeek, Gemini, Grok and Qwen. DeepSeek accessed every requested page once and answered correctly in all four conditions, disallowed included. Gemini did the same with one hallucinated answer, and did so while exposing an identifiable Google user agent, which makes it the one product in the study that could have been bound by a named rule and was not observed requesting the file that carries them.

This is a different finding from the one in the canary-token work we reported in A robots.txt block did not stop 12 of 18 AI chatbots, and the two should not be merged. That study asked what a chatbot still repeats after access is withdrawn, which is a question about content already held. This one asks whether the file is consulted at the moment a live retrieval happens, which is a question about the fetch itself. A site can lose on either without losing on the other, and the remedies are not the same. For the retrieval that feeds Google AI Overviews and its equivalents, only the second question describes what happens when someone asks a question today.

Why an assistant-specific rule cannot bind a fetcher you cannot name

The most useful result in the paper is not the compliance count. It is the correlation underneath it. Every assistant that arrived at the server without a distinguishable name also arrived without asking for robots.txt. The authors state it directly: all the assistants using generic or non-identifiable user agents in their logs, being Copilot, DeepSeek, Diffy Chat, Grok and Qwen, accessed target pages without requesting the file during the active trials or at any point in the past.

That correlation converts a policy problem into a measurement problem. A rule written for a product token can only be applied by something that reads the token and admits to it. Grok's retrieval reached the server as an ordinary desktop Chrome string, which is indistinguishable in a log from a person with a browser. Nothing in the file could have addressed it, because nothing in the file had a name for it. This is the practical version of a point this blog has made from the product's own side in A user agent is a claim, not an identity: a header is an assertion, and here the assertion was simply withheld.

Copilot is the case worth reading twice. Table 1 of the study records its retrieval arriving under the user agent Diffbot-User, which is a name, but not Microsoft's and not Copilot's. The authors group Copilot with the non-identifiable set for exactly that reason: the token identifies a third-party content-extraction service rather than the assistant the user was talking to. A site owner writing rules for the assistants their readers actually use would have no reason to write that name down, and the requests would not be attributable to Copilot even in a log that captured them faithfully.

The gap between a published token and an observed one is why the two lists are worth keeping separate. Lantad's own registry of crawler tokens is published at the AI crawlers reference, and we measured on 27 August 2026 that only 6 of 15 crawler tokens publish a User-Agent string that can be matched literally in a log. That measurement was about documentation. This study measures the other end of the same pipe, which is what actually shows up. Neither is the whole picture, and a site owner who assumes a documented token covers the traffic is making the same error in the other direction that we described in Fourteen Google agents that a robots.txt wildcard does not stop.

One further detail belongs here because it is a model of careful reporting. The authors observed requests carrying a Google user agent whose source addresses failed Google's own recommended reverse-DNS validation, and declined to attribute those requests to Google at all. That is the right call, and it is the reason a user-agent count is a weaker evidence type than it looks.

Why the answer on screen is not evidence of what was fetched

The study's second structural finding is that retrieval and answer quality come apart in both directions, which means neither can be used to infer the other. This matters to anyone trying to audit AI access by asking an assistant what it can see, because that method reads the wrong end of the pipe.

ChatGPT is the clearest example of the first direction. It requested robots.txt once and then accessed no target page in any of its 20 trials, and returned no correct answer in any condition. It would be easy to score that as perfect compliance, and the authors explicitly refuse to: an assistant that also fails to reach the pages it is permitted to reach has demonstrated a retrieval failure, not a policy. The paired allowed and disallowed conditions exist precisely to catch that, which is why the design tests permission in both directions rather than only testing the block.

Copilot ran the opposite pattern. It fetched all 20 requested pages, disallowed ones included, and returned the correct secret number in none of the 20 trials. From the user's chair it looked like a product that could not read the site. From the server's chair it read every page it was asked for. Diffy Chat sat between the two, accessing 11 pages across the four conditions and returning the correct number once.

Then there is volume. Grok generated 52 page accesses in the allow-all condition, 48 in disallow-all, 48 in the generic allow condition and 50 in the generic disallow condition, which is 198 accesses in response to 20 prompts. Passive monitoring after the active window closed recorded a further 173 accesses over the following week, revisiting pages touched during the tests. ChatGPT continued requesting robots.txt twice a day using OAI-SearchBot/1.3 and OAI-SearchBot/1.0, and Perplexity requested the file five times over the three days after its first active request and hit the root page three times after the last one. The authors kept watching for more than a month.

The operational reading is that assistant-mediated retrieval is a process with its own footprint on your server, running on its own schedule, and that footprint is not visible from the conversation that triggered it. Lantad's own tooling works from the server side for the same reason: what GPTBot sees is built on a fetch, not on an assistant's self-report, and the same principle governs how AI visibility is scored rather than asserted. If you want to know whether a platform such as Perplexity reached a page, the log is the evidence and the answer text is not.

What OpenAI and Anthropic publish, and where the logs agreed

Two vendors in the study publish a position on this exact question, and the two positions are not the same. Reading them next to the results is the most useful thing a site owner can do with this paper, because it shows which promises are load bearing.

OpenAI's crawler documentation, read at source on 6 September 2026, names four agents. GPTBot crawls content that may be used to train the foundation models. OAI-SearchBot surfaces websites in ChatGPT's search features. OAI-AdsBot validates the safety of pages submitted as ads. ChatGPT-User covers certain user actions in ChatGPT and Custom GPTs, and about that last one the OpenAI crawler documentation says, in its own words, that because these actions are initiated by a user, robots.txt rules may not apply. That is a published carve-out, stated plainly, and it is the reason a site owner should not read a ChatGPT-User request in their log as a rule violation. It is behaving as documented.

Anthropic's published guidance takes the other route. Its support article names ClaudeBot for model training, Claude-User for user-initiated web access and Claude-SearchBot for search quality, and states that its bots respect do-not-crawl signals by honouring industry standard directives in robots.txt, with no carve-out for the user-initiated case. The same page notes that blocking by IP address rather than by directive can fail, because it prevents the bot reading the file in the first place. Anthropic's crawler and robots.txt guidance is worth reading in full if you maintain rules for it.

Set that against the logs. Claude-User was one of two agents in the study that accessed no disallowed page in either restricted condition, and it told the user why. That is a vendor commitment and an independent controlled measurement pointing the same way, which is rarer than it should be and is worth saying out loud. ChatGPT's result cannot be scored either way for the reason given above, since it fetched nothing at all.

The practical consequence is that per-platform guidance has to be written per platform. The rules a site needs for getting cited by ChatGPT are not the rules it needs for getting cited by Claude, and a single copied robots.txt block treats two documented and measurably different behaviours as though they were one.

What to check on your own site

None of the above is a measurement of your site, and this is where the reporting stops and the checkable part starts. Four things follow from the study that a site owner can act on without taking anyone's word for anything.

Check what your file actually says per token rather than in aggregate. A wildcard group and a named group produce different verdicts for different agents, and the difference is invisible until you evaluate each name separately. We measured on 5 September 2026 that across 29 hostnames serving a parseable file, the site root was allowed for GPTBot on 23 and for OAI-SearchBot on 27, and that on 4 of 29 sites the verdict changed with the agent. Twenty-one of those 29 named none of the three tokens at all, so the wildcard decided every verdict and nobody had made a decision about ChatGPT.

Read your access log for the agents that are not on anybody's list. The study's finding is that the requests you most want to attribute are the ones carrying an ordinary browser string. You cannot write a rule for those, and you should not conclude from their absence in a token report that the traffic is not happening. That limitation applies to our own numbers as much as to anyone's, and it is stated in the crawlability study alongside the sample it rests on.

Do not test AI access by asking an assistant. Copilot fetched every page and reported that it could not, and ChatGPT reported nothing while requesting the policy file twice a day for a month afterwards. The conversation is downstream of retrieval, generation and rendering, and any of those can fail without the others failing.

Finally, treat permission as the first question and not the only one. A page can be fully allowed and still hold nothing a crawler can read, which is the part of generative engine optimization that a robots.txt audit never reaches. Lantad's scope is the narrow, checkable end of this: it reads the published file, evaluates named tokens against a URL, and fetches the page to see what survives. It observes no assistant, holds no third-party logs, and every figure above belongs to the authors of arXiv:2607.14447 rather than to us.