llms.txt vs robots.txt: 19 of the 23 sites publishing one say nothing about AI in the other

llms.txt vs robots.txt: which file does an AI crawler actually read?

Start with what each document says about itself, because the two do not overlap at all.

robots.txt is specified by RFC 9309, the Robots Exclusion Protocol, a Standards Track document published in September 2022. It defines user-agent lines that name a product token, allow and disallow rules that say whether a URI matching a path may be fetched, and a parsing limit that the specification says MUST be at least 500 kibibytes. It says nothing about which content may be used to train a model and it defines no index of anything. It is a file about fetching, and the only question it answers is whether a named client may request a path. Resolving a real path against a real file with several groups in it is harder by eye than it looks, which is what the robots.txt tester here exists for.

The llms.txt specification describes a different artefact. Its v2 proposal, published 3 September 2024 and last modified 10 August 2026, sets out a file containing, in order, an optional byte order mark, an H1 with the name of the project or site, which it calls the only required section, a blockquote with a short summary, zero or more markdown sections of detail, and zero or more H2 delimited sections containing file lists. It is an index. It mentions no permission, no blocking and no access control, and it sets no size limit. We have written before about what the v2 revision added, and none of it was a way to refuse anything.

So the two files cannot substitute for each other even in principle. One withholds access and the other advertises content. A site that wants to say both things has to write both files, and the measurement below asks how many did.

How many real sites publish each file?

Of the 200 hostnames, 145 answered the robots.txt request with HTTP 200 and a plain text body. 47 answered with something else, which was 27 responses of 403, ten of 503, seven of 404, and one each of 406, 429 and 451. Five answered HTTP 200 with an HTML document, which is a page rather than a file, and three returned nothing at all: two connection failures and one timeout at 20 seconds. The 403s are worth dwelling on, because a scanner that reads a 403 as a refusal to be crawled is reading an edge rule aimed at unknown clients. That difference is the reason a scan here names what it could not fetch instead of grading around it.

The llms.txt column is thinner. 23 hostnames answered with HTTP 200 and a plain text body, 21 of them labelled text/plain and two labelled text/markdown. 156 answered with something else, of which 107 were 404 and 34 were 403. Five returned nothing. And 16 answered HTTP 200 with an HTML body, which is a site's catch all page dressed as a file. That last number is the trap in any adoption count: 39 hostnames returned HTTP 200 at that path and only 23 of them have a file. A crawler that checks the status code and stops has found 16 files that do not exist, and we have measured the same failure before at a smaller rate on a corpus chosen for likely publishers, which is what makes the rate here worth writing down: the further you get from the enthusiasts, the more of the 200s are false.

Read straight across, robots.txt is published as a real file about six times as often as llms.txt in this sample. This frame is the industry one; the separate platform frame is what feeds the crawlability study, and that study measures whether pages are readable rather than whether files are present. That is not surprising. It is worth stating anyway, because the same site can serve very different robots.txt files on different hostnames, which is something we found on three of six sites in an earlier check, and adoption counted once per hostname is the only honest way to count it.

What the 23 llms.txt publishers say in robots.txt

This is the finding. Take the 23 hostnames that publish a real llms.txt, look at the robots.txt each of them also serves, and count the user-agent lines that name one of the 15 AI crawler tokens this scanner evaluates. Four of the 23 name at least one: coursera.org, canonical.com, circleci.com and acehardware.com. The other 19 do not name a single one.

Those 19 have published a file addressed to language models and have said nothing to language models in the file that governs whether they may fetch anything. The rules that do apply to GPTBot on those sites are whatever the wildcard group says, which is a group written for search engine crawlers years before any of this. We have measured that pattern from the other direction too: on six real sites captured in July, all six robots.txt files named the wildcard and none named an AI crawler.

Across all 145 parseable files the picture is the same in the wider sample. Those files declare 1,516 user-agent groups between them and 110 of them name at least one sitemap, so these are maintained files rather than defaults. 28 of the 145 name at least one AI crawler token. GPTBot is the most named at 18 files, then ClaudeBot at 17, Google-Extended at 15, and CCBot and Bytespider at 14 each. The user-directed tokens trail the training ones: Claude-User and Claude-SearchBot appear on five files each and Perplexity-User on six.

One caution about what that count is. Naming a token is not blocking it and not allowing it either. It only means the file has a group addressed to that client, so its rules are decided deliberately rather than inherited. Whether the named crawler then obeys is a separate question, and a controlled study of ten assistants found six never requested the file at all.

Three different jobs are being done under one filename

Reading all 23 files makes the gap easier to explain. They are not 23 attempts at the same thing. The structure holds up: every one of the 23 opens with an H1 on its first line, 18 carry a blockquote in their first twelve lines, and 21 carry at least one H2 section. But five contain no absolute markdown link at all, which for a file whose defined purpose is a list of links is a strange thing to publish.

Look at what those five contain instead. cancer.ca serves 22,104 bytes at /llms.txt whose first line describes it as a crawling preferences file and whose body is robots.txt syntax: two user-agent lines, one for GPTBot and one wildcard, 468 Allow lines, 24 Disallow lines, a Crawl-delay and two Sitemap lines, followed by four fields no standard defines at all, named AI-Usage, AI-Attribution, AI-Training and Contact. The robots.txt that site actually serves names no AI crawler token at any point. americanexpress.com serves 643 bytes, headed as an LLM discovery file, carrying front matter dated 24 July 2026, one blockquote reading "All automated access and content use must comply with our robots.txt", and then a section of fun facts about travellers cheques and 1940s bowling teams. allbirds.com and brooklinen.com both serve a file headed as agent instructions, which tells a shopping assistant to install a skill file hosted at shop.app rather than script the storefront, a use that has nothing to do with indexing and everything to do with being reachable by an agent that arrives to transact.

Three different jobs, then: permissions, discovery and agent instructions, all at the same path. That is what an unratified convention looks like while it is still being decided, and it is a reason to be careful about any tool that reports whether a site has llms.txt as though the answer were one thing. It also explains the 19. A site that has written its preferences into llms.txt reasonably believes it has expressed them, and the evidence that anything reads that file remains thin.

A 110 MB llms.txt against a 500 kibibyte parsing floor

The other thing the two documents disagree about is size, and one file in the sample makes the disagreement concrete.

canadiantire.ca serves an llms.txt that arrives with a Content-Length of 27,074,258 bytes compressed and decodes to 110,896,787 bytes of text/plain. It is a correctly formed file: an H1, a blockquote, eleven H2 sections, and then 295,011 absolute markdown links, most of them product detail pages. Every other file in the sample put together comes to 1,169,107 bytes and 4,517 links. The median across the 23 is 7,286 bytes.

Nothing about that file is against its specification, because the specification sets no limit. RFC 9309 does set one for robots.txt, and it points the other way: a crawler MUST parse at least 500 kibibytes, which both guarantees a floor and tells an author that anything past it may be ignored. There is no equivalent sentence for llms.txt, so an author has no way to know what a consumer will read, and a consumer has no committed budget. Crawlers do enforce limits of their own elsewhere: Googlebot reads the first two megabytes of an HTML page and drops the rest.

It is worth being blunt about who reads any of this. OpenAI's crawler documentation, fetched on 7 September 2026 with this scanner's user agent, returns HTML containing the string robots.txt eleven times, describing it as the surface a webmaster uses to control OAI-SearchBot and GPTBot. The string llms.txt appears twice on that page, both times in a piece of site chrome pointing a reader at OpenAI's own documentation index. Their file is for their docs. Nothing on the page says their crawlers fetch yours, which is the same conclusion we reached when no AI crawler in a fixture set asked for markdown by content negotiation.

What this measurement does not establish

The sample is 200 hostnames chosen by a rule, not drawn at random from the web, so the counts describe this sample and nothing wider. They are also a single point in time: one request per path on one afternoon, with no retry, so a 503 here is a 503 then rather than a site that never serves the file. Where a hostname answered 403 to a request carrying our user agent, that is a fact about how an edge treats an unfamiliar client and not evidence about the file behind it.

The token count is a name check. It reports that a robots.txt has a group addressed to a named crawler, and deliberately not what that group then permits, because a group can allow everything. Nor does any of this measure whether a crawler obeyed, because obedience happens in your own logs and this scanner only ever sees the outside of a hostname, which is the boundary our own crawler's conduct page states from the other side.

What a site owner can do is check the same two things for themselves and see whether they agree. Fetch the robots.txt, resolve the tokens that matter against the paths that matter, then look at whether the pages the answer engines need are actually readable once fetched, which is a separate failure and the one a crawler-view fetch surfaces. If the answer is that llms.txt carries the preferences and robots.txt carries a wildcard group from 2019, then the preferences are being expressed in the file with the least evidence behind it, and the file with the standard behind it is silent. That is a cheap thing to fix and it is the first thing we would fix, ahead of anything else filed under AI visibility. We ship a generator for llms.txt as well, and the honest reading of this measurement is that what it produces is an index and not a policy, whatever the 19 sites above have written into theirs.