Blog / What the evidence actually says about llms.txt
What the evidence actually says about llms.txt
Ahrefs measured 137,210 domains in May 2026 and found 97 percent of valid llms.txt files were never fetched at all. We ship an llms.txt tool, and we think you should know this.
In short
- Ahrefs studied 137,210 domains in May 2026 and found that 97 percent of the valid llms.txt files it identified received zero traffic that month.
- In the same study, zero requests came from AI bots for llms.txt files that do not exist, and 98 percent of requests to non-existent llms.txt files came from humans, so AI systems do not go looking for the file.
- Of the bot requests that did reach an llms.txt file, 19.5 percent came from named AI tools, and AI retrieval bots specifically accounted for 1.1 percent of AI bot requests.
- Lantad ships llms.txt analysis and a free llms.txt generator, and publishes this finding anyway, because a tool that hides the evidence against its own feature is not a measurement tool.
- The honest case for writing one is that it costs a few minutes and one file, not that it has been shown to change what AI engines do.
llms.txt is a proposed convention: a Markdown file at the root of your site telling AI systems which pages matter and summarising what they contain. It is a sensible idea, it is easy to implement, and plenty of tools will generate one for you. Lantad is one of them.
In June 2026 Ahrefs published the first large measurement of whether anything actually reads these files. The answer is mostly no. We are writing this up because we sell an llms.txt feature, and a scanner that only publishes the evidence flattering to its own features is not a scanner.
What the study measured
Ahrefs looked at 137,210 domains through its own web analytics product during May 2026 and identified which published a valid llms.txt. Roughly 28 percent did, around 38,000 sites. It then looked at the request logs for those files.
97 percent of them received zero traffic in the month. Not low traffic. None. Of the 3 percent that received anything, 96 percent of those requests came from bots rather than people, and 19.5 percent of the bot requests came from named AI tools. The subset that matters most for being cited in an AI answer, retrieval bots, made up 1.1 percent of AI bot requests.
The study is at ahrefs.com/blog/llmstxt-study, published 15 June 2026. We have not reproduced this measurement ourselves and report it as their finding rather than ours.
137,210 domains, May 2026
- Domains studied 137,210
- Published a valid llms.txt about 28 percent
- Of those, received any traffic at all 3 percent
- Of that bot traffic, from named AI tools 19.5 percent
- Of AI bot requests, retrieval bots 1.1 percent
The finding that matters most
The headline number is the 97 percent, but the more decisive finding is quieter.
Ahrefs also looked at requests for llms.txt files that do not exist, the 404s. If AI systems were probing for the file the way crawlers probe for robots.txt, those 404s would be full of AI bots. They are not. 98 percent of requests to non-existent llms.txt files came from humans, and zero came from AI bots.
That is the difference between a convention and a standard. Crawlers fetch robots.txt without being asked, because it is part of how crawling works. Nothing fetches llms.txt unless something told it the file was there. A file nobody looks for is not directing attention, whatever it contains.
robots.txt
- Fetched unprompted by crawlers
- Part of the crawl sequence
- A missing one is requested and 404s
- Honoured by every major AI vendor
llms.txt
- Not fetched unprompted
- Zero AI bot requests for missing files
- 97 percent of real ones never fetched
- A proposal, not an adopted standard
So should you write one
Probably yes, for reasons that survive the evidence above, and definitely not for the reason usually given.
The bad reason is that AI engines will read it and cite you more. There is currently no measurement supporting that and one large measurement pointing the other way. Anyone telling you otherwise should be asked for their data.
The reasonable reasons are smaller and real. It costs a few minutes and one file. Writing one forces you to state plainly which pages matter and what each is for, which is useful whether or not a machine reads the result. Some tools and some humans do fetch it when pointed at it. And if adoption arrives, having one already is cheaper than noticing late.
What it is not is a strategy. If your time is limited, the things with measurable effect are whether an AI crawler can fetch your pages at all, whether the text survives without JavaScript, and whether your pages say who you are in a form a machine can parse. Those are checkable today, on your own site, and their effect is not in dispute.
- Can an AI crawler fetch the page at all 25 pts Directly measurable from robots.txt and the response. A blocked page cannot be cited by anything.
- Does the text survive without JavaScript 50 pts Directly measurable by comparing the crawler view against the rendered view.
- Is the page structured so a machine can parse it 25 pts Directly measurable from headings, structure and schema.
- Does the site publish an llms.txt Cheap and harmless. No measurement currently shows it changes what AI engines do, and one large study shows almost nothing fetches it.
Common questions
Is llms.txt worth writing in 2026?
It costs a few minutes and it is harmless, so probably yes. Just do not expect it to change what AI engines do. Ahrefs measured 137,210 domains in May 2026 and found 97 percent of valid llms.txt files received zero traffic, with zero AI bot requests for files that did not exist.
Do AI crawlers look for llms.txt the way they look for robots.txt?
No, and this is the clearest finding in the Ahrefs study. Requests to non-existent llms.txt files came 98 percent from humans and zero percent from AI bots. Crawlers fetch robots.txt unprompted; nothing fetches llms.txt unless it was told the file exists.
Why does Lantad still offer an llms.txt tool if the evidence is this weak?
Because customers ask for it, it costs one fetch to check, and it is harmless to have. What we will not do is present it as a differentiator or imply it drives citations.
What should I do instead if I want AI engines to cite me?
Check the three things with measurable effect first: that AI crawlers are not blocked from your pages, that your text is present without JavaScript, and that your pages carry structure and schema identifying who you are. Those three are the whole of Lantad's AI Visibility Score: access is 25 points, prose parity 50, and structure and schema the remaining 25. llms.txt scores nothing, which is the honest answer to why we publish this.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.