BlogFindings
llms.txt v2 adds two link relations and drops the tool that read v1
The llms.txt proposal was revised to version 2 and the page now carries a modified date of 10 August 2026. The change its author names first is discoverability: two standard link relations, carried in an HTML link element or an HTTP Link header, so an agent can find a page's markdown version and the llms.txt covering it without guessing. The context expansion tool that version 1 described as the consumer of these files is no longer part of the proposal.
Lantad has measured nothing about version 2. It was published the day before this post, no AI crawler documentation has been revised to mention it, and there is no adoption figure worth quoting, including ours. So what follows reports what the author wrote, and then separates the parts a site owner can act on today from the parts that depend on somebody else implementing them first. We have reported before that the measured evidence for llms.txt moving traffic is thin, and nothing in version 2 changes that. A revision to a specification is not evidence of consumption, and anyone who tells you otherwise this week is selling something. What version 2 does do is replace guessing with a declared path, and a declared path is at least the kind of thing that can be checked later.
In short
- The llms.txt proposal by Jeremy Howard, first published on 3 September 2024, carries a modified date of 10 August 2026 and is now titled The /llms.txt file, v2.
- Version 2 answers what its changes page calls the commonest request, discoverability, with two standard link relations: rel=alternate with type text/markdown for a page's markdown version, and rel=describedby for the llms.txt file that covers it.
- Those relations may be carried as HTML link elements or as an HTTP Link header, and the changes page states the header form also works for non-HTML resources and can be added in web server or CDN configuration without modifying any pages.
- Version 2 removes the llms_txt2ctx context expansion tooling from the proposal, and with it the mechanical meaning of the Optional section, which had existed to tell those tools what to omit.
- Lantad has not measured version 2 adoption and reports the state of the proposal from llmstxt.org as read on 11 August 2026, not from any scan.
| Aspect | Version 1, September 2024 | Version 2, August 2026 |
|---|---|---|
| Finding a page's markdown version | Not specified | rel=alternate with type text/markdown |
| Finding the llms.txt that covers a page | Not specified | rel=describedby |
| Where those relations may be carried | Not applicable | HTML link element, or an HTTP Link header |
| Markdown URL form | page.html.md only | page.html.md or page.md |
| llms.txt in a subpath | Permitted, meaning undefined | Covers pages under its path, most specific applies |
| Context expansion tooling | llms_txt2ctx described in the proposal | No longer part of the proposal |
| The Optional section | Told those tools what to omit | Convention only, no mechanical semantics |
What version 2 of llms.txt actually changed
The proposal is a single page, authored by Jeremy Howard, published on 3 September 2024 and now carrying a modified date of 10 August 2026 with the title The /llms.txt file, v2. A separate changes page, headed v2 (August 2026), sets out what moved and why, and it is the more useful of the two documents if you already knew version 1.
Its stated reason for the revision is adoption rather than theory. The changes page says that thousands of sites now publish an llms.txt file, that documentation platforms generate one automatically, and that coding agents use them reliably. The proposal page adds that the AI labs publish llms.txt files for their own developer documentation, naming OpenAI, Anthropic and Gemini, and notes that Chrome's Lighthouse audits sites for one as part of its agentic browsing checks, which is a detail we covered separately when that audit category appeared. None of those are numbers, and the word thousands is the most precise quantity either page offers, so treat the adoption claim as the author's characterisation rather than a measurement.
Five things changed, and they are worth separating because they carry very different weight. The first is discoverability, which the changes page calls the commonest request and which gets the rest of this post. The second is that markdown versions of pages may now use either of two URL forms rather than one. The third is that an llms.txt file placed in a subpath now has a defined meaning instead of merely being permitted. The fourth is that the context expansion tooling is gone from the proposal. The fifth is that the Optional section, which existed to serve that tooling, keeps its name and loses its mechanics.
The framing question the changes page uses for the first of those is precise enough to quote: given a page, how does an agent find its markdown version, or the llms.txt file that covers it, without guessing. That is a discovery problem rather than a content problem, and it is the reason this revision matters more to anyone doing generative engine optimisation than a longer list of formatting rules would have. A file nothing can locate is not a channel. It is a file. The rest of this post takes the five changes in the order that affects what you would actually edit, which is not the order they appear on the changes page, and ends with what none of it settles.
-
DiscoverabilityAdded Two standard link relations, rel=alternate with type text/markdown and rel=describedby, carried in an HTML link element or an HTTP Link header. -
Markdown URL formRelaxed Version 1 specified .md appended to the full page URL. Version 2 also allows the extension to be replaced, because publishing tools already did that. -
Subpath semanticsDefined A file covers the pages under its path and the most specific file applies. Version 1 permitted subpath files without saying what they meant. -
Context expansion toolingRemoved The llms_txt2ctx tool that version 1 described for expanding a file into an LLM context is no longer part of the proposal. -
The Optional sectionDemoted Still allowed and still a useful convention for secondary links, but it no longer carries mechanical semantics now that the tooling it served is gone.
How an agent is supposed to find the file now
Version 2's answer to discovery is two link relations that already existed. A rel of alternate with a type of text/markdown points at a page's markdown version. A rel of describedby points at the llms.txt file that covers that page. The proposal recommends rather than requires them, and its wording is that these links can be provided as HTML link elements, or as an HTTP Link response header.
The header form is the part worth pausing on, because it is the one that changes who can deploy this. The changes page states that the header form also works for non-HTML resources, such as the markdown files themselves, and can be added in web server or CDN configuration without modifying any pages. That is a deployment property rather than a semantic one, and it matters: a site owner who cannot touch a template, or whose pages are generated by a platform, can still emit these relations from a CDN rule. The Link header itself is not new and is not part of this proposal. It is defined by RFC 8288, Web Linking, authored by M. Nottingham and published in October 2017, which states that the Link header field provides a means for serialising one or more links into HTTP headers. Version 2 is reusing an existing mechanism rather than inventing a carriage format, which is the correct instinct and also the reason implementing it is cheap.
It is worth contrasting the shape of this with the one preference file that AI crawlers demonstrably do fetch. RFC 9309 fixes robots.txt at a single path on the authority, so discovery is not a problem there: a crawler knows where to look before it has seen a single page, and our robots.txt tester resolves that one file per named token without needing any hint from the site. llms.txt has never had that property, because version 1 allowed the file at the root or a subpath and gave a crawler no way to learn which. Version 2 does not fix that by fixing the path. It fixes it by having each page point at the file that covers it, which is a different and more flexible design, and one that only works if the page is fetched first.
That last clause is the constraint the marketing around this will skip. A relation on a page is only discovered by something that fetched the page. It does nothing for a crawler that only ever requests the site root, and it cannot help a system that never visits your site at all and answers from a model's weights or a third-party index, which is the awkward middle of answer engine optimisation generally. Discovery by link relation is an improvement for agents already reading your pages, and it is not a mechanism for reaching agents that are not.
Flow: Agent fetches an HTML page to Reads the link relations; Reads the link relations (rel=alternate) to Markdown version of that page; Reads the link relations (rel=describedby) to llms.txt covering that path; llms.txt covering that path to Follows the links it lists.
Two URL forms for a markdown page, and what a subpath now means
Version 1 specified one URL form for the markdown version of a page: .md appended to the full page URL, so page.html became page.html.md. Version 2 allows that form and also allows the extension to be replaced, so page.html may instead be served as page.md. The changes page is candid about why, saying that some publishing tools already replaced the extension and that version 2 blesses practice that had diverged. The proposal adds a rule for URLs with no file name in them, which should append index.html.md or index.md instead.
This is a small change with a real consequence, and it is the one place where the two versions of the proposal can disagree about the same site. Under version 1 there was one guessable URL for a page's markdown twin. Under version 2 there are two, which is exactly why the link relations matter more than the URL rule does: with a declared relation, a client does not have to try both, and with two permitted forms and no relation, it does. The proposal is coherent here. The convenience of a predictable URL was always a guess, and version 2 replaces the guess with a declaration while keeping the guessable forms as a fallback.
The subpath change is the other half of the same idea. Version 1 permitted an llms.txt file at a path within a site without defining what that scoped to. Version 2 states that a file covers the pages under its path and that where more than one file applies, agents should use the most specific one, so /docs/llms.txt covers everything in /docs/. The changes page names the case that motivated it: a site that only controls a path, such as a GitHub Pages project site, can now participate fully. That is a real gap closed, and it also quietly makes llms.txt behave more like a set of scoped manifests than a single site-wide file.
For a site owner, none of this changes what you would write in the file. It changes where you may put it and how many of them you may have, which for a large documentation set or a monorepo with several products is the difference between one unwieldy file and several accurate ones. If your stack generates pages rather than serving them from disk, the relevant question is whether your framework can emit the relation and the markdown twin at all, which for something like a Next.js site is a routing question rather than a content one. What a crawler receives from a generated page is the thing to check either way, and what GPTBot sees on a given URL is a different question from what your build produced.
Sample Illustrative, not a measurement of any real site.
The version 2 header form, on an ordinary HTML response
- GET /docs/page.html 200 text/html
- Link: </docs/page.html.md>; rel="alternate"; type="text/markdown" the markdown version of this page
- Link: </docs/llms.txt>; rel="describedby" the llms.txt that covers /docs/
- GET /docs/page.html.md 200 text/markdown
- GET /docs/llms.txt 200 text/markdown
What version 2 removed, and why a removal is the honest part
Version 1 said nothing about how llms.txt should be consumed, but it did describe a tool, llms_txt2ctx, for expanding a file into an LLM context. Version 2 removes that tooling from the proposal and replaces it with a statement of expectation: agents view or search the llms.txt to find what they need, then follow the relevant links, which should point to LLM-friendly content. The file itself stays small enough to fit in context, and the detail lives behind the links, fetched only when needed.
Read those two states next to each other and the direction is clear. Version 1 shipped with a concrete consumer, a program you could run that did something definite with the file. Version 2 has no consumer in the document and instead describes what agents are expected to do. That is a reasonable thing for a proposal to do once real agents exist, and the changes page is explicit that this reflects two years of adoption. It is also, read plainly, a proposal with a weaker claim to a mechanism than it had before, and the author does not hide that.
The Optional section is the casualty worth naming. In version 1 it carried mechanical meaning, telling context expansion tools what to leave out when they built a context from the file. With the tooling gone, the changes page states that Optional sections are still allowed and remain a useful convention for secondary links, but that they no longer carry mechanical semantics. So a field that used to instruct a program now signals intent to a reader. If you built an llms.txt around the assumption that Optional was load bearing, it is not any more, and nothing will tell you: the file still parses and the section still renders.
This is the pattern that turns up across nearly every machine-readable signal we look at, and it is why we keep writing about it. A sitemap tells a crawler where pages are rather than that they changed. A licence directive in robots.txt can state terms while changing no access. In each case the file is a declaration, and the enforcement, or the consumption, lives with whoever chooses to read it. Version 2 of llms.txt has moved further into that category rather than out of it, which does not make it useless and does mean the honest description of it is a convention rather than a protocol. For measuring AI visibility, the practical consequence is that the presence of the file remains a weak signal, and our registry of crawler tokens still records which vendors document reading which files, which is a shorter list than the proposals available to write.
Version 1
- Said nothing about how the file should be consumed
- Described llms_txt2ctx, a tool that expanded a file into an LLM context
- The Optional section told that tool what to omit
- A concrete program in the document, no stated expectation of agents
Version 2
- States the expectation directly: agents view or search the file, then follow the links
- The context expansion tooling is no longer part of the proposal
- Optional sections stay allowed, with no mechanical semantics
- An expectation of agents in the document, no program
What to check on your own site, and what this does not settle
The actionable half of version 2 is short, and it is a headers and templates job rather than a content job. If you already publish an llms.txt, the file itself needs no rewrite. What is new is that you can now declare where it is from the pages it covers, and declare where each page's markdown twin lives, using the two relations in a link element or a Link header. If you publish markdown twins, check which of the two permitted URL forms you use and be consistent, because a relation pointing at a URL that returns 404 is worse than no relation at all. If you serve documentation under a path you do not fully control, the subpath rule is the change that lets you participate.
We are obliged to apply that to ourselves, and the result is mixed. Every public page on lantad.co is already served as markdown at the same URL plus .md, which is the extension-replacement form version 2 now permits, and the markdown index lists all of them, with the home page at /index.md because /.md is not a servable filename. Those twins carry noindex and name the HTML page as canonical, so they are an alternate representation rather than a duplicate site. What we do not currently emit is either of the two relations version 2 recommends. Our layout publishes a rel of alternate for the RSS feed and nothing with a type of text/markdown, and the markdown endpoint sets no Link header. So on the specific thing this revision is about, our own site does the URL half and not the discovery half, as of this post. That is a gap in our implementation, not a criticism of the proposal, and we would rather write it down than let a reader assume otherwise.
What version 2 does not settle is the part that decides whether any of this is worth your afternoon. No AI crawler documentation we are aware of commits to fetching a page's link relations to locate an llms.txt, and a proposal cannot create that commitment on its own. The measured evidence on llms.txt influencing traffic was weak before this revision and no new measurement accompanied it. So the correct expectation for version 2 is that it makes a good idea implementable, not that it makes it effective, and the effect remains an open question that somebody will need to measure with a controlled cohort rather than assert, which is the shape of the crawlability study we publish.
Meanwhile the things that decide whether an AI system can read your pages at all have not moved. Whether the text a human sees is present in the served HTML before any JavaScript runs is still the largest single factor, which we score as prose parity. Whether your structured data survives to the crawler is still checkable. How your robots.txt resolves for a specific named token is still the thing that decides access, and our methodology sets out how we weight each of those and what we refuse to grade. A markdown twin is a good thing to publish and a poor substitute for a page a crawler can read, and the advice we give for being cited by ChatGPT starts in the same place it did last week. Our own crawler's conduct and opt-out are documented at our bot page, and it fetches robots.txt before anything else, because that is still the file every vendor agrees to read.
- Publish an llms.txt covering the pages you want agents guided through Unchanged by version 2. The file format is broadly the same; only its scoping rules gained a definition.
- Serve a markdown version of each page a client might want lantad.co serves every public page as markdown at the same URL plus .md, which is the extension-replacement form version 2 permits.
- Point at that markdown with rel=alternate and type text/markdown Not emitted on lantad.co today. The layout carries a rel of alternate for the RSS feed only.
- Point at the covering llms.txt with rel=describedby Not emitted on lantad.co today, in a link element or in a Link header.
- Keep robots.txt correct for each named crawler token Unrelated to this revision and still the only preference file every major AI vendor documents reading.
Lantad
Published .
A proposal changing version number is not usually worth a post. This one is, for a narrow reason. llms.txt is the file that site owners have been told to add for AI more often than any other, and until 10 August 2026 the document behind it had not moved since September 2024. The second version leaves the contents of the file broadly alone. What it changes is how anything is supposed to find the file in the first place, and discovery is the part of this that an outside scan can actually see. Everything below is read from the proposal page and its companion changes page, both fetched on 11 August 2026.
Common questions
Does llms.txt v2 change where the file goes?
Not the permitted locations, but it defines what a location means. Version 2 keeps the file at /llms.txt or at any subpath, and adds the rule that a file covers the pages under its path and that the most specific file applies when more than one does. Version 1 permitted subpath files without defining their scope.
Do I need to add rel=alternate and rel=describedby to my pages?
Only if you want agents that fetch your pages to find your markdown versions and your llms.txt without guessing. The proposal recommends rather than requires them, and they can be added as HTML link elements or as an HTTP Link header, the latter in web server or CDN configuration without editing any pages.
Is llms_txt2ctx still part of the llms.txt proposal?
No. Version 2 removes the context expansion tooling from the proposal and replaces it with a statement that agents view or search the file and then follow the links. The Optional section, which existed to tell that tooling what to omit, remains allowed as a convention but no longer carries mechanical semantics.
Does version 2 mean AI crawlers now read llms.txt?
No, and nothing in the revision claims that. Version 2 is a change to a proposal, published on 10 August 2026, and no crawler vendor documentation was updated alongside it. The measured evidence that publishing an llms.txt moves traffic was weak before this revision and no new measurement was published with it.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
llms.txt
llms.txt is a proposed plain-text markdown file placed at a site's root that gives AI systems a curated map of its most important content, so models can find and read what matters without parsing every page.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
Prose Parity
Prose Parity is Lantad's metric for the share of a page's rendered, human-visible text that is present in the raw HTML an AI crawler receives, where 100% means a crawler sees everything a human visitor sees.
