Where does the llms.txt file go?

At the root of your domain, served as plain-text markdown at /llms.txt, for example https://example.com/llms.txt. An optional companion file, llms-full.txt, can hold the expanded content of the pages you list.

Does llms.txt replace robots.txt or a sitemap?

No. robots.txt controls which paths a crawler may fetch, and a sitemap lists every URL you want indexed. llms.txt is a curated summary of your most important pages for AI systems. Keep all three; llms.txt is additive, and the reliable win is still making your real pages readable to AI crawlers.

Glossary / llms.txt

llms.txt

Also: /llms.txt, llms.txt file, llms-full.txt

llms.txt is a proposed plain-text markdown file placed at a site's root that gives AI systems a curated map of its most important content, so models can find and read what matters without parsing every page.

llms.txt is a plain-text markdown file that a site publishes at its root to tell AI systems which pages matter most and what they contain. It was proposed by Jeremy Howard of Answer.AI in September 2024 as a way to give language models a clean, curated entry point into a site instead of leaving them to parse full HTML.

The file is a proposal, not an official standard, and using it is voluntary. It does not replace robots.txt or a sitemap, and it does not change what an AI crawler can actually read on your live pages. It works as a supplement once the underlying content is already reachable and readable.

What an llms.txt file looks like

An llms.txt file is a markdown document served at the root of a domain, for example https://example.com/llms.txt. The proposal defines a specific structure: an H1 with the site or project name, an optional blockquote that summarizes what the site is, then H2 sections that list links to key pages, each with a short description. An optional section can hold secondary links that are useful but not essential. A common companion file, llms-full.txt, gathers the expanded text of those pages in one place. The format is readable by both people and machines, and it is meant to be curated by hand rather than generated from every URL on the site.

How llms.txt differs from robots.txt and sitemaps

These three files answer different questions. robots.txt is an established standard that tells a crawler which paths it is allowed to fetch; it controls access and says nothing about what to read first. A sitemap is an XML file that lists every URL you want indexed, built for completeness and consumed by search crawlers. llms.txt differs in intent: it is a short, curated markdown file that points AI systems at your most important content and describes it in plain language. robots.txt and sitemaps are widely supported and honored; llms.txt is a proposal, and following it is voluntary. It adds to those files rather than replacing either one.

Why llms.txt matters for AI visibility

AI systems work from limited context and from HTML that is often heavy with navigation, scripts, and markup. In principle, a curated llms.txt can reduce that noise by naming your key pages and describing them in clean prose. It does not change the underlying requirement: for a model to use a page, it first has to fetch that page and read its text. An llms.txt that links to pages an AI crawler cannot read points at content that never arrives. Being fetched and read is the prerequisite for being cited, and an llms.txt entry does not change that.

Is llms.txt required, and how to get value from it

llms.txt is not required, and no major AI vendor has publicly confirmed that its crawlers use the file to select or cite content. Treat it as an optional supplement, useful once the pages it points to are already readable. Lantad does not score llms.txt itself. It measures what an AI crawler actually receives from your live pages: how much of your visible text reaches a raw, non-JavaScript fetch (Prose Parity), whether named AI crawlers are allowed (Access), whether the page loads cleanly (Retrievability), and whether headings and structured data give the content shape (Structure and Schema). Those results tell you whether an llms.txt would point at content a crawler can use or at pages that return an empty shell. Fix the fundamentals first, then add llms.txt as a curated index on top.

Common questions

Is llms.txt required?: No. llms.txt is a proposed convention, not an official standard, and publishing it is voluntary. No major AI vendor has publicly confirmed that its crawlers use the file to select or cite content, so treat it as an optional supplement rather than a requirement.
Where does the llms.txt file go?: At the root of your domain, served as plain-text markdown at /llms.txt, for example https://example.com/llms.txt. An optional companion file, llms-full.txt, can hold the expanded content of the pages you list.
Does llms.txt replace robots.txt or a sitemap?: No. robots.txt controls which paths a crawler may fetch, and a sitemap lists every URL you want indexed. llms.txt is a curated summary of your most important pages for AI systems. Keep all three; llms.txt is additive, and the reliable win is still making your real pages readable to AI crawlers.

See if AI can read your site

Paste a URL and see exactly what AI crawlers can and cannot read, with a graded report and ranked fixes. Free, no signup.

Scan your site free How the score works