# llms.txt example: 31 real files, and one whose 75 links all 404 > Lantad requested /llms.txt once from each of 40 hostnames on 5 September 2026, chosen because they were the sites most likely to publish one. 31 returned a plain text file, 7 returned HTTP 404, and 2 returned HTTP 200 with an HTML document. The 31 files hold 5,567 markdown links between them, and every one of the 75 links in the file at docs.mistral.ai returned HTTP 404. - Canonical page: https://lantad.co/blog/llms-txt-example-31-files-and-75-dead-links - This file: https://lantad.co/blog/llms-txt-example-31-files-and-75-dead-links.md - Last substantive update: 2026-09-05 ## Key facts - **Published:** 2026-09-05 - **Category:** Findings - **Author:** Lantad - **Length:** 3429 words - **Takeaway 1:** Lantad requested /llms.txt once from each of 40 hostnames on 5 September 2026, sent as LantadBot/1.0 with redirects followed. 31 returned a plain text file, 7 returned HTTP 404, and 2 returned HTTP 200 carrying an HTML document, the larger of them 513,949 bytes. - **Takeaway 2:** Every llms.txt example in that set of 31 is an index rather than a summary: the files hold 5,567 markdown links between them, the median file holds 116, and the largest, at docs.anthropic.com, holds 700. - **Takeaway 3:** All 75 unique links in the llms.txt served by docs.mistral.ai returned HTTP 404 when Lantad fetched every one of them on 5 September 2026. - **Takeaway 4:** The llms.txt format specification, modified 10 August 2026, makes an H1 the only required section. Of the 31 files, 30 carry exactly one H1, 22 carry the recommended blockquote summary, and docs.cohere.com carries no H1 at all. - **Takeaway 5:** Of 150 links sampled from those files and fetched on 5 September 2026, 139 returned HTTP 200, and 17 of those 139 returned an HTML document rather than the markdown the format exists to point at. ## Summary Anybody about to write one of these files searches for an llms.txt example first, and what comes back is a template. A template describes the shape the format asks for. It does not say what the sites that already publish one actually shipped, whether the links inside those files still resolve, or what an agent receives when it follows them. Those are answerable questions, and the answers are measurements rather than opinions, so we went and took them. Our own file is at the same path as everybody else's and our [llms.txt tool](https://lantad.co/tools/llms-txt) writes one, which is a reason to be careful here rather than a reason to be encouraging. On 5 September 2026 we requested /llms.txt once from each of 40 hostnames, sent as LantadBot/1.0 with redirects followed, a 20 second timeout and no JavaScript executed, from the same client whose [conduct policy and address](https://lantad.co/bot) we publish. The 40 are not a random sample of the web and were never meant to be. They are AI model vendors, developer documentation platforms and JavaScript frameworks, picked because they are the sites most likely to have adopted [llms.txt](https://lantad.co/glossary/llms-txt) early. That bias runs in one direction and it decides how every number below should be read: what follows is closer to a best case than to an average. Of the 40, 31 returned a plain text file, 7 returned HTTP 404, and 2 returned HTTP 200 with an HTML document. We then parsed the 31, counted what they hold, and fetched 150 of the links inside them. ## What does a real llms.txt example look like? Start with what the format asks for, because the gap between that and what shipped is the interesting part. [The llms.txt specification](https://llmstxt.org/), which carries a modified date of 10 August 2026, sets out an ordered structure: an optional byte order mark, an H1 with the name of the project or site, which it calls the only required section, a blockquote with a short summary, zero or more markdown sections of any type except headings, and zero or more sections delimited by H2 headers containing file lists. A file list entry is a markdown list item holding a required markdown hyperlink, optionally followed by a colon and notes. A section named Optional is reserved by convention for links an agent can skip when it needs shorter context. Measured against that, the 31 files are close to conformant on the parts that are required and looser on the parts that are not. Thirty of the 31 open with exactly one H1, and almost all of them are the bare product name: "# Vercel", "# Bun", "# Perplexity", "# Model Context Protocol". The exception is docs.cohere.com, a 662 byte file holding two links that opens directly with its blockquote and carries no H1 anywhere, so the one element the specification calls required is the one element it is missing. Twenty two of the 31 carry the recommended blockquote summary and nine do not, which includes some of the largest files in the set. The H2 file lists are where the drift shows. One file, at modelcontextprotocol.io, carries 349 links and no H2 heading at all, so every link sits in an undelimited run with nothing to tell an agent which links belong together or which it could skip. Ten of the 31 hold at least one link that is not inside a markdown list item, and two are badly out: docs.mistral.ai puts none of its 75 links in a bullet, and zapier.com puts 24 of its 116 in one. That matters less for a human reader than it looks, and more for a parser, because a file list is defined as a list and a link floating in a paragraph is not one. Content type is the last surprise, and it goes the other way from what the name suggests. Twenty eight of the 31 files were served as text/plain and only three as text/markdown, being docs.stripe.com, docs.github.com and docs.render.com. Nothing in the specification requires a particular content type, so this is not a defect. It is worth knowing if you are writing a client, because [the second version of the proposal leaned on discovery](https://lantad.co/blog/llms-txt-v2-link-relations-and-discovery) rather than on content type, and a client that filters on text/markdown would reject nine files out of ten here. The largest file in the set, [Anthropic's developer documentation index](https://docs.anthropic.com/llms.txt), is 73,533 bytes of text/plain. ## An llms.txt is an index, not a summary The file is often described as a summary of a site for a language model, and none of these files is that. Across the 31, the file lists hold 5,567 markdown links. The median file holds 116, the largest holds 700 at docs.anthropic.com, and the smallest holds 2 at docs.cohere.com. The files themselves are small: 637 bytes at the low end, which is llmstxt.org's own file, 90,052 bytes at the high end at docs.stripe.com, a median of 15,906 bytes, and 826,499 bytes for all 31 together. That is the whole point of the format working as designed. What it means in practice is that reading the file is one request and acting on it is not. At the median that is 116 further requests to see what the file points at. At docs.anthropic.com it is 700. An agent that fetches the index and stops has learned a table of contents; an agent that follows the index has done a small crawl, and the file has saved it discovery rather than saved it bandwidth. That is the same distinction we drew when six sites [declared 200,712 URLs in sitemaps against 985 links found by crawling them](https://lantad.co/blog/scan-a-website-for-all-pages-985-linked-200712-declared): a manifest is cheap to publish and expensive to honour, and the cost lands on the consumer rather than the publisher. Which makes the comparison to a sitemap the right one, and it is not flattering to either file. A sitemap declares URLs and optionally when each last changed, and we have written before about [what a lastmod value actually asserts](https://lantad.co/blog/sitemap-lastmod-is-an-assertion). An llms.txt declares URLs, an anchor for each, and optionally a note. Neither tells a consumer what changed since it last looked, so an agent has no way to fetch the difference and must either refetch everything or trust nothing. [Cloudflare's own file](https://developers.cloudflare.com/llms.txt) is a good illustration of the shape at 15,906 bytes and 106 links, all of them grouped under nine H2 sections, which is about as navigable as this format gets. Two files link entirely by relative path, at vuejs.org with 94 links and docs.solidjs.com with 271, and across all 31 files 456 of the 5,567 links are relative and 5,111 are absolute. Both forms are legal markdown and both resolve, but they resolve against the location of the file, which is a detail a client has to get right and a publisher never sees go wrong. We ship [a markdown version of every page on this site](https://lantad.co/md) for the same reason these files exist, and the index for it carries absolute URLs for exactly this reason. ## Do the links inside an llms.txt resolve? Publishing an index creates a maintenance obligation, and an index nothing fetches is an index nobody notices breaking. So we followed the links. For each of the 31 files we took the first five unique links in file order, resolved relative paths against the file's own location, and fetched each one with the same client and the same timeout. Two files hold fewer than five links, so the sample is 150 requests rather than 155. First five in file order is a deterministic rule and not a random one, which is a limitation worth stating plainly: it favours whatever each publisher put at the top. Of the 150, 139 returned HTTP 200 and 11 did not. That is the reassuring half of the finding and it is genuinely reassuring: 27 of the 31 files returned 200 on every link sampled from them. The 11 failures came from four files. Two links at docs.github.com returned HTTP 400, both of them API endpoints rather than documents, at /api/article and /api/article/body. One link in the file at bun.sh pointed at bun.com/docs/index.md and returned 404, a cross domain link to a second domain the project also runs. Three links from www.pinecone.io failed, two with HTTP 405 and one, at docs.pinecone.io/skill.md, with 404. The fourth file is the reason this post has a number in its title. All five links sampled from docs.mistral.ai returned 404, so we fetched the rest: every one of the 75 unique links in that file returned HTTP 404 on 5 September 2026, and so did the same paths with the .md extension removed. The file itself is 14,658 bytes, returns HTTP 200, opens with a correct H1 and reads as a perfectly ordinary index of an agents and connectors documentation set. Nothing about fetching the file tells you that none of it resolves. That is a statement about one file on one day, and a publisher can fix it in an afternoon, which is precisely why it is worth measuring rather than assuming. It is also the failure mode the format invites. A robots.txt is short, hand edited and read constantly, so an error in it surfaces. An llms.txt is generated from a documentation build, is hundreds of lines long, and by the best available evidence is fetched by almost nobody, so a route change that invalidates every link in it produces no error anywhere. Nothing in [how we reach a verdict](https://lantad.co/methodology) would have caught this either, because our scanner checks that the file exists and parses, not that its contents are alive. We found the same silent gap when we checked whether [any AI crawler asked for markdown through content negotiation](https://lantad.co/blog/no-ai-crawler-used-content-negotiation) and none did. ## What comes back when an agent follows the links A 200 is not the end of the question, because the reason to publish this file at all is to hand a machine clean prose instead of an application shell. The specification is explicit about the mechanism: a site should provide a clean markdown version of a page at the same URL as the original, either with .md appended or with the extension replaced. So the useful test is not whether a link resolves but what it returns. Across all 31 files, 4,557 of the 5,567 links point at a URL ending in .md, which is 81.9 percent. Six of the 31 files contain no .md link at all: developers.cloudflare.com, docs.github.com, fly.io, hono.dev, svelte.dev and docs.cohere.com. Those files are indexes of ordinary HTML documentation pages, which is legal and is a different product from the one the format describes. An agent following them ends up fetching the same pages a crawler would have fetched anyway, and it still has to strip navigation out of them, which is the work [prose parity](https://lantad.co/glossary/prose-parity) exists to measure. Following the sample bears that out. Of the 139 links that returned 200, the declared content types were 85 text/markdown, 35 text/plain, 4 application/json and 15 text/html. Sniffing the bodies rather than trusting the headers found 17 HTML documents among the 139, being those 15 plus two at docs.solidjs.com that declared text/markdown and returned an HTML page. So of 150 links followed, 122 returned something that was not an HTML document, 17 returned HTML, and 11 returned nothing usable. The four JSON responses are interesting rather than wrong: they include an OpenAPI document and a /.well-known/ai-catalog.json, which are machine readable by design and are simply not markdown. That 17 matters more than its size suggests, because an HTML page reached through an llms.txt link is the exact case the file was written to avoid, and the consumer has no way to know in advance. It is the same class of problem as an AI crawler meeting a client rendered page: we measured that [only 2 of 9 crawler operators state whether their crawler executes JavaScript](https://lantad.co/blog/do-ai-crawlers-render-javascript-two-of-nine), so a publisher cannot reason about it either. Markdown is not automatically safer content, which is worth saying because we have also found [advertising and tracking markup inside the markdown AI crawlers read](https://lantad.co/blog/ads-inside-the-markdown-ai-crawlers-read). And a format's file extension is not a guarantee of anything, which is why the observation that [Google indexes twenty file types while AI crawler documentation names none](https://lantad.co/blog/google-indexes-twenty-file-types-ai-docs-name-none) still applies here. [OpenAI's own index](https://developers.openai.com/llms.txt) is a compact 5,853 bytes with 38 links, 18 of which end in .md and 20 of which do not. ## What this measurement does not establish The limits are load bearing here, because a post that counts files is easy to read as a survey and this is not one. It is not an adoption rate. The 40 hostnames were chosen because they were likely to publish the file, so 31 of 40 is an upper bound on a deliberately favourable set and says nothing about the web. It is also one fetch per host at one moment on one day. A 404 today can be a file tomorrow, a dead link can be repaired in an afternoon, and every figure above should be read with its date attached rather than as a standing property of any company named. It observes no consumer. Nothing here establishes that any [AI crawler](https://lantad.co/glossary/ai-crawler) or agent fetched any of these files, because we watched no access logs but our own. The best evidence on that question is not ours and it is not encouraging: we published it in full when we set out [what the evidence actually says about llms.txt](https://lantad.co/blog/what-the-evidence-says-about-llms-txt), where a 137,210 domain study found 97 percent of valid files were never requested at all in May 2026. The one documented consumer we have found is a browser audit rather than a search engine, and we wrote that up when [Chrome's agentic browsing checks looked for the file while Google Search ignored it](https://lantad.co/blog/lighthouse-agentic-browsing-checks-llms-txt). The link sample is five per file in file order, not a random draw and not a full audit, so the 11 failures are a lower bound on the number of broken links in these files rather than an estimate of it. The one file we did audit fully was the one already failing, which biases that number upward and is stated here rather than buried. We also did not check whether any file's contents match the documentation the site actually publishes, so a file whose links all return 200 could still be pointing at a stale subset. And publishing the file changes nothing about whether a crawler can read your pages, which is the layer underneath it. That is the distinction we set out in [the two layers that decide whether AI can read your site](https://lantad.co/blog/two-layers-decide-if-ai-can-read-your-site): access first, then what the response actually holds. An llms.txt sits above both. If you want one, the honest case for it today is that it costs little and may pay later, not that anything measured is reading it, and [the specification itself](https://llmstxt.org/) is short enough to read before you decide. ## Questions and answers **What is an llms.txt file supposed to contain?** An H1 with the name of the project, which the specification calls the only required section, then optionally a blockquote summary, then zero or more H2 sections holding markdown lists of links. Each list item carries a markdown hyperlink and may add a colon and a note. The specification, modified 10 August 2026, also asks that the linked pages be served as clean markdown at the same URL with .md appended or substituted. **Do the links inside a real llms.txt work?** Mostly, on this sample. Of 150 links fetched on 5 September 2026, five from each of 31 files, 139 returned HTTP 200 and 11 did not. The failures were concentrated: all 75 links in the file at docs.mistral.ai returned 404, and the remaining six failures came from three other files. **How many links does an llms.txt usually hold?** Across the 31 files measured on 5 September 2026 the median was 116 links, the largest held 700 and the smallest held 2. The files themselves ranged from 637 to 90,052 bytes. Reading the file is one request; following what it points at is 116 more at the median. **Should I publish an llms.txt?** This post does not answer that, and no measurement here supports a recommendation either way. What it does establish is that the file is an index rather than a summary, that a broken index fails silently because almost nothing fetches it, and that 6 of the 31 files measured link only HTML pages rather than the markdown the format exists to point at. If you publish one, the maintenance obligation is real and nothing will tell you when it breaks. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot