llms-full.txt: 32 of 44 sites served one, and the specification never names it

Is llms-full.txt part of the llms.txt specification?

No. The proposal is published at llmstxt.org, which shows a published date of 3 September 2024, a modified date of 10 August 2026 and a version 2 label. Read the whole of it and llms-full.txt is absent. What the document does specify is narrow and easy to check: a file named llms.txt at the root path of a site or at any subpath, an optional byte order mark, an H1 carrying the name of the project, which it states is the only required section, an optional blockquote holding a short summary, zero or more markdown sections of detail, and zero or more H2 sections holding file lists in which each entry is a markdown hyperlink with optional notes after a colon.

The one thing in the specification that does resemble a full text mechanism is a recommendation rather than a second file. It asks that pages provide a clean markdown version at the same URL as the original, either with .md appended or with the extension replaced. That is a per page convention, and a crawler reaching for it fetches one page at a time rather than one file holding everything. We measured whether any AI crawler actually takes that route in an earlier post, which found that ten AI crawlers fetched markdown by URL and none of them by content negotiation. This site publishes its own markdown twins for the same reason, and every page here has one at the markdown index.

So the difference between the two files is not a difference of degree. One is specified, with a required element and a defined shape a parser can validate. The other is a filename that acquired a meaning by being generated, and nothing anywhere states what it must contain, how large it may be, or what a client should do when it is too big to read. The version 2 revision of the proposal did change things, and it changed other things: it added two link relations and dropped the tool that read version 1. It did not add this file.

That matters for a practical reason rather than a pedantic one. A site owner deciding whether to add llms-full.txt cannot appeal to a standard to settle what good looks like, because there is no standard to appeal to, so the only available evidence is what the files on real sites actually do. Everything below is that evidence, and none of it is a recommendation the proposal makes.

How many sites that publish llms.txt also serve an llms-full.txt?

32 of the 44, on this frame, on this day. The frame is 54 hostnames of documentation and developer tool sites, chosen because they are where these files actually live, and it is a list somebody assembled rather than a probability sample of anything. Each hostname got one GET of /llms.txt and one of /llms-full.txt as LantadBot/1.0, with redirects followed and no retry. A response counts as serving the file only when it answered HTTP 200 with a text/plain or text/markdown content type, which is the distinction that decides several of these numbers.

Ten of the 54 served neither file as text, and they failed in three different ways worth separating. Four returned a plain 404 for both paths, being docs.astro.build, tailwindcss.com, huggingface.co and biomejs.dev, although two of those answered the 404 with a full HTML page of around 48,000 and 133,000 bytes rather than an empty body. Three returned HTTP 200 with an HTML document at both paths, being docs.deepgram.com, docs.llamaindex.ai and docs.replicate.com, which is a documentation site rendering its application shell for a path it does not have, and to a client expecting text that is worse than a 404 because it looks like success. platform.openai.com returned 404 for both. docs.cursor.com returned 200 with 537,795 bytes of HTML at /llms.txt and nothing at all at the other path. docs.mintlify.com returned HTTP 530 with a seventeen byte body for both, which is an edge error rather than a statement about the file, and it is the one result here we would not repeat a conclusion from.

The direction of adoption is the part that surprised us. Not one of the 54 hostnames served an llms-full.txt without also serving an llms.txt. If the full text file were being adopted as an independent idea, some sites would have skipped the index, and none did. That is consistent with both files arriving together from the same generator rather than from two separate decisions, which is the subject of a later section.

Discovery is where the picture gets worse. Of the 44 llms.txt files, only 14 mention the string llms-full.txt at all, and 22 of the 32 sites that serve one never name it in their own index. The specification makes llms.txt the entry point, and on two thirds of these sites that entry point does not lead to the larger file sitting next to it. The only way a client finds it is by guessing the filename, which is exactly the fragility that a defined discovery mechanism exists to remove. Four sites manage the inverse error and name a file they do not serve: nextjs.org and www.assemblyai.com point at an llms-full.txt that returns 404, and vercel.com and elevenlabs.io point at one that returns HTTP 200 carrying HTML, 2,634,912 and 1,062,686 bytes of it respectively.

Dead references inside these files are not a new finding here. An earlier scan of 40 hostnames found 31 real llms.txt files and one whose 75 links all returned 404, and the same class of error has now turned up one level further out, in the links between the two files rather than the links inside them. Whether any of this affects what an AI crawler can read is a separate question from whether the files are well formed, and the honest answer remains the one an earlier review of the evidence on llms.txt reached.

How big is an llms-full.txt file, and will a crawler read all of it?

The 32 files came to 145,304,405 bytes between them, with a median of 1,911,858 bytes. The median llms.txt beside them was 33,721 bytes, so the typical full file in this frame is about fifty seven times the size of its own index. The distribution is not a gentle slope either. developers.cloudflare.com served 50,820,143 bytes at that path, which is 4,699,094 words in one response. docs.anthropic.com served 34,305,754 bytes after two redirects that land on platform.claude.com, which is 3,586,183 words. Then supabase.com at 6,923,922, neon.com at 6,512,465 and ai-sdk.dev at 5,990,775.

Whether anything reads all of that has a documented answer for at least one crawler. Google's Googlebot documentation, last updated 3 February 2026, states that Googlebot crawls the first 2MB of a supported file type and the first 64MB of a PDF, and that it stops fetching once the cutoff is reached, with only the downloaded portion sent on for indexing. 15 of the 32 files exceed that. The count is the same whether 2MB is read as 2,000,000 bytes or as 2,097,152, which is a small mercy given that the abbreviation is ambiguous and the page does not disambiguate it. On the Cloudflare file, 2,000,000 bytes is 3.9 percent of what was served. We looked at the same limit from the page side in an earlier post on how Googlebot reads the first 2MB while three AI crawler documentation pages name no limit at all, and that asymmetry is the real problem here: for most of these crawlers there is no published number to check a 50MB file against.

What is inside the large files is the second half of the size question, and it is not all documentation prose. The Cloudflare file contains the phrase Skip to content 5,684 times. That is a navigation link, present once per page in the source, concatenated straight through into a file whose stated purpose is to hand an AI client clean text. Each page in it also opens with a YAML frontmatter block carrying description, title and an image URL, then that navigation link, then a three line blockquote pointing at a per directory llms.txt. None of that is wrong, and some of it is genuinely useful provenance, but a client truncating at 2MB spends its budget on whatever the first few thousand pages happen to be plus their chrome, and neither the client nor the site gets to choose which pages those are.

Ordering therefore does more work than it should. A file with no defined structure has no defined priority either, so the pages that survive a truncation are the ones the generator happened to emit first. A site that cares which parts of its documentation reach an AI client has more control through the per page markdown the specification actually recommends, or through a smaller hand written index, than through a single file large enough that every consumer has to cut it somewhere different. If you want to see what a fetcher receives from your own pages before worrying about either file, what GPTBot sees answers that directly, and the llms.txt tool checks the index itself.

Three files named full were no larger than the index beside them

The name carries a promise and three of the 32 do not keep it, in three different ways that are each instructive about what happens when nothing defines a file.

docs.expo.dev served 54,859 bytes at both paths, and the two responses are byte identical: both hash to the same MD5 digest, 9323db5044060b42a8321d187d996528. The site has one file published under two names, so a client fetching the second after the first spends the bytes twice and learns nothing. clerk.com is the sharpest case. Its llms-full.txt is 768 bytes, against 19,574 for its index, and it is not documentation text at all. It is a list of links to five other llms-full.txt files, one each for documentation, articles, blog, changelog and glossary, plus a link to a dashboard llms.txt on another hostname. A file named full that holds an index of full files is a reasonable engineering decision for a large site, and it is also precisely the kind of thing a client written against a filename convention will not expect. upstash.com is the quiet version of the same problem: at 5,368 bytes its full file is smaller than its 5,833 byte index, and reads as another sectioned link list rather than concatenated prose.

Two more, resend.com at 8,056 bytes and www.prisma.io at 8,180, are only slightly larger than their own indexes. The Resend file describes itself in its opening lines as a longer companion to its llms.txt and links back to it, which is honest about what it is. It is still nothing like a full text corpus, and a client sizing its expectations from the median of 1,911,858 bytes would be wrong about it by two orders of magnitude.

Put the three no larger cases together with the seven hostnames that answered a text path with HTML and the seven that returned 404 or 530, and the practical conclusion is that the filename predicts very little. A client cannot know from the path whether it is about to receive a 768 byte index, a duplicate of a file it already has, an application shell, or 50MB of concatenated prose with navigation text through it. That is the cost of a convention without a specification, and it is the same shape of problem as a site publishing rules in one file and contradicting them in another, which is what we found when 19 of 23 sites publishing an llms.txt said nothing about AI in their robots.txt.

None of this makes the file useless. A single fetch that returns a whole documentation set is genuinely convenient for an agent that wants to read everything and has the context to hold it, and several of these files are exactly that. The claim worth resisting is the one that treats the path as an interface.

Where llms-full.txt comes from, and what to check on your own site

A file that no specification defines, that 32 of 44 adopters serve, and that 22 of those 32 never link to, is unlikely to be the product of 32 independent decisions. The documentation platform is the common cause, and at least one of them describes the behaviour in its own documentation. Mintlify, whose docs sit at mintlify.com, states that it automatically hosts an llms.txt at the root of a project listing all available pages, that the file is always up to date and requires zero maintenance, and that it also automatically hosts an llms-full.txt combining the entire documentation site into a single file, with each page appearing as its title, source URL, description and full markdown content. Its page for this is at https://www.mintlify.com/docs/ai/llmstxt and it carries no date we could read.

The size rules on that page are the interesting part, because they are asymmetric. Generated llms.txt index files cannot exceed 100,000 characters, and when an index is larger the platform keeps llms.txt as a directory and moves groups of page links into generated markdown files under a /_llms/ path. No equivalent cap is stated for llms-full.txt. That is a coherent design: an index has a job that a cap protects, and the full file is defined as everything. It also explains the shape of our measurement without anybody having chosen it, because a documentation set that grows produces an index that stays small and a full file that grows without bound. Several hostnames in the frame return Vercel in the server header at these paths, which is where Mintlify runs, but a server header is not proof of a platform and we are not attributing any individual file here to one.

The thing to take from this is about defaults rather than about any vendor. Most of the 32 sites did not decide to publish a 2MB file. They chose a documentation platform, and the file appeared. That is the same mechanism we keep finding at the other layers: a hosting default wrote the robots.txt on two of the six Webflow sites that named GPTBot, and a platform default is why 35 of 36 Framer sites served markdown to a request asking for it. Knowing which files your stack publishes on your behalf is a prerequisite to having a view about them, and for the same reason it is worth knowing that a parallel convention exists at a well known path where the training default is deny.

Four checks follow from the measurement, and one caution. Fetch both paths yourself and look at the status and the content type, not just the status, because seven hostnames here answered 200 with HTML. If you serve an llms-full.txt, link it from your llms.txt, because 22 of 32 did not and the index is the only defined discovery point. Look at the size against the one published limit anybody offers, which is Google's 2MB, and accept that for most AI crawlers no such number is published. And check what a plain fetch of your actual pages returns before spending any effort on either file, because a documentation set that renders its text only in a browser has a prose parity problem that no amount of markdown at a side path repairs. The method behind each of these checks is written up on the methodology page, and the rest of this blog's measurements are indexed on the research page.

The caution is that none of this tells you the file is read. Nothing in this scan observed a crawler: no access log, no server side record, no evidence that any AI client requested either path on any of these 54 sites. Chrome's agentic browsing audit does check for the index, which we covered when it shipped and Google Search continued to ignore the file, and that remains the honest state of the evidence. Each hostname was asked once, on one day, from one network location, as one identified user agent, and sites that vary their response by user agent or geography were seen from a single vantage point. Byte counts are what the transfer reported and were not recomputed against a second fetch. The 54 hostnames are a list chosen for the likelihood of finding these files, which makes every proportion here a statement about this frame and not about the web, and the direction of that bias is upward: a random sample of sites would find far fewer of both files. What the scan supports is that the second file is widely deployed, undefined, frequently large, sometimes not what its name says, and usually unlinked. That is enough to decide whether to add one, and it is less than a standard would give you.