BlogFindings
agents.md: 46 of 1,419 sites served one at the web root, and 42 of the 46 were Shopify
The AGENTS.md format is specified for the root of a code repository, and the site that publishes it calls it a README for agents. Shopify's developer changelog of 28 May 2026 states that every store includes a default agents.md file accessible at /agents.md, and that /llms.txt and /llms-full.txt point to that content by default. Lantad asked all 1,419 hostnames in this repository's two committed corpus seed files for /agents.md on 4 November 2026. 170 answered HTTP 200, but 121 of those were hosts that answered the same way for a control path that does not exist, and 3 more returned an HTML document. 46 served a file, and 41 of the 46 carried the Shopify storefront default's own heading.
Something else has happened in parallel, and it is a platform decision rather than a standards one. Shopify's developer changelog entry titled Customize /llms.txt, /llms-full.txt and /agents.md, dated 28 May 2026 and published at shopify.dev/changelog/posts/customize-llmstxt-llms-fulltxt-and-agentsmd, states that a store includes a default agents.md file accessible at /agents.md, and that the paths /llms.txt and /llms-full.txt also point to that content by default. So on one platform, serving an agents.md over HTTP is now the default behaviour of every storefront, whatever the format's own specification says about repositories. That makes the adoption question answerable rather than rhetorical, and this blog has measured the neighbouring files before, from the one agent card in 1,419 sites to what the evidence actually says about llms.txt. So Lantad asked all 1,419 hostnames in its two committed corpus seed files for /agents.md on 4 November 2026 and counted what came back.
In short
- An agents.md file is specified for a code repository, not a website. The agents.md site, read at source on 4 November 2026, says to create the file at the root of the repository and calls it a README for agents, and GitHub's custom instructions documentation classifies AGENTS.md as a repository custom instruction alongside CLAUDE.md and GEMINI.md.
- Lantad requested /agents.md once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 4 November 2026, as LantadBot/1.0 with redirects followed and no JavaScript executed. 170 answered HTTP 200 and 46 served an actual file. 869 answered 404 and 227 answered 403.
- The gap between 170 and 46 is almost entirely one measurement error that a status code alone cannot catch: 121 of the 170 hosts answered HTTP 200 for a control path invented for this run and present on no site, and 3 more returned an HTML document at /agents.md rather than a text file.
- 42 of the 46 files are attributable to one platform. 41 opened with the heading Agent Instructions followed by the store's own name and pointed the reader at a Shop skill file, which is the shape of the default Shopify documents, and shopify.com served a bespoke file of its own. Only fly.io, metalbear.com, coinbase.com and zooplus.de published one independently of it.
- On the same 1,419 hostnames the same day, 297 served a readable /llms.txt against 46 for /agents.md, every one of the 46 also served an llms.txt, and exactly 1 site served a file at /.well-known/agents.md.
| Path | Where its own source says it belongs | Answered 200 | Served a real file |
|---|---|---|---|
| /agents.md | The root of a code repository, per agents.md | 170 | 46 |
| /llms.txt | The root path of a website, per the llms.txt spec | 422 | 297 |
| /.well-known/agents.md | No published source places it here | 122 | 1 |
What is agents.md, and where does its specification say it lives?
The format is deliberately thin. The agents.md site describes a plain Markdown file with no required fields and no fixed headings, holding the detail that would clutter a README: how to build the project, how to run its tests, what conventions to follow. It says the file emerged from collaborative efforts across the AI software development ecosystem, naming OpenAI Codex, Amp, Jules from Google, Cursor and Factory, and that it is now stewarded by the Agentic AI Foundation under the Linux Foundation. The site states the format is used by over 60 thousand open-source projects and links a GitHub code search as the evidence for that figure, which is worth reading as what it is: a count of files inside repositories, not a count of websites.
Every location statement on that page is about a directory tree. The file goes at the root of the repository. A large monorepo may add nested files, and an agent reads the nearest one in the directory tree, so the closest takes precedence. GitHub's documentation is consistent with that and adds a precedence order of its own: on GitHub.com it ranks agent instructions below path-specific and repository-wide instruction files, with personal instructions above all of them. Nothing in either source describes a server, a URL, or a public path. That matters because the two conventions a site owner is likely to confuse have genuinely different homes: the llms.txt specification is explicit that its file sits at the root path /llms.txt of a website or at any subpath, which is a statement about HTTP, while AGENTS.md is a statement about a filesystem.
It is also worth being precise about who reads what, because the reader is the part that decides whether publishing a file does anything. An AI crawler fetching your pages for a search index is not a coding agent working in your repository, and neither is a shopping agent trying to buy something. OpenAI's crawler documentation, read on 4 November 2026, names four user agents, OAI-SearchBot for search, GPTBot for training, ChatGPT-User for user-initiated fetches and OAI-AdsBot for checking ad landing pages, and it does not mention AGENTS.md at all. The only file it tells a site owner to manage is robots.txt. That is the gap this measurement sits in: a format with real adoption in repositories, a platform serving it over HTTP by default, and no crawler documentation asking for it.
| Question | AGENTS.md | llms.txt |
|---|---|---|
| Where its source puts it | Root of a code repository, plus nested files in a monorepo | Root path /llms.txt of a website, or any subpath |
| Who it addresses | AI coding agents working in a checkout | AI agents reading a site for an overview and links |
| Who stewards it | Agentic AI Foundation under the Linux Foundation | Published at llmstxt.org |
| Named in OpenAI's crawler docs | No | Only as a link to OpenAI's own docs index |
Do websites serve an agents.md file at the web root?
Lantad requested https://host/agents.md once for each of the 1,419 hostnames in this repository's two committed corpus seed files on 4 November 2026. The two frames are worker/seeds/corpus-seeds-platform.json, holding 392 hostnames in ten platform strata, and worker/seeds/corpus-seeds-industry.json, holding 1,027 in eight industry strata, with no host in both. Every request went out as LantadBot/1.0 with redirects followed and no JavaScript executed, which is the single-request method set out in the method behind every score and on the scanner's own crawler page.
The status distribution is the first half of the answer. 869 hosts answered 404 and 227 answered 403, which together account for more than three quarters of the corpus. 170 answered 200. 61 answered 503 and 23 answered 400. 25 requests failed at the connection or TLS layer and 11 timed out at twenty seconds. The remaining 20 were spread across 202, 429, 406, 500, 502, 401, 418, 451, 490, 498 and 529, which accounts for all 1,419. A 404 here is the honest answer rather than a fault: a site that has never heard of the file correctly says it does not have it, and 869 sites saying so cleanly is a sign of servers behaving properly.
The second half is that 170 is not the number of files, and the difference is large enough that reporting the status code alone would have overstated adoption by a factor of nearly four. 46 of the 1,419 served an actual file. Against the same hosts on the same day, 297 served a readable /llms.txt, so the older convention is roughly six times as widely published as the newer one even though the newer one is the default on a major commerce platform. Every one of the 46 that served an agents.md also served an llms.txt, so not a single site in this corpus has adopted the new path in place of the old one. The companion file fared worse still: llms-full.txt reached 32 of 44 sites in an earlier measurement of a narrower frame, and here exactly one host served anything at /.well-known/agents.md, a path no published source asks for.
Why did 170 HTTP 200 responses come to only 46 files?
Because a 200 is a claim by the server and not evidence about a file. Before counting anything, Lantad asked every host that had answered 200 for a second path invented for this run, /lantad-control-7f3a9c2e-not-a-real-path.md, which exists on no site anywhere. 121 of the 170 answered that control with HTTP 200 as well. Those hosts answer 200 for every path, so their response at /agents.md carries no information about whether they publish one, and counting them would have been counting a routing default. This is the soft 404 shape this blog has measured directly before, when 125 of 1,094 hosts answered 200 for a page that was not there, and it is the single most important control in a file-presence measurement.
Three more of the 49 that survived the control returned an HTML document at /agents.md: warbyparker.com, klook.com and viarail.ca. In each case the control path behaved correctly, returning 403 or 404, so these are not blanket catch-alls. They are hosts serving a page where a Markdown file was requested, which a crawler reading the body rather than the status would have to discard anyway. Removing them leaves 46 responses whose body was not HTML, and those are the files reported throughout this post. Of the 46, 42 arrived as text/markdown and 4 as text/plain.
The .well-known result needed a second control of its own, and it changed the answer. 122 hosts answered 200 at /.well-known/agents.md, and 19 of those survived the root-level control. But a host can be a catch-all inside one directory without being one at its root, so Lantad probed each of the 19 with a control path under /.well-known/ as well. 16 of the 19 had returned HTML. Of the three that had not, healthline.com answered the same 38 byte JSON document at both the real path and the invented one, which is the same behaviour that made it a false positive in the agent card measurement, and spectator.co.uk returned zero bytes at both. That leaves metalbear.com as the only host in the corpus serving a genuine file at /.well-known/agents.md. One site is not a convention, and the honest reading is that this path is not in use rather than that it is slightly in use.
Flow: 1,419 hostnames asked to 170 answered HTTP 200; 1,419 hostnames asked (no file) to 1,249 did not, 869 of them 404; 170 answered HTTP 200 (catch-all) to 121 also served the control path; 170 answered HTTP 200 to 49 survived the control; 49 survived the control (not a file) to 3 returned an HTML document; 49 survived the control to 46 served a real file.
Why were 42 of the 46 files Shopify stores?
Because one platform turned the file on everywhere, and a default applied across a platform outruns a convention adopted by hand. 41 of the 46 files opened with a first-line heading reading Agent Instructions followed by the store's own name, and all 41 pointed the reader at a Shop skill file for browsing and checkout. Their sizes cluster tightly, from 4,252 to 5,820 bytes, which is what a generated file looks like. Added to shopify.com itself, which serves a bespoke 12,523 byte file of its own, 42 of the 46 are attributable to that one platform. The remaining four published independently: fly.io and metalbear.com serve genuine coding-agent onboarding instructions over HTTP, coinbase.com serves a file opening with YAML front matter describing a Coinbase agent skill, and zooplus.de serves a German-language file for shopping agents that explicitly positions itself alongside robots.txt for crawling rules and llms.txt for a content overview.
The platform's own documentation predicts a second observable, and it holds. Shopify's changelog says the paths /llms.txt and /llms-full.txt point to the agents.md content by default, and that a request falls back to the agents.md template before the generated default. If that is true at scale then the two paths should return the same bytes on most stores, and they do: 41 of the 46 hosts returned an identical first 4,000 bytes at /agents.md and at /llms.txt. Those two groups of 41 are not the same 41, and the difference is the interesting part. shopify.com serves one bespoke file identically at both paths. magicmind.com serves the storefront default's heading at /agents.md but something different at /llms.txt, which is what overriding one of the three templates the changelog documents looks like from outside. So 32 storefronts are running the untouched default, one has been customised, and the rest of the corpus has not engaged with the file at all.
Two measurement details in that stratum are worth stating because they would quietly distort a smaller sample. Of the 34 stores in the shopify-dtc platform stratum, 33 served the file and one answered 404, so the default is close to universal on that platform but not absolute. And gymshark.com resolved only through a cross-host redirect, with the final response coming from us.checkout.gymshark.com, which is the kind of hop that makes a file look absent to anyone testing the apex with redirects disabled. This blog measured that pattern on robots.txt and found 122 of 391 redirects crossing hosts. The wider point about this platform is one the corpus has already shown from a different angle, when Shopify stores served 1,112 products as open JSON: what a storefront exposes to an agent is mostly decided by the platform, not the merchant, and the per-stack detail for that one is in the Shopify guide.
| What the file was | Hosts | Content type | Same first 4,000 bytes at /llms.txt |
|---|---|---|---|
| Shopify storefront default shape | 41 | text/markdown | 32 of 41 |
| shopify.com's own bespoke file | 1 | text/plain | Yes |
| Coding-agent instructions, published independently | 2 | text/markdown and text/plain | No |
| Agent skill definition with YAML front matter | 1 | text/plain | No |
| Shopping-agent file written for the site | 1 | text/plain | No |
Should you add an agents.md file to your website?
On the evidence here, not for the reason it is usually suggested. Lantad has measured no effect of any agents.md file on any AI answer, and nobody else has published one either, so a recommendation to add one rests on the same thin ground as the llms.txt recommendations this blog examined when it concluded that the evidence for llms.txt does not exist yet. What has changed since that post is only that a platform now ships the file by default, which is a fact about Shopify rather than a finding about crawlers. If you are on Shopify the file is already there and the question is whether its contents are right. If you are not, writing one is a bet, and it should be described as a bet.
There is a narrower case where the file is doing real work, and the four independent publishers show its shape. fly.io and metalbear.com are selling to developers whose coding agents will read a URL handed to them in a prompt, and a file at a stable public address is a reasonable way to serve that. That is a different job from being cited in a search answer, and it is worth separating, because the two get conflated under the same heading. Agent-facing surfaces generally have this property: the corpus found that WebMCP tools need a browser to be discovered at all, and that MCP labels stop at the MCP server. A file only matters to the agent that is actually told to read it.
The thing worth checking is cheaper and it is not about this file. 227 of the 1,419 hosts answered 403 to a plain request for a text file from an identified crawler, and 121 answered 200 for a path that does not exist. Both of those say more about whether an AI system can read a site than any discovery file does, and both are measurable today rather than speculative. The corpus numbers behind that are in the crawlability study, the per-crawler tokens are listed on the AI crawlers reference, and if you want to see what your own site publishes at these paths the llms.txt tool reads them for you. The ordering that matters for generative engine optimization has not changed: a page a crawler cannot fetch is not improved by a Markdown file describing it, and this blog found the same thing when it measured content negotiation for Markdown on 718 hosts.
- No crawler documentation asks for an agents.md OpenAI's crawler documentation, read on 4 November 2026, names four user agents and mentions only robots.txt as the file a site owner manages. It does not mention AGENTS.md.
- The specification is about repositories, not web roots The agents.md site says to create the file at the root of the repository and describes nested files resolved to the nearest in the directory tree. GitHub classifies it as a repository custom instruction.
- If you are on Shopify, you already serve one Shopify's changelog of 28 May 2026 says every store includes a default agents.md at /agents.md. Lantad found 33 of the 34 stores in its Shopify stratum serving it on 4 November 2026.
- Check whether your /llms.txt is the same bytes as your /agents.md Shopify's default points /llms.txt and /llms-full.txt at the agents.md content. 41 of the 46 publishing hosts returned an identical first 4,000 bytes at both paths.
- Do not test the apex with redirects disabled gymshark.com resolved only through a cross-host redirect ending at us.checkout.gymshark.com. An earlier corpus reading found 122 of 391 robots.txt redirects crossing hosts.
- A 403 to an identified crawler matters more than any discovery file 227 of the 1,419 hosts answered 403 to a single plain request for a text file as LantadBot/1.0 on 4 November 2026, and 121 answered 200 for a path invented for this run.
Lantad
Published .
A website cannot be a repository, and that is the whole of the confusion worth measuring here. AGENTS.md arrived as a file for coding agents working inside a checkout: build steps, test commands, conventions. The site that publishes the format, agents.md, says in its own words to create an AGENTS.md file at the root of the repository, describes nested files in a monorepo being resolved to the nearest one in the directory tree, and calls the whole thing a README for agents. It never mentions a web root. GitHub's custom instructions documentation, read on 4 November 2026, puts it in the same bracket: it names AGENTS.md, CLAUDE.md and GEMINI.md as agent instruction files and classifies them as a type of repository custom instruction.
Common questions
What is an agents.md file?
It is a plain Markdown file of instructions for AI coding agents, with no required fields or fixed headings. The agents.md site, read on 4 November 2026, says to create it at the root of a code repository, calls it a README for agents, and says the format is stewarded by the Agentic AI Foundation under the Linux Foundation. GitHub's custom instructions documentation classifies AGENTS.md as a repository custom instruction alongside CLAUDE.md and GEMINI.md.
Should an agents.md file go on my website or in my repository?
Its own specification only describes a repository. Neither the agents.md site nor GitHub's documentation mentions serving the file over HTTP, and OpenAI's crawler documentation does not mention the format at all. The web-root usage exists because Shopify made it a storefront default on 28 May 2026, not because any specification asks for it. Lantad found 46 of 1,419 corpus hostnames serving one at the web root on 4 November 2026.
How many websites actually serve an agents.md?
46 of the 1,419 hostnames in Lantad's two committed corpus seed files served a file at /agents.md on 4 November 2026. 170 answered HTTP 200, but 121 of those also answered 200 for a control path that exists on no site, and 3 returned an HTML document. 42 of the 46 are attributable to Shopify. On the same hosts the same day, 297 served a readable /llms.txt.
Does serving an agents.md file improve AI visibility?
Lantad has not measured any effect and knows of no published study that has, so the honest answer is that it is unproven. What is measurable is access: on 4 November 2026, 227 of the same 1,419 hosts refused a plain request for a text file from an identified crawler with HTTP 403, and 121 answered 200 for a path that does not exist. Neither problem is fixed by publishing a discovery file.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
llms.txt
llms.txt is a proposed plain-text markdown file placed at a site's root that gives AI systems a curated map of its most important content, so models can find and read what matters without parsing every page.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
Agentic SEO: 1 of 1,419 sites published an agent card, and 55 of the 69 OAuth documents came from a platform default
The only agent discovery path any standards registry lists is /.well-known/agent-card.json. Lantad asked all 1,419 hostnames in this repository's committed corpus for it on 19 September 2026, and one returned a usable card. 69 of the same hosts served an OAuth protected resource document at the path an MCP client reads, and 55 of those 69 carry the signature of a Shopify or WordPress default rather than of a file written for that site.
What the evidence actually says about llms.txt
Ahrefs measured 137,210 domains in May 2026 and found 97 percent of valid llms.txt files were never fetched at all. We ship an llms.txt tool, and we think you should know this.
llms.txt vs robots.txt: 19 of the 23 sites publishing one say nothing about AI in the other
Lantad requested /robots.txt and /llms.txt once each from 200 hostnames on 7 September 2026. 145 returned a parseable robots.txt and 23 returned an llms.txt as plain text. Of those 23, 19 name none of the 15 AI crawler tokens this scanner evaluates anywhere in their robots.txt.
