BlogFindings
ai.txt: 22 policy fields, and the Training default is deny
An IETF Independent Submission dated 12 June 2026 asks IANA to register /.well-known/ai.txt and /.well-known/ai.json. It defines 22 field names, and a file that never mentions training has already refused it.
What makes it worth reading rather than filing is the size of the vocabulary. RFC 9309, the Robots Exclusion Protocol published on the Standards Track in September 2022, gives you a user-agent line and two rule keywords. This draft defines 22 field names covering training, scraping, indexing, caching, per-path rules, SPDX licence identifiers, commercial licensing URLs, per-agent rate limits, attribution requirements, AI disclosure requirements and audit receipts. That is a large jump in what a site owner can say, and the interesting consequence is not what the file lets you declare. It is what it declares on your behalf when you leave a line out.
In short
- draft-car-ai-txt-wellknown-00, an Independent Submission dated 12 June 2026 and expiring on 14 December 2026, requests registration of two well-known URIs, ai.txt and ai.json, for declaring AI training, scraping, indexing and caching policy.
- The ai.txt draft defines 22 distinct field names, against the user-agent line and two rule keywords RFC 9309 gives robots.txt, and adds licensing, attribution, rate limit and audit declarations that robots.txt cannot carry.
- Section 2.4 of the draft sets the default for Training to deny while Scraping, Indexing and Caching each default to allow, so an ai.txt that omits the Training line has refused training without writing a word about it.
- The IANA Well-Known URIs registry, fetched on 4 September 2026, lists security.txt as permanent since 13 July 2021 and tdmrep.json as provisional since 15 February 2024, and holds no entry for ai.txt, ai.json or llms.txt.
- Section 1.2 of the draft states that where multiple carriers disagree for the same site and resource, conflict resolution is out of scope, so a site publishing robots.txt, an AIPREF preference and an ai.txt has no specified precedence order.
| Carrier | Where it lives | What backs it | Beyond crawl or no crawl |
|---|---|---|---|
| robots.txt | /robots.txt, at the root | RFC 9309, Standards Track, September 2022 | Nothing |
| AIPREF Content-Usage | Inside robots.txt, or an HTTP header | draft-ietf-aipref-attach-05, IETF stream, Proposed Standard | Usage preferences |
| llms.txt | /llms.txt, at the root | A community proposal, on no standards track | Nothing, it is a reading list |
| ai.txt | /.well-known/ai.txt | draft-car-ai-txt-wellknown-00, Independent Submission, Informational | Usage, licensing, fees, audit |
| security.txt | /.well-known/security.txt | RFC 9116, April 2022 | Not an AI file at all |
What is ai.txt, and where does the draft put it?
The draft requests registration of two well-known URIs under the path prefix that RFC 8615 established in May 2019: ai.txt and ai.json. The text form MUST be served at /.well-known/ai.txt over HTTPS in production, with the Content-Type text/plain and a charset of utf-8. The JSON companion MUST be served at /.well-known/ai.json with the Content-Type application/json. Both are stated as MUST in the draft's own words, which matters because almost everything else in the document is a SHOULD.
The syntax is deliberately familiar. Each line carries a key, a colon and a value, comment lines begin with a hash, and indented lines belong to the preceding block, which the draft says is inspired by robots.txt. That similarity is the point and also the trap, because a person who reads a file that looks like robots.txt will read it with robots.txt reflexes, and the two documents disagree on the thing those reflexes are built on. A crawler that finds no matching rule in your robots.txt is told by RFC 9309 section 2.2.1 that no rules apply. It proceeds. The default in this file runs the other way, and the next section is about that.
The 22 fields fall into seven groups. Two site fields are REQUIRED, Site-Name and Site-URL, and five more are optional: Spec-Version, Generated-At, Description, Contact and Policy-URL. Four content policy fields set the site-wide defaults. Two training path fields, Training-Allow and Training-Deny, take glob patterns where more specific patterns win. Two licensing fields carry an SPDX identifier and a URL to a commercial pricing page, which is the first time any of these files has offered a place to name a price rather than describe one, a subject this blog has covered before in the RSL licence directive that changes no access. Agent blocks override the site-wide policy per crawler and add an advisory Rate-Limit in an N per window form. Two content requirement fields state whether attribution and AI disclosure are required, recommended or none. Two compliance fields ask for audit receipts in a named format. One more field, AI-JSON, points the text file at its JSON twin.
Sample Illustrative, not a measurement of any real site.
GET /.well-known/ai.txt
- GET /.well-known/ai.txt HTTP/1.1 200
- Content-Type: text/plain; charset=utf-8 as the draft requires
- # ai.txt
- Spec-Version: 1.0
- Site-Name: My Blog
- Site-URL: https://myblog.com
- Training: deny stated
- Scraping, Indexing, Caching absent, so allow
Training defaults to deny, and the other three default to allow
Section 2.4 of the draft lists four content policy fields and gives each one a default. Training is OPTIONAL with a default of deny. Scraping is OPTIONAL with a default of allow. Indexing is OPTIONAL with a default of allow. Caching is OPTIONAL with a default of allow. Each accepts the values allow or deny, and one further value, conditional, is valid only on Training, where it turns on the per-path rules; the draft says an implementation meeting conditional on any other field SHOULD treat it as deny.
Read that table twice, because the asymmetry is the whole story. Three of the four fields fail open and one fails closed. A site owner who publishes the two REQUIRED fields and stops has said nothing about training and has, under the draft's own defaults, denied it. A site owner who wanted to deny caching and said so, and left the rest alone, has denied training as well. This is not a criticism of the choice, which is defensible and matches where a lot of publisher sentiment sits, and the draft states it plainly rather than burying it. It is a warning about the reading habit. Everything else in this space fails open. RFC 9309 says so in section 2.2.1 for robots.txt. An unavailable robots.txt imposes no restrictions, which is why a 404 on robots.txt allows every crawler while a 503 blocks them all. Silence has meant yes for thirty years, and in this file silence means no on exactly one line.
The JSON twin closes the gap and creates a second one. Section 3.2 states that the specVersion member, the policies member with all four of its training, scraping, indexing and caching members, and the agents member are all REQUIRED, and that defaults the text format applies implicitly MUST be stated explicitly in JSON documents. So the same policy, written in the two formats the same document defines, looks different on the page: the text file may leave Training out and mean deny, and the JSON file may not. Section 4.1 then tells agents they SHOULD prefer the JSON format when both are available. An operator who generates both from one source of truth is fine. An operator who hand-edits the text file, as people hand-edit robots.txt, and forgets the JSON, has published two policies and told crawlers to read the stale one. Anyone auditing this on a live site can start with our robots.txt tester, which will not read either file yet, and that limit is stated further down.
-
TrainingDefaults to deny Omit the line and training is refused. The only one of the four that fails closed. Accepts a third value, conditional, which activates the Training-Allow and Training-Deny path globs. -
ScrapingDefaults to allow Whether AI agents may scrape or read content. Omit the line and reading is permitted. -
IndexingDefaults to allow Whether AI systems may index content for retrieval. Omit the line and retrieval indexing is permitted. -
CachingDefaults to allow Whether AI systems may cache content. Omit the line and caching is permitted.
What backs ai.txt, and what the IANA registry actually holds
A well-known URI is not something a specification can simply assert. RFC 8615 section 5.1 says these names are registered on the advice of one or more experts under a Specification Required policy, and that IANA added a Status column to the registry marking existing registrations permanent. So a draft that requests a name has requested it, and until the expert review completes the path is a proposal that any site is free to serve and no client is obliged to fetch.
We fetched the IANA Well-Known URIs registry on 4 September 2026 and read it. security.txt is there, referencing RFC 9116, with a status of permanent and a registration date of 13 July 2021. agent-card.json is there, permanent, change controller the Linux Foundation, registered 1 August 2025, which is worth noticing because it is an AI agent file that did complete the process. tdmrep.json is there as provisional, change controller the W3C, registered 15 February 2024, and it is the machine-readable text and data mining opt-out that no crawler documentation page names. gpc.json is there as provisional since 28 January 2022. Neither ai.txt nor ai.json appears anywhere in the file. Neither does llms.txt, which does not claim the well-known path and sits at the root instead. Neither does robots.txt, for the same reason: it predates the registry by two decades and lives at the top-level path RFC 9309 section 2.3 names.
That is the honest standing, and it is not an argument that the draft is bad work. It is a well-organised document that names its own limits. It says in section 4.2 that ai.txt is advisory, that compliance is expected in good faith and is not enforced by the file itself, and that servers MUST enforce rate limits and access control independently of anything declared in it. That is the same limit RFC 9309 states about itself, in the sentence this site quotes more than any other: these rules are not a form of access authorization. A declaration is a request. What actually decides whether a fetch succeeds is the layer underneath, which is the argument in two layers decide whether AI can read your site, and no new file changes it.
| Suffix | Status | Change controller | Registered |
|---|---|---|---|
| security.txt | Permanent | IETF | 2021-07-13 |
| agent-card.json | Permanent | Linux Foundation | 2025-08-01 |
| tdmrep.json | Provisional | W3C | 2024-02-15 |
| gpc.json | Provisional | W3C | 2022-01-28 |
| ai.txt | No entry | Requested by the draft | Not registered |
| ai.json | No entry | Requested by the draft | Not registered |
| llms.txt | No entry | Does not claim the path | Not applicable |
Four carriers can now speak for one page, and none of them ranks the others
Here is the practical problem the draft creates and openly declines to solve. Section 1.2 positions ai.txt as a third carriage mechanism for AI usage preferences, alongside the two the AIPREF working group is defining, and says implementations SHOULD treat preferences carried in ai.txt as equivalent in authority to the same preferences carried by the AIPREF robots.txt or HTTP header mechanisms. Then it says that where multiple carriers disagree for the same site and resource, conflict resolution is out of scope for this document and may be addressed by future AIPREF output.
Count the carriers a real site can already have speaking about one URL. A robots.txt Disallow line governs access. An AIPREF Content-Usage line inside that same robots.txt, defined by draft-ietf-aipref-attach which reached revision 05 on 19 August 2026, governs usage, and this blog has covered both the standard you cannot deploy yet and the later change where preferences stopped inheriting. A Content-signal record from Cloudflare's Content Signals Policy sits in the same file and is now read by exactly one of ten Cloudflare crawl endpoints. An HTTP response header can carry the AIPREF preference per response. And now an ai.txt can carry a site-wide version of all of it, plus a licence and a fee. Five places, one page, and the newest of them says the tie-break is somebody else's problem.
There is a second collision, and it is in the name. Section 1.3 acknowledges that Spawning Inc. published a file called ai.txt in 2023, served at /ai.txt at the root, scoped to text and data mining opt-out per file pattern, at site.spawning.ai/spawning-ai-txt. The draft calls itself a strict superset and a successor declaration surface rather than a competitor. Two files, the same name, two different paths, and a crawler that fetches /ai.txt gets the 2023 format while one that fetches /.well-known/ai.txt gets this one. That is a live ambiguity for anyone writing a fetcher, and it is not resolved by either document. Understanding which token is asking, and for what, is the same problem set out in our AI crawler reference and in the AI crawler list this scanner maintains.
Flow: Agent wants one URL (first) to robots.txt access rules; robots.txt access rules (then usage) to AIPREF line or header; robots.txt access rules (same file) to Content-signal record; AIPREF line or header to ai.txt site-wide policy; Content-signal record to ai.txt site-wide policy; ai.txt site-wide policy to Carriers disagree; Carriers disagree (section 1.2) to Out of scope.
Should you publish an ai.txt today, and what does Lantad do about it?
The case for writing one is the same case as for llms.txt, and it is weaker than it sounds. It costs a few minutes and one file, it states your position in a form a machine can parse, and if the format is adopted you were early. The case against is that we fetched five crawler documentation pages on 4 September 2026, being OpenAI's bots page at developers.openai.com and its platform.openai.com twin, Anthropic's crawler article at support.claude.com, Perplexity's bots guide at docs.perplexity.ai and Google's crawler overview at developers.google.com, and the string ai.txt appeared in none of them, the draft itself names no implementation, and a policy nobody reads is a policy that changes nothing. We have published that argument against our own product before: the evidence on llms.txt is that a study of 137,210 domains in May 2026 found 97 percent of valid files received no requests at all, and we ship an llms.txt tool anyway and say so on the page.
If your requirement is narrow, a narrower file is the better answer, and that is worth saying plainly rather than steering you to the newest option. If all you need is a text and data mining reservation with legal effect under EU Directive 2019/790, tdmrep.json is the file with a registered path and a standards body behind it, published by the W3C TDM Reservation Protocol community group, and the draft itself says sites with TDM-only requirements MAY use it alone. If all you need is to stop a named crawler, robots.txt is the only one of these files that every major AI vendor documents reading, and it is where the decision actually lands. If what you want is a licensing conversation rather than a refusal, the honest route today is a contract, not a field.
Lantad does not check for /.well-known/ai.txt, and there is an awkward detail in our own code worth publishing. AGENT_FILE_PATHS in core/src/trafficrollup.ts, the constant that decides which file requests are counted as interesting rather than filtered out as assets, lists six paths: /robots.txt, /llms.txt, /llms-full.txt, /sitemap.xml, /sitemap-index.xml and /ai.txt. Every one of them is at the root. So the traffic rollup would count a crawler asking for the 2023 Spawning file and would not count one asking for the path this draft specifies, which is a setting somebody chose and not a finding about anything. We are leaving it as it stands until there is a reason to move it, and the reason would be a crawler that fetches the new path, which is the thing none of us has seen yet. What we do measure, and how, is on the methodology page, our own crawler's conduct is published at /bot, and the vocabulary behind all of this is in the glossary entries for AI visibility and llms.txt.
- robots.txt stops a named crawler RFC 9309, Standards Track since September 2022, and the one file every AI vendor's own crawler documentation tells you to write.
- tdmrep.json reserves TDM rights Provisional in the IANA registry since 15 February 2024, W3C change controller, and scoped to one legal purpose it does well.
- ai.txt names a price for training Training-License takes an SPDX identifier and Training-Fee a URL. No documented consumer, and the path is not in the IANA registry as of 4 September 2026.
- ai.txt is read by a shipping crawler Five crawler documentation pages fetched on 4 September 2026 carried the string ai.txt nowhere. Section 4.2 states compliance is expected in good faith and is not enforced by the file.
- Lantad reports on /.well-known/ai.txt It does not. AGENT_FILE_PATHS in core/src/trafficrollup.ts lists /ai.txt at the root and five other root paths, and no well-known path at all.
Lantad
Published .
A new file wants to sit at the root of your site and tell AI systems what they may do with your words. It is called ai.txt, it lives at /.well-known/ai.txt, and it arrived as an Internet-Draft on 12 June 2026 under the name draft-car-ai-txt-wellknown-00, written by Kayla Cardillo as an Independent Submission with an intended status of Informational. It expires on 14 December 2026. Lantad has measured nothing about how many sites publish one, because as far as we can tell almost nobody does, and this post reports the specification from the specification rather than from any scan.
Common questions
What is ai.txt?
A proposed policy declaration file served at /.well-known/ai.txt, specified by draft-car-ai-txt-wellknown-00, an IETF Independent Submission dated 12 June 2026 with an intended status of Informational and an expiry of 14 December 2026. It uses a key and colon syntax inspired by robots.txt and defines 22 field names covering training, scraping, indexing and caching policy, per-path training rules, SPDX licensing, commercial licensing URLs, per-agent overrides and rate limits, attribution and AI disclosure requirements, and audit receipts.
Does ai.txt replace robots.txt?
No, and the draft says so in section 1.1. It states that ai.txt is complementary to robots.txt and does not replace it, and that both files may coexist. The division it proposes is that robots.txt declares crawling restrictions while ai.txt adds training, licensing and per-agent policy declarations that robots.txt cannot express. Access is still decided by the Disallow lines and by whatever your infrastructure does when the request actually arrives.
If I publish an ai.txt without a Training line, what have I said?
You have denied training. Section 2.4 gives Training a default of deny, while Scraping, Indexing and Caching each default to allow. The JSON companion at /.well-known/ai.json works differently: section 3.2 requires that defaults the text format applies implicitly are stated explicitly, so the same policy is written out in full there. Section 4.1 tells agents to prefer the JSON file when both exist.
Does Lantad check for ai.txt?
Not at the well-known path. Read out of this repository on 4 September 2026, AGENT_FILE_PATHS in core/src/trafficrollup.ts names six root paths, being /robots.txt, /llms.txt, /llms-full.txt, /sitemap.xml, /sitemap-index.xml and /ai.txt, and none under /.well-known/. That constant is a setting rather than a measurement, and the /ai.txt entry in it refers to the root path Spawning Inc. used in 2023 rather than to the path this draft specifies.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
llms.txt
llms.txt is a proposed plain-text markdown file placed at a site's root that gives AI systems a curated map of its most important content, so models can find and read what matters without parsing every page.
What the evidence actually says about llms.txt
Ahrefs measured 137,210 domains in May 2026 and found 97 percent of valid llms.txt files were never fetched at all. We ship an llms.txt tool, and we think you should know this.
The AI preferences standard you cannot deploy yet
The IETF vocabulary for expressing AI usage preferences reached its sixth draft on 28 April 2026. The companion draft that defines where to write one down expired on 1 May 2026, so there is still nothing on your site for a crawler to read.
Three crawler documentation pages name robots.txt, and none names TDMRep
The W3C TDM Reservation Protocol defines four places a site can record that AI mining rights are reserved. Read on 4 August 2026, the crawler documentation published by OpenAI, Anthropic and Google names robots.txt in every case and mentions none of the four.
RSL writes a licence into robots.txt and says it changes no access
The Really Simple Licensing 1.0 specification, published on 10 December 2025, adds a directive named License to robots.txt, and states in the same section that it does not modify the access permissions expressed by Allow or Disallow. RFC 9309 tells crawlers they MAY interpret records it does not define. Four vendor crawler documentation pages, read on 9 August 2026, do not mention RSL at all.
