# RSL writes a licence into robots.txt and says it changes no access > The Really Simple Licensing 1.0 specification, published on 10 December 2025, adds a directive named License to robots.txt, and states in the same section that it does not modify the access permissions expressed by Allow or Disallow. RFC 9309 tells crawlers they MAY interpret records it does not define. Four vendor crawler documentation pages, read on 9 August 2026, do not mention RSL at all. - Canonical page: https://lantad.co/blog/rsl-license-directive-changes-no-access - This file: https://lantad.co/blog/rsl-license-directive-changes-no-access.md - Last substantive update: 2026-08-09 ## Key facts - **Published:** 2026-08-09 - **Category:** Findings - **Author:** Lantad - **Length:** 3388 words - **Takeaway 1:** The Really Simple Licensing 1.0 specification, document RSL-SPEC-1.0, published on 10 December 2025 with the status Recommendation, extends the Robots Exclusion Protocol with a robots.txt directive named License whose value must be an absolute URI resolving to an RSL licence document. - **Takeaway 2:** The same specification, section 4.4.2, states that the License directive does not modify the access permissions expressed by Allow or Disallow fields and that it solely identifies the governing license document, so adding the line changes nothing about which paths any crawler may fetch. - **Takeaway 3:** RFC 9309, published on the Standards Track in September 2022, defines only user-agent, allow and disallow in its ABNF grammar, and its section 2.2.4 states that crawlers MAY interpret other records that are not part of the robots.txt protocol, so a crawler ignoring every License line on the web remains conformant with it. - **Takeaway 4:** The crawler documentation published by OpenAI, Anthropic, Perplexity and Google, all four fetched and searched on 9 August 2026, contains no occurrence of the string RSL or of Really Simple Licensing. - **Takeaway 5:** Lantad's robots.txt parser in core/src/robots.ts recognises four field names, user-agent, allow, disallow and sitemap, discards crawl-delay deliberately, and records every other field including License as a parse note reading unknown field, ignored. That is how the parser is written, not a measurement of any crawler. ## Summary Every argument about paying for AI training data arrives eventually at the same file. A robots.txt file is the only machine readable document most sites publish about who may read them, it is fetched by more or less every [AI crawler](https://lantad.co/glossary/ai-crawler) that respects anything at all, and it is therefore the obvious place to write terms down. Really Simple Licensing takes that step directly. Its 1.0 specification defines a robots.txt directive named License, and a publisher who adds one line to a file they already maintain has, in a real sense, published licensing terms covering their whole site. What that line does is narrower than it looks, and the specification is unusually clear about it. What follows is a reading of two documents against each other, the RSL 1.0 specification and RFC 9309, plus a check of what four crawler vendors say about RSL and a description of what this site's own parser does with the directive. Lantad has measured nothing about RSL. We hold no count of sites carrying a License line, have observed no crawler fetching a licence document, and publish no adoption figure, which is a limit of what [our methodology](https://lantad.co/methodology) can see from outside a hostname rather than an omission. The specification is at rslstandard.org/rsl and is not linked below, because that host is not on this site's outbound register. ## What RSL 1.0 adds to robots.txt The specification identifies itself as RSL-SPEC-1.0, an Industry Specification with the status Recommendation, published 2025-12-10. Its editors are listed as representatives of the RSL Collective, Conde Nast, Ziff Davis, Schema.org, Yahoo, Automattic, O'Reilly Media and Fastly. That editor list matters for reading the rest of the document, because it is a publisher-side standard written by publishers, and the obligations in it fall almost entirely on the client. Section 4.4 is the part that touches the file this site reads on every scan. In the specification's words, RSL extends the Robots Exclusion Protocol with a new directive, License, that associates a website or section of a website with one or more governing RSL license documents. The syntax is a single line, given in ABNF as the literal string License, a colon, optional whitespace and an absolute URI. Comments beginning with a hash follow standard robots.txt conventions and are ignored, and multiple License directives may appear in one file. Placement decides scope, and the rule has a sharp edge worth knowing before you write the line. A License directive outside any User-agent group is global and applies to all clients. A License directive inside a User-agent group applies only to clients that select that group under RFC 9309's group selection rules. The specification then states that if a client selects a group containing one or more License directives, it must treat those as the candidate licence set and must ignore any global License directives. So a site that publishes a global licence and also gives one named crawler its own group with its own License line has, for that crawler alone, switched the global terms off. Nothing in the file says so, and nothing in a [robots.txt tester](https://lantad.co/tools/robots-txt-tester) that only evaluates Allow and Disallow would surface it either, because the precedence lives in a layer above access. The rest of the vocabulary is XML rather than plain text. The License URI resolves to a document in the namespace rslstandard.org/rsl, served as application/rsl+xml, whose root element contains content elements with license children. Those licence elements carry permits, prohibits, payment, reporting and legal terms. None of that is in robots.txt. The robots.txt line is a pointer, and the pointer is the only part of RSL that lives in the file every crawler already fetches. ## What RFC 9309 says a crawler must do with a line it never defined RSL says it builds on RFC 9309, and it does. The question is what building on a standard obliges the other side to do, and here the two documents point in opposite directions. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the Robots Exclusion Protocol, was published on the Standards Track in September 2022. Its formal syntax in section 2.2 gives an ABNF grammar containing exactly three field names: user-agent as the start of a group, and allow and disallow as rules within one. Sitemap is not in the grammar. Neither is crawl-delay, neither is License, and the grammar carries an explicit comment addressed to implementers telling them to define additional lines they need, giving Sitemaps as the example. Section 2.2.4, headed Other Records, is where the whole question sits. It states that crawlers MAY interpret other records that are not part of the robots.txt protocol, gives Sitemaps as its example, and adds that crawlers MAY be lenient when interpreting other records. The only requirement in that section runs the other way: parsing of other records must not interfere with the parsing of explicitly defined records, so a Sitemap line must not terminate a group. A crawler that reads a robots.txt file, extracts the user-agent groups and the allow and disallow rules, discards every other line unread and never fetches a licence document is fully conformant with RFC 9309. RSL's own language for the client is stronger. Section 4.4.2 says clients must retrieve and interpret these licenses before accessing or processing any content from the origin, and section 4.4.3 says that where no licence can be discovered by any mechanism, the resource must be treated as unlicensed for the purposes of the specification. Those are real obligations, but they bind only a client that has adopted RSL. Reading the two documents together, the obligation to honour a License line is entirely self-imposed. This is the same structural gap that showed up when [Cloudflare attached a consequence to crawl-delay](https://lantad.co/blog/crawl-delay-can-unverify-a-crawler), a directive RFC 9309 also never defined, and it is not a criticism of either specification. It is what an extension to a voluntary protocol can be. ## Where the enforcement actually lives The single most useful sentence in the specification for a site owner is one line of section 4.4.2: the License directive does not modify the access permissions expressed by Allow or Disallow fields, and it solely identifies the governing license document or documents. Read plainly, a publisher who adds a License line and changes nothing else has changed zero bytes of what any crawler is permitted to fetch. The archive that was crawlable yesterday is crawlable today. The terms are now discoverable, which is not the same thing as being applied. Enforcement in RSL is a separate mechanism at a separate layer, and the specification is explicit about that too. A server holding licensed content may answer a request with 401 Unauthorized or 402 Payment Required, and that response should carry either an inline RSL document with Content-Type application/rsl+xml or a Link header naming the governing licence with rel set to license. Acquiring a licence runs over the Open License Protocol, which the specification describes as an extension of the OAuth 2.0 framework of [RFC 6749](https://www.rfc-editor.org/rfc/rfc6749.html). Presenting one runs over the Crawler Authorization Protocol, which introduces a new HTTP authentication scheme named License, carried in the standard Authorization and WWW-Authenticate headers and built on the authentication framework in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html). That is a substantial amount of server engineering, and none of it is implied by editing a text file. The specification also notes that the Crawler Authorization Protocol should be paired with a bot management service or with [Web Bot Auth](https://lantad.co/blog/what-web-bot-auth-actually-specifies), because CAP verifies a licence and not an identity, and a token presented by a client whose identity nobody has checked is a weaker guarantee than it looks. That layering is honest and it is also the point: the licence check assumes a solved problem that [a user agent string does not solve](https://lantad.co/blog/a-user-agent-is-a-claim-not-an-identity). There is one more consequence that matters for anyone measuring. If enforcement is a 402 at the origin, then the licensing posture of a site is not legible from its robots.txt at all. This blog has already reported the empirical side of that: a probe of media properties found [tolls that no robots.txt declared](https://lantad.co/blog/publisher-tolls-are-invisible-in-robots-txt), and Cloudflare's own paid crawling product [runs on a status code the HTTP specification leaves undefined](https://lantad.co/blog/pay-per-crawl-runs-on-an-undefined-status-code). RSL is the first serious attempt to put a pointer to those terms back into the file, and the pointer is deliberately inert. ## The usage vocabulary, and what it borrows Inside the licence document, the permits and prohibits elements carry tokens from a vocabulary the specification declares normative. The usage vocabulary defines six tokens. All covers any automated processing, including AI training and search. Ai-all covers any use by an AI system, and the specification states explicitly that it includes the three narrower AI terms below it as well as AI uses not yet enumerated. Ai-train is training or fine tuning. Ai-input is input into a model, named as covering retrieval augmented generation, grounding, and other use of content to produce generative AI responses or search summaries. Ai-index is inclusion in an AI system's internal index or retrieval database. Search is building a search index and returning results. Two things about that list are worth a site owner's attention. The first is that it draws the training and retrieval line that most crawler vendors do not let you draw in robots.txt. The whole reason blocking advice is confusing is that a single product token often covers both purposes, which is why [six vendors publishing one token each](https://lantad.co/blog/six-vendors-publish-one-crawler-token) is a structural problem rather than a documentation gap. RSL puts the distinction in the licence rather than in the user agent, which sidesteps the vendor's token design entirely. It also means the distinction only exists for clients that read the licence. The second is provenance. The specification states that this category includes the Cloudflare Content Signals vocabulary and may include terms from additional standardised vocabularies as they become available, naming IETF AI Preferences as its example. Both are already subjects here. Cloudflare's content signals are a comment block that [asks rather than blocks](https://lantad.co/blog/content-signal-asks-disallow-blocks), and the IETF work is [a standard you cannot yet deploy](https://lantad.co/blog/the-ai-preferences-standard-you-cannot-deploy-yet). RSL is consolidating vocabularies that were separately unenforceable into one document that is separately unenforceable, and adding a payment and authorisation layer underneath them that the others never had. Whether that is the missing piece depends entirely on adoption, which is measurable and which nobody, including this site, has measured. The specification also allows tokens from extension namespaces, with the condition that they must not be interpreted unless the processor explicitly recognises and supports them. That is the correct rule for an extensible vocabulary and it repeats the pattern at the layer above: an unrecognised term is silently inert, exactly as an unrecognised robots.txt field is. ## Do the crawler vendors mention RSL? A licensing standard whose obligations bind only its adopters raises one obvious question, and it is answerable without any measurement of crawler behaviour: does any crawler vendor say it reads these files? On 9 August 2026 four crawler documentation pages were fetched and searched for the strings RSL and Really Simple Licensing. [OpenAI's crawler documentation](https://platform.openai.com/docs/bots), which names GPTBot, contains neither. [Anthropic's crawler page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), which names ClaudeBot, contains neither. [Perplexity's bots documentation](https://docs.perplexity.ai/guides/bots), which names PerplexityBot, contains neither. [Google's crawler overview](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers), which names Googlebot, contains neither. State clearly what that is and is not. It is an absence in four documents on one date. It is not evidence about what any of those crawlers does when it meets a License line, because a crawler can honour something it has not documented and can document something it does not honour. It is the same kind of finding as [three crawler pages naming robots.txt and none naming TDMRep](https://lantad.co/blog/tdmrep-not-named-in-crawler-documentation), and it carries the same weight, which is modest but real: a publisher deciding whether to invest in a licence server has no vendor commitment to point at. The asymmetry is what makes this worth publishing rather than shrugging at. Under RSL, a client that finds no licence must treat the resource as unlicensed. Under RFC 9309, a crawler that reads no licence has broken nothing. So the site that publishes terms and the crawler that ignores them can both be behaving correctly under the document each of them is following, and the disagreement surfaces nowhere in the protocol. It surfaces in court, or in a server log, or in an invoice, none of which is a place a scanner can look. ## What a site owner can check today The honest summary for anyone weighing this up is that RSL asks for two decisions and only one of them is cheap. Publishing a licence document and pointing a License line at it costs an afternoon and changes no access. Running a licence server, issuing tokens over OAuth 2.0, verifying them at the edge and answering unlicensed requests with 401 or 402 is a payments and authorisation project, and it is where any actual revenue would come from. Before either, there is a smaller check that applies whether or not RSL is in your plans, because it is about the file rather than the standard. Whatever your robots.txt says about licensing, the lines that decide whether an AI answer engine can quote you are still the Allow and Disallow rules and the groups they sit in. Those are what [our scanner](https://lantad.co/tools/what-gptbot-sees) evaluates per named crawler, and a licence line sitting above a group that disallows the crawler you wanted to license is a contradiction the file will never report. It is worth confirming that a robots.txt edit has reached the crawlers at all, since [a cached copy can be up to 24 hours old](https://lantad.co/blog/when-a-robots-txt-edit-reaches-a-crawler), and that the file returns a status code that means what you think, because [a 404 and a 503 are opposites here](https://lantad.co/blog/robots-txt-404-and-503-are-opposites). It is also worth knowing what a robots.txt reader does with a directive it does not recognise, because that behaviour is the practical fate of a License line today. This site's parser lives in core/src/robots.ts. It handles four field names by name: user-agent starts or extends a group, allow and disallow become rules, and sitemap is collected into a list. Crawl-delay is matched and then deliberately discarded, with a comment saying it is captured nowhere because the scanner never crawls at volume and it does not affect access verdicts. Every other field falls to the default branch, which appends a parse note recording the line number and the field name and the word ignored. A License line therefore produces a visible note and no verdict change. That is a description of code in this repository, verified on 9 August 2026, and not a claim about how any other crawler behaves. If you want the underlying question answered rather than the licensing one, it has not changed shape. Whether a model can quote you still depends on whether a named crawler is allowed to fetch the URL and whether the words survive the fetch, which is what the [AI visibility](https://lantad.co/glossary/ai-visibility) score measures and what [the crawlability research](https://lantad.co/research/crawlability-study) here is about. Licensing sits on top of that and does not replace it. A page no crawler can read is not made more valuable by attaching terms to it, and a page every crawler can read is not protected by them either. The [glossary entry on GEO](https://lantad.co/glossary/geo) covers where the two concerns meet, and the [bot page](https://lantad.co/bot) documents how this scanner identifies itself, which is the same courtesy RSL asks of everybody else. ## Questions and answers **Does adding an RSL License line to robots.txt block AI crawlers?** No. RSL 1.0 section 4.4.2 states that the License directive does not modify the access permissions expressed by Allow or Disallow fields and that it solely identifies the governing license document. Blocking still requires a Disallow rule in a group the crawler selects, or a refusal at the server. **Must a crawler read an RSL licence document?** RSL says yes and RFC 9309 says no. The specification states that clients must retrieve and interpret the licences before accessing content, but that obligation binds only clients that have adopted RSL. RFC 9309 section 2.2.4 states that crawlers MAY interpret records outside the protocol, so ignoring a License line is conformant. **Have OpenAI, Anthropic, Perplexity or Google committed to RSL?** Their crawler documentation does not mention it. All four pages were fetched and searched on 9 August 2026 and none contains the string RSL or Really Simple Licensing. That is an absence in documentation on one date, not evidence about what any of those crawlers does when it encounters a License line. **Where does RSL actually enforce anything?** At the origin server, not in robots.txt. The specification defines a Crawler Authorization Protocol using a new HTTP authentication scheme named License in the Authorization header, and servers answer unlicensed requests with 401 Unauthorized or 402 Payment Required carrying either an inline RSL document or a Link header naming the licence. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot