Three crawler documentation pages name robots.txt, and none names TDMRep

What the TDM Reservation Protocol actually specifies

TDMRep is a Final Community Group Report published on 10 May 2024, edited by Laurent Le Meur of EDRLab for the Text and Data Mining Reservation Protocol Community Group. Its status section is unambiguous about what that means, and the sentence is worth reading before treating the document as a standard: it states that the specification is not a W3C Standard nor is it on the W3C Standards Track. That is not a criticism of the work, it is a description of its authority. A Community Group Report carries no W3C endorsement, and any weight it has comes from adoption and from the legal instrument it was written to serve rather than from the consensus process that produces a Recommendation.

The data model is deliberately small. A reservation has one mandatory property, tdm-reservation, whose value is 0 or 1. The value 1 means, in the specification's words, that TDM rights are reserved, and that if a TDM policy is set an agent may use it to find out how to acquire authorisation to mine the content. The value 0 means rights are not reserved and an agent can mine the content without contacting the rightsholder. A second optional property, tdm-policy, holds a URL pointing at the terms. Two fields and a link is the whole vocabulary, which is a reasonable design for something that has to be readable by a machine that will never negotiate.

Four techniques carry those fields, and the specification defines all four rather than picking one. A JSON file at /.well-known/tdmrep.json holds an array of rules, each with a mandatory location pattern, a mandatory tdm-reservation value and an optional tdm-policy URL, so a site can reserve rights over one directory and leave another open. A tdm-reservation HTTP response header carries the same value on a per-response basis. An HTML meta tag with the name tdm-reservation carries it per document, alongside an optional tdm-policy meta. EPUB and PDF files carry it in their own embedded metadata, which is the channel that explains the protocol's centre of gravity among book and journal publishers.

The location patterns in the well-known file are not a new matching language. The specification re-uses the robots.txt wording and tells agents that, for interoperability, they should follow the rules Google documents in How Google interprets the robots.txt specification under URL matching based on path values. That is a sensible reuse and it produces a small trap for anyone reading quickly: the file borrows robots.txt path semantics while being a completely separate file with a completely different meaning, in much the way that a page can carry rich structured data and still be unreadable to the crawler that never fetched it. Same syntax, different question.

robots.txt is not one of the four channels, and it answers a different question

The most common misreading of this area is that TDMRep and robots.txt are competing ways to say the same thing, so a site that has written one has covered the other. They are not, and the difference is structural rather than a matter of emphasis. RFC 9309 standardises robots.txt as a protocol for telling an automated client which URLs on a host it may request. Every rule in it is about access. A Disallow line is a request not to fetch, and the whole file is evaluated by selecting a group whose product token matches the client's name, which is why a robots.txt tester has to be told which crawler it is testing as.

TDMRep never touches access. Its fields describe what may be done with content the agent has lawfully obtained. A site can serve a tdm-reservation of 1 on a page that robots.txt allows every crawler to fetch, and there is no contradiction in that: fetch it, index it, do not mine it. It can equally disallow GPTBot in robots.txt and carry no TDM reservation anywhere, in which case it has restricted access without making any rights statement at all. Treating either file as a substitute for the other produces a confident wrong answer in both directions, and it is the sort of wrong answer that only shows up in a dispute.

The legal framing is what makes the distinction matter rather than being a taxonomy exercise. The specification describes itself as a technical answer to the constraints set by Article 4 of the European copyright directive, and summarises that exception as applying on condition that the use of content for TDM has not been expressly reserved by rightsholders in an appropriate manner, such as machine-readable means. The directive it refers to is Directive (EU) 2019/790, and the phrase carrying the weight is machine-readable means. TDMRep exists because that phrase names a requirement without naming a format, and somebody had to propose one.

Whether a Disallow line also satisfies that phrase is a legal question this post is not qualified to settle, and it is genuinely contested rather than merely unsettled in the abstract. What can be said precisely is the engineering half. robots.txt encodes an access decision keyed to a crawler name; TDMRep encodes a rights declaration keyed to a path, with no crawler name anywhere in the model. A system reading only one of them cannot reconstruct the other, and the failure is silent, in the same family as the finding that a 404 and a 503 on robots.txt mean opposite things while looking equally like an error to anyone not reading the specification.

What OpenAI, Anthropic and Google tell site owners to use

A protocol is only worth deploying if the agents it addresses read it, so the question that decides whether TDMRep is operationally real is what the crawler operators document. Three pages were read directly on 4 August 2026, one from each of the three vendors whose crawlers appear most often in the logs of the sites this scanner is pointed at. The result is uniform, and uniform in a way that is easy to state and easy for a reader to check in about two minutes.

OpenAI's crawler documentation documents four user agent tokens, GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User, and states that OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI. It gives the split most site owners actually want, allowing OAI-SearchBot to appear in search results while disallowing GPTBot to keep content out of training, and it points at published IP ranges as a supporting measure. It does not mention TDM, tdm-reservation or TDMRep.

Anthropic's crawler article names ClaudeBot, Claude-User and Claude-SearchBot, says the bots respect do not crawl signals by honouring industry standard directives in robots.txt, and gives worked examples of a Crawl-delay and a Disallow for ClaudeBot. It warns that alternate methods such as blocking IP addresses may not work correctly or guarantee an opt-out. It does not mention TDM, tdm-reservation or TDMRep either.

Google's crawler overview, carrying Last updated 2026-06-12 UTC, identifies robots.txt as the mechanism and states that its common crawlers always respect robots.txt rules for automatic crawls. The same is true of it: no TDM, no tdm-reservation, no TDMRep. Three pages, three vendors, one named mechanism between them.

That is a negative result and it should be read as exactly what it is. Absence from a documentation page is not proof that no system anywhere at those companies reads a tdmrep.json, and none of the three pages is an exhaustive statement of everything its publisher honours. What it does establish is the thing a site owner has to act on, because a site owner can only follow instructions that exist: the published instructions from all three point at one file, and that file is not TDMRep's. This is the same shape as the earlier count on this blog showing six of the nine vendors in the registry publish exactly one crawler token, where the constraint on what a site can express turned out to be set by what vendors had bothered to name. The AI crawler registry behind this site's own checks is built from those same published pages, which is why it contains robots.txt tokens and nothing else.

Where the reservation is written decides who can see it

The four channels are not interchangeable, because the specification gives them an order. A TDM agent must check the well-known file on the origin server before it starts scraping. A tdm-reservation HTTP header supersedes any value inferred from that file. HTML metadata supersedes the previous values. Metadata inside an EPUB or PDF supersedes those in turn. The most specific statement wins, which is the right design for a rights declaration and has a consequence anybody building a checker should sit with for a moment.

It means the site-level file does not tell you the answer for a page. A publisher can serve a tdmrep.json reserving nothing and then set tdm-reservation to 1 in the head of a particular article, and the correct answer for that article is 1. A tool that fetches only /.well-known/tdmrep.json and reports the site's position will be wrong about exactly the pages a publisher cared enough to mark individually. Reading the file alone is cheaper and gives a plausible answer, which is the combination that makes an incorrect check survive.

The specification also says an agent must keep the well-known file in cache to optimise further processing, and it sets no refresh interval at all. Compare that with robots.txt, where RFC 9309 asks crawlers not to use a cached copy for more than 24 hours and Google's documentation describes caching the file for up to a day. A cache with no stated maximum age means an edit to a reservation has no defined moment at which it takes effect for a given agent, which is a sharper version of a gap this blog has already reported, that a robots.txt edit does not take effect when you save it. At least there the interval is written down.

TDMRep is also not the only signal that lives outside robots.txt, and the pattern is becoming the norm rather than the exception. Microsoft's AI opt out is a meta tag rather than a product token. The IETF AIPREF working group is building a vocabulary and an attachment mechanism for the same problem from a different direction, which this blog covered as the AI preferences standard you cannot deploy yet. One more piece of context belongs here because it is checkable and rarely mentioned: IANA lists tdmrep.json in its Well-Known URIs registry with the status provisional, registered on 15 February 2024 with W3C as change controller, and that registry is at www.iana.org/assignments/well-known-uris/well-known-uris.xhtml. robots.txt is absent from it, being defined by its own RFC, and llms.txt is absent too. Provisional is a real registration and it is also not a permanent one.

The W3C group publishes an adopters file listing organisations that have implemented the protocol, and its composition is the most informative thing about it. Academic and STM publishers dominate, mostly deploying tdmrep.json with a policy URL, with trade publishers and a set of mostly French newspapers behind them. The only adoption figure the file carries is an undated estimate attributed to a national publishers' association, which is why no percentage from it appears in this post. That absence is the honest summary of the deployment evidence: the protocol has real adopters concentrated in publishing, and no public measurement of its presence across the web that meets the standard this blog applies to a number.

What this scanner checks, and what it does not

The part of this that is Lantad's to answer is narrow, and stating it plainly is more useful than implying broader coverage. This scanner measures whether a crawler can read a page: it fetches robots.txt and evaluates it per crawler token, fetches the page as a plain client and as a browser, and compares the two. It does not request /.well-known/tdmrep.json. It does not read a tdm-reservation header or meta tag. It holds no measurement of TDMRep deployment on any site, and nothing on this site should be read as one.

That gap is worth naming rather than quietly closing, because the two things are not the same measurement and merging them would be a category error. AI visibility as this site defines it is a readability question: can the client fetch the page, and does the served markup contain the prose a human sees. A TDM reservation answers a permissions question that is orthogonal to all of it. A site scoring well here can carry a reservation on every page, and a site with no reservation anywhere can be completely unreadable. Adding a tdmrep check to a readability score would make the number mean less, not more, which is the same reasoning set out in this site's scan methodology for what does and does not enter a grade.

There is a genuine argument for reporting it as a separate observation, and it is a strong one: the check is cheap, one GET against a fixed path plus two fields already present in a fetch this scanner performs anyway, and the answer is something a site owner frequently does not know about their own site. The argument against shipping it today is that a check reporting only the well-known file would be wrong about page-level overrides for precisely the pages a publisher marked deliberately, and a wrong permissions answer is worse than no permissions answer. Nothing is promised here on timing; this is a note about what is not measured, not a roadmap.

What a reader can do without any tool is short enough to write out. Decide whether the site's position is a rights position, an access position, or both, because they are separate decisions and most sites have only ever made the second one. If access is what is wanted, that is robots.txt, and it is keyed to crawler names published by the vendors, which is what LantadBot declares about itself and what this scanner reads back. If a rights reservation is what is wanted, TDMRep gives four places to write it, the well-known file is the one an external party can check, and the vendor documentation surveyed above means no major crawler currently tells you it will read it. Both of those can be true at once, and a site owner is better off knowing that than assuming a Disallow line has covered a question it never addressed. The dated write-ups behind claims like this one are collected on this site's research page.