BlogFindings
Text and data mining opt out: 15 sites reserved the right in prose, and none served the machine-readable file
Lantad requested robots.txt, then the home page, then every legal page the home page linked, for all 1,419 hostnames in this repository's two committed corpus seed files on 14 October 2026, executing no JavaScript. 656 hostnames delivered a readable legal page. Reading all 32 candidate clauses by hand confirmed 15 whose own terms restrict the use of site content for AI or machine learning training, and 11 of those 15 leave all nine training crawler tokens this scanner evaluates allowed at the root. Six hostnames in the whole corpus serve a TDMRep file at /.well-known/tdmrep.json, and none of the six is one of the 15.
An AI crawler is not a reader. It requests /robots.txt, applies whatever rules it finds for its own product token, and then requests the page. It does not open the terms page on the way, and no crawler vendor documents a route by which the words on a terms page could reach it. That gap is what this post measures: not whether a text and data mining opt out is legally meaningful, which is not ours to judge, but whether the sites that wrote one also expressed it anywhere a crawler looks. Of the 15, four did, and the other eleven serve a robots.txt that this scanner's own evaluator reads as an open door for all nine training crawlers it knows by name. If you want to see what your own file says before reading further, the robots.txt tester answers per crawler.
In short
- Lantad read the legal pages of 656 hostnames on 14 October 2026 and confirmed 15 by hand whose own terms restrict using the site's content to train AI or machine learning models. 11 of the 15 left all nine training crawler tokens allowed at the root of their robots.txt.
- A text and data mining opt out only reaches a crawler if it is machine readable, and on 14 October 2026 six of the 1,419 hostnames in this corpus served a TDMRep file declaring a reservation at /.well-known/tdmrep.json: cambridge.org, france24.com, nature.com, science.org, sciencedirect.com and unesco.org.
- None of the 15 hostnames that reserved the right in prose served that file. 13 answered 404 at the path and 2 answered 403, and no tdm-reservation meta element appeared on any of the 1,058 home pages read.
- OpenAI's crawler documentation and Anthropic's crawler support article both name robots.txt as the control and neither offers a terms of service route, so a clause no crawler fetches changes nothing about what it fetches.
- sonarsource.com forbids using its Content for training, fine-tuning or retrieval-augmented generation in its website terms, and its robots.txt names 31 crawler tokens, GPTBot, ClaudeBot and CCBot among them, and allows every one of them at the root.
| Hostname | Where the clause sits | Training tokens named | Training tokens blocked |
|---|---|---|---|
| bostonglobe.com | Terms of service | 9 | 9 |
| statnews.com | Terms of service | 9 | 9 |
| slate.com | Terms of service | 8 | 8 |
| coursera.org | Terms of use | 5 | 1 |
| aafp.org | Terms of use | 0 | 0 |
| acog.org | Privacy statement | 0 | 0 |
| alvys.com | Terms of service | 0 | 0 |
| docker.com | Terms of use | 0 | 0 |
| foxnews.com | Terms of use | 0 | 0 |
| kayak.com | Terms of use | 0 | 0 |
| psychiatry.org | Copyright page | 0 | 0 |
| replit.com | Terms of service | 0 | 0 |
| sonarsource.com | Website terms of use | 8 | 0 |
| wagamama.com | Terms and conditions | 0 | 0 |
| wiley.com | Copyright page | 0 | 0 |
What does a text and data mining opt out have to be to work?
It has to be a file or a header, because those are the only two things a crawler fetches before it decides. RFC 9309, the Internet Standards Track document from September 2022 that specifies the robots exclusion protocol, is specific about where the rules live: they "MUST be accessible in a file named '/robots.txt' (all lowercase) in the top-level path of the service". It also caps reuse, saying crawlers "SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable". Nothing in it points anywhere else on the host.
The vendors say the same thing in their own words. OpenAI's crawler documentation describes its robots.txt tags for GPTBot and OAI-SearchBot, and states that "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models". It offers no second mechanism. Anthropic's crawler support article says opting out "requires modifying the robots.txt file in the manner above", names ClaudeBot, Claude-User and Claude-SearchBot, and warns that blocking by IP address "may not work correctly or persistently guarantee an opt-out". Neither page mentions terms of service, a copyright notice or a legal page as a route to the same result.
There is a machine-readable form of exactly the reservation these 15 sites wrote. The TDM Reservation Protocol, a W3C Community Group Final Report dated 2 February 2024, defines four ways to express it: a tdmrep.json file in the /.well-known directory, a tdm-reservation HTTP response header, a tdm-reservation meta element in HTML, and package metadata in EPUB. The value is a boolean, where 1 means rights are reserved, and an optional tdm-policy names a URL where the terms sit. Its own status line is the honest part of the picture: it "is not a W3C Standard nor is it on the W3C Standards Track", and when we read the crawler documentation of OpenAI, Anthropic and Google in August 2026, none of the three named TDMRep at all. So the protocol gives a site a place to put the reservation and no vendor has promised to read it, which makes deploying it a weaker claim than a robots.txt disallow rather than a stronger one.
Flow: Crawler arrives at host to GET /robots.txt; GET /robots.txt to Match its own product token; Match its own product token to Allowed or disallowed; Allowed or disallowed (if allowed) to GET the page; Crawler arrives at host (no vendor documents reading it) to tdmrep.json or header; Crawler arrives at host (never requested) to Terms or copyright page.
How we read the legal pages of 656 sites
Every hostname in the two committed corpus seed files was asked for /robots.txt once as LantadBot/1.0, redirects followed, twenty second timeout. 1,090 answered HTTP 200, 1,053 of those delivered something other than an HTML page at that path, and 1,007 parsed into at least one user-agent group. The 1,407 hostnames whose robots.txt did not disallow our own token at the root were then asked for their home page, and 1,058 answered HTTP 200 with an HTML content type.
Finding the legal page was done the way a crawler would have to do it, from the home page itself. Every same-host anchor whose path or anchor text named terms, legal matters, conditions of use, acceptable use, copyright, an end user licence, a user agreement, a disclaimer or intellectual property was collected, and up to two of them plus one privacy page were fetched per host. 576 of the 1,058 readable home pages linked at least one such page. 1,359 pages were fetched, 1,266 answered HTTP 200 carrying at least 80 words of text, and 656 hostnames ended with at least one readable legal page. That number is the ceiling on this finding and the main thing to hold against it: a site whose terms are reachable only through a menu built in JavaScript, or which are not linked from the home page at all, is invisible here, and the methodology page states the same no-JavaScript rule the rest of our scanning follows.
Candidate clauses were found by a term search and then read one by one. The search took every occurrence of an AI, machine learning or data mining term, took 260 characters of context either side, and kept the window only if it also contained a prohibition addressed to a reader and a reference to the site's own content or pages. That produced 32 hostnames. Reading all 32 by hand cut them to 15. The 17 discarded were mostly sites selling an AI product, whose clauses restrict what you may do with their AI features or disclaim the accuracy of its output, and one was not a legal page at all but a news article about AI and court records. A regular expression cannot tell those apart from a reservation over published content, which is why the number in the title comes from reading rather than from matching. Our own LantadBot page states what this crawler requests and why, and the robots.txt validator findings from 19 September cover how often these files are broken in the first place.
15 sites reserved the right, and 11 allowed all nine training crawlers
The nine tokens this scanner classes as training crawlers are GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot. For each of the 15 hostnames the root path was evaluated against each token with the same function the product uses, and 11 came back allowed on all nine. Four did not. bostonglobe.com and statnews.com block all nine, slate.com blocks eight of the nine, and coursera.org blocks one, Meta-ExternalAgent, while naming five. Three of those four are news publishers, which is the pattern the whole corpus shows: of the 1,007 usable robots.txt files, only 109 block at least one training token and 26 block all nine.
So the clause and the file agree on four sites out of fifteen. On the other eleven the reservation exists in a sentence a person reads and nowhere a machine reads, and five of those eleven are the kind of organisation with the most at stake in the question. aafp.org, acog.org and psychiatry.org are medical bodies publishing clinical guidance. wiley.com is an academic publisher whose copyright page reserves "rights for text and data mining and training of artificial intelligence technologies or similar technologies". wagamama.com forbids text or data mining "for any purpose, including the development, training, fine-tuning or validation of AI systems or models". Each of those sentences is stronger than anything in the file served at the same hostname.
Widening from the 15 confirmed clauses to a plain phrase count across all 656 sites shows the same shape at a larger scale, and the count is mechanical rather than read: the phrase appears in the text, which is all that is claimed. 120 of the 656 use the phrase artificial intelligence somewhere in a legal page and 99 of the 115 with a usable robots.txt allow all nine training tokens. 108 mention scraping, 67 machine learning, 32 data mining, and only 5 use the full phrase text and data mining. A phrase appearing is not a reservation, which is exactly why the headline figure came from reading 32 windows rather than from this table, but the direction is consistent at every term. Last month's measurement of the Content-Signal declarations found the same disconnect one layer further in, where 7 of 22 sites declaring ai-train=no blocked no training crawler by name. The AI crawler reference lists the tokens and what each vendor says it does.
| Phrase in a legal page | Hostnames | With usable robots.txt | Allowing all nine training tokens |
|---|---|---|---|
| artificial intelligence | 120 | 115 | 99 |
| scrape or scraping | 108 | 102 | 87 |
| machine learning | 67 | 65 | 57 |
| data mining | 32 | 29 | 24 |
| large language model | 22 | 20 | 17 |
| text and data mining | 5 | 5 | 3 |
One host, two files, two answers
sonarsource.com is the clearest case in the set because it did not forget the crawlers. Its website terms of use prohibit content that "trains or uses the Content in connection with the development or operation of a machine learning or artificial intelligence (AI) system", and the clause goes further than any other we read, naming "training, fine-tuning, or grounding the machine learning or AI system or as part of retrieval-augmented generation". That last phrase covers the retrieval crawlers as well as the training ones, which is a distinction most terms pages miss entirely.
Its robots.txt, fetched the same day, carries a comment introducing an AI crawler section as an explicit allow, then 31 user-agent lines in a single group: GPTBot, ChatGPT-User, Claude-Web, Google-Extended, Google-CloudVertexBot, GoogleAgent-Mariner, Gemini-Deep-Research, Google-NotebookLM, PerplexityBot, Perplexity-User, anthropic-ai, ClaudeBot, Claude-User, Claude-SearchBot, CCBot, BingBot, Applebot-Extended, Meta-ExternalAgent, OAI-SearchBot, DuckAssistBot, Bytespider and others. The rules attached to that group are Allow: / with two parameter and tag path exclusions. Eight of our nine training tokens are named in it and all eight are allowed at the root. The file then adds a Content-Signal line reading ai-train=yes, search=yes, ai-input=yes, which is Cloudflare's declarative syntax saying in machine-readable form the opposite of what the terms page says in prose. When we opened Cloudflare's ten Browser Run quick action documentation pages in September, only one of the ten named Content Signals or robots.txt at all, so the line is a declaration whose readership is still being built.
This is not carelessness about crawlers. Somebody wrote 31 tokens by hand, kept the list current enough to include agents that shipped this year, and added a signal line. The terms page and the robots file were simply written by different people solving different problems, and only one of them is read by the thing both are about. Where the two conflict, the file wins, because the file is the only one fetched. What GPTBot sees renders the same split for any URL: the verdict from the file, not the intent from the page.
GET https://sonarsource.com/robots.txt
- User-agent: * Allow: /
- User-agent: GPTBot named
- User-agent: ClaudeBot named
- User-agent: anthropic-ai named
- User-agent: Google-Extended named
- User-agent: Applebot-Extended named
- User-agent: Meta-ExternalAgent named
- User-agent: CCBot named
- User-agent: Bytespider named
- Allow: / all 8 allowed at the root
- Disallow: /*? parameter paths only
- Content-Signal: ai-train=yes, search=yes, ai-input=yes declares training permitted
Who served the machine-readable reservation
All 1,419 hostnames were asked for /.well-known/tdmrep.json. 922 answered 404 and 237 answered 403. Eight answered HTTP 200 with a body that was not an HTML page, and reading each one leaves six that actually declare a reservation: cambridge.org, france24.com, nature.com, science.org, sciencedirect.com and unesco.org. Every one of the six sets tdm-reservation to 1 for location "/" and names a tdm-policy URL, four of them on a publisher domain rather than the site itself. Of the other two, healthline.com serves a Global Privacy Control document at that path instead, carrying gpc and a lastUpdate of 2023-12-15 and no reservation of any kind, and spectator.co.uk answers 200 with an empty body.
The header and the markup are rarer still. Asking all 1,419 for their home page with a HEAD request returned a tdm-reservation response header on two hosts, sciencedirect.com and unesco.org, both set to 1, and both are among the six already serving the file. Across the 1,058 home pages read in full, not one carried a tdm-reservation meta element. So of the four methods TDMRep defines, this corpus uses two, and six hostnames out of 1,419 account for all of it.
None of the six is one of the 15 that wrote the reservation in prose. At that path the 15 returned 13 answers of 404 and 2 of 403. The split is close to a split between kinds of organisation: the six that shipped a file are four academic and scientific publishers, one broadcaster and one UN agency, and none of the 15 is any of those things. wiley.com is the instructive exception, because its copyright page reserves the same rights as its peers and it serves no file. Robots.txt tells the same story for the six, and not a flattering one: nature.com and sciencedirect.com block all nine training tokens, unesco.org seven, france24.com and science.org five, and cambridge.org blocks none at all, so it ships the machine-readable reservation and leaves the documented mechanism open. A standard nobody has promised to read is a weak place to put your only copy of a rule, which is the same trap as the AI preferences standard you cannot deploy yet and the reason publisher tolls stay invisible in robots.txt.
| Hostname | tdmrep.json | Header | Training tokens blocked of 9 |
|---|---|---|---|
| nature.com | reservation 1 | no | 9 |
| sciencedirect.com | reservation 1 | yes | 9 |
| unesco.org | reservation 1 | yes | 7 |
| france24.com | reservation 1 | no | 5 |
| science.org | reservation 1 | no | 5 |
| cambridge.org | reservation 1 | no | 0 |
The four that blocked, and a blocklist aimed at the last decade
The four hostnames whose file matches their clause did it the ordinary way, by naming the tokens. bostonglobe.com and statnews.com each name all nine and disallow all nine, in files of 1,373 and 1,338 bytes carrying 25 and 26 groups. slate.com names and blocks eight, and the ninth, anthropic-ai, is the legacy Anthropic token that newer blocklists tend to drop. coursera.org names five and blocks one. None of that requires a standard that does not exist yet or a lawyer: it requires a group per token and a Disallow.
wiley.com is worth reading closely because it shows the failure is not indifference to crawlers either. Its robots.txt is a maintained file with nine groups. Eight of them disallow a named crawler at the root: CazoodleBot, MJ12bot, dotbot, Gigabot, trendkite-akashic-crawler, AhrefsBot, MauiBot and Barkrowler. Every one of those is an SEO, backlink or media-monitoring crawler, the set a publisher worried about bandwidth and competitive scraping would have blocked in 2015. Not one of the nine AI training tokens appears, and the same company's copyright page reserves text and data mining and AI training rights explicitly. The blocklist was built for the previous decade's problem and never updated for this one.
What none of this settles is whether the prose has legal force. A reservation of rights may well bind somebody who reads it, and the EU framework specifically contemplates machine-readable expression, which we traced in July when we found that the EU AI Act never writes robots.txt while its Code of Practice names the file twice. That is a question for lawyers and we are not answering it. The narrow measurement is this: on 14 October 2026, eleven of these fifteen sites served every documented AI training crawler an open path to content their own legal pages say may not be used for training. If the worry on the other side is search traffic, the two controls are separate: on 16 September we found that 52 of the 54 sites refusing Google-Extended still allowed Googlebot, so a training refusal does not have to touch the search crawler at all.
What the copyright page reserves
- All rights reserved, including rights for text
- and data mining and training of artificial
- intelligence technologies or similar technologies.
- Of the nine AI training tokens: 0 named.
What robots.txt disallows at the root
- CazoodleBot
- MJ12bot
- dotbot
- Gigabot
- trendkite-akashic-crawler
- AhrefsBot
- MauiBot
- Barkrowler
What to check on your own site
The check is short because the mechanism is short. Open your terms, copyright and acceptable use pages and find out whether they say anything about AI, machine learning, data mining or scraping. If they do, open the robots.txt at the same hostname and read it for the tokens that clause is aimed at. Those two documents were almost certainly written by different people, and on eleven of the fifteen sites here they do not agree.
Three details decide whether the file does what the clause intends. Tokens have to be spelled as the vendor publishes them, because a group header is matched against the product token and nothing else; we found 84 mis-cased tokens that still bound on 14 October, but a token spelled wrongly rather than mis-cased binds nothing. Old names go stale, and we have watched renamed crawler tokens leave stale robots rules behind that match no crawler at all. Edits are not instant either. RFC 9309 asks crawlers not to reuse a cached copy for more than 24 hours, and the published field evidence is worse than that: a controlled experiment on university web logs reported fewer than 40 percent of AI assistant and AI search bots re-fetching robots.txt inside a 168 hour window, which we read in August when asking when a robots.txt edit actually reaches a crawler.
If the aim is the opposite, being read and cited rather than refused, the same file decides it and the same precision applies, which is the whole of what AI visibility measures. Either way, the sentence on the terms page is not the control. It is a statement of intent published next to the control, and if the two disagree, every crawler in the list above acts on the file.
- Read your own legal pages for AI, mining or scraping language 656 of 1,419 corpus hosts served a legal page a crawler could reach without JavaScript; 15 carried a clause restricting AI training
- Evaluate robots.txt per training token, not per wildcard 11 of those 15 allowed all nine of GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot at the root
- Check the token spellings against each vendor's own page 5 of the 15 named a training token at all; a mis-spelled group header matches no crawler and reports nothing
- Decide where the machine-readable copy lives 6 of 1,419 hosts served a TDMRep file and no vendor documents reading it, so robots.txt remains the only mechanism all of them name
Lantad
Published .
Fifteen sites in this corpus tell you in writing that you may not use their pages to train a model. aafp.org states that unauthorized use of its content to train generative artificial intelligence technologies "is expressly prohibited". psychiatry.org writes that "All rights, including for text and data mining (TDM), Artificial Intelligence (AI) training, and similar technologies, are reserved." foxnews.com spells out that you "may not and shall not copy, data mine, scrape or in any way extract or use the Content" for training any artificial intelligence algorithm, system, model or tool. Those sentences are unambiguous and they are addressed to a reader.
Common questions
Does a terms of service clause stop AI crawlers?
Not by itself, because no crawler fetches it. OpenAI's crawler documentation and Anthropic's crawler support article both name robots.txt as the control and neither offers any other route, and RFC 9309 locates the rules a crawler reads at /robots.txt in the top-level path. A clause may have legal force against somebody who reads it, which is a separate question, but on 14 October 2026 eleven of the fifteen sites in this corpus with such a clause served all nine training crawler tokens an allowed root.
What is the machine-readable way to reserve text and data mining rights?
The TDM Reservation Protocol, a W3C Community Group Final Report dated 2 February 2024, defines four: a tdmrep.json file under /.well-known, a tdm-reservation HTTP response header, a tdm-reservation meta element in HTML, and EPUB package metadata. The value is a boolean where 1 means reserved. The report states it "is not a W3C Standard nor is it on the W3C Standards Track", and the crawler documentation of OpenAI, Anthropic and Google named none of the four when we read it in August 2026, so it is an addition to a robots.txt disallow rather than a replacement for one.
Which robots.txt tokens cover AI training?
The nine this scanner evaluates as training crawlers are GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot. They are not interchangeable with the search and user-triggered tokens such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, which decide whether an answer engine can cite you rather than whether a model can train on you. Of 1,007 usable robots.txt files in this corpus on 14 October 2026, 109 blocked at least one of the nine and 26 blocked all nine.
How many sites write an AI clause but allow the crawlers?
In this corpus, 11 of 15. Those 15 came from reading 32 candidate clauses found across the legal pages of 656 hostnames on 14 October 2026, and 11 of the 15 left all nine training tokens allowed at the root. A wider phrase count on the same 656 sites points the same way without being a reading of intent: 120 used the phrase artificial intelligence in a legal page and 99 of the 115 with a usable robots.txt allowed all nine.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
Block AI training: 7 of 22 sites declaring ai-train=no blocked no crawler
Lantad requested /robots.txt once from each of 1,027 hostnames on 13 September 2026. 736 answered HTTP 200 and 38 of those carry a Content-Signal line. Twenty two of the 38 declare ai-train=no. Fifteen of the 22 also disallow eight or nine of the nine training crawler tokens this scanner evaluates by name, and seven disallow none of them at the site root.
Three crawler documentation pages name robots.txt, and none names TDMRep
The W3C TDM Reservation Protocol defines four places a site can record that AI mining rights are reserved. Read on 4 August 2026, the crawler documentation published by OpenAI, Anthropic and Google names robots.txt in every case and mentions none of the four.
The EU AI Act never writes robots.txt, and its Code of Practice does twice
Regulation (EU) 2024/1689 obliges general-purpose AI model providers to identify and comply with machine-readable rights reservations, and the phrase robots.txt appears nowhere in it. The Commission's Code of Practice names the file and the RFC that defines it, and the Commission's power to fine under Article 101 applies from 2 August 2026.
Content signals robots.txt: one of ten Cloudflare endpoints reads it
Cloudflare's changelog of 31 August 2026 records that the Browser Run crawl endpoint now respects the Content Signals use directive and rejects a job with HTTP 400 when the caller declares a more permissive level than the site does. We opened all ten Browser Run quick action documentation pages on 3 September 2026: nine of them name neither Content Signals nor robots.txt.
