Is robots.txt legally binding? An EU draft makes it evidence, and 4 of 1,419 sites served the other file it names

Is robots.txt legally binding?

No, and the draft that prompted this post is careful about it in both directions. RFC 9309, the Robots Exclusion Protocol, published on the standards track in September 2022, is a protocol specification. It defines two rule keywords, allow and disallow, and it tells a well behaved crawler what to do with them. It confers no rights on a site owner and imposes no duty on anybody who declines to implement it. That has always been the position, and it is why a robots.txt line has historically been a request rather than a boundary.

What EDPB Guidelines 03/2026 adds is not enforcement but evidential weight, inside a test that already exists. Article 6(1)(f) of the GDPR allows processing that is necessary for a controller's legitimate interests except where those interests are overridden by the interests or fundamental rights of the data subject. Working out which side wins involves asking what the data subject could reasonably expect. Paragraph 64 of the draft lists five things a controller should consider when answering that, and one of them is the restrictions imposed by the scraped website, through measures the draft names as robots authentication to view content, robots.txt or ai.txt files, or CAPTCHA. The draft's own words on the consequence, read from the published PDF rather than from a summary of it, are that if data subjects are aware that a website implements these measures and a website is nevertheless scraped, it is less likely that they can expect the processing of their personal data by a scraping entity.

Then it closes the obvious loophole in the other direction. Paragraph 45 states that the absence or non-applicability of a robots.txt file on a web site does not amount to consent within the meaning of the GDPR. That matters more than it looks, because the protocol itself says the opposite about access: a 4xx at the robots.txt address means the file is unavailable and the crawler may go ahead, which is the behaviour an earlier measurement here covers in detail. So the two documents now disagree in a useful way. Under RFC 9309 a missing file is an open door. Under this draft a missing file is simply silence, and silence is not permission. A site that has never written a robots.txt has not thereby agreed to anything, even though every crawler will read the absence as a yes.

The practical reading for a site owner is unglamorous. A robots.txt rule is still the only instrument that actually changes what a crawler fetches, which is why it is worth getting right whatever any regulator says, and the robots.txt tester will tell you what yours says per crawler. What the draft adds is that the same file may later be read as evidence of what you wanted, which is an argument for writing it deliberately rather than inheriting it from a template. None of that makes it binding, and nothing in this post is legal advice: this is a reading of a published draft, and the draft is still open for comment.

What the draft asks a scraper to check before it collects

The draft describes dataset building as four steps, and puts the obligation early. At paragraph 11 the collection step includes implementing measures to exclude certain sources, and the example it gives is websites which oppose web scraping with measures such as robots.txt or CAPTCHA. At paragraph 37, under data minimisation, it becomes a list of things the controller should do before collecting anything, one of which is to exclude from the collection websites which clearly oppose the scraping of their content, through the use of technical measures, such as the use of robots authentication to view content, robots.txt or ai.txt files, or CAPTCHA. The verb is exclude and the timing is before, which together describe a filter applied to a crawl frontier rather than a defence raised after the fact.

Two worked examples in the draft show how much weight each measure carries on its own, and the answer is that the draft prefers them stacked. Example 4 describes a person uploading data to a freely accessible content sharing platform that does not contain any prohibitions against web scraping and that communicates the possibility of the content being scraped to its users, and concludes that data subjects can reasonably expect third parties to scrape that data to develop AI models. Example 5 is the mirror: a platform that prohibits scraping through the use of robots.txt files and the implementation of CAPTCHA, and expressly states on their site that they do not allow the use of their users' data for the development of AI models. There, the draft says, they cannot reasonably expect third parties to scrape that data for that purpose.

Read as an engineer rather than a lawyer, Example 5 is a conjunction of three separate things: a machine readable rule, an anti bot measure, and prose on a page. Each of those lives in a different place and is maintained by a different person at most organisations, and only the first is something a crawler fetches as a matter of routine. The gap between prose and a machine readable rule is not hypothetical either. An earlier count on this same corpus found fifteen sites whose own terms restrict using their content to train models, and eleven of the fifteen left all nine training crawler tokens allowed at the root of their robots.txt. The sentence was addressed to a reader. The crawler never opens the terms page. The same split is running live in the other direction this month, since Meta's own commercial terms take effect on 30 October 2026 and 2 of 1,011 robots.txt files in this corpus named the crawler they describe.

That is the context for the measurement in the next two sections. If the draft's test turns partly on which technical measures a site has implemented, then how many sites have implemented them is a question with a number attached, and it is a question this repository is already set up to answer. The method here is deliberately plain, and it is the same one the rest of the research on this site uses: ask the server, record what comes back, and treat anything ambiguous as ambiguous until it has been read by hand. What a scanner can see is exactly what the draft's filter would see, because both are reading the same bytes from the same paths.

The second file the draft names is served by four sites in 1,419

robots.txt appears four times in the draft and ai.txt twice, in both cases as an alternative of equal standing: robots.txt or ai.txt files. So we asked for it. On 20 October 2026 Lantad requested two paths from every one of the 1,419 hostnames in this repository's two committed corpus seed files, worker/seeds/corpus-seeds-platform.json with 392 hostnames in ten platform strata and worker/seeds/corpus-seeds-industry.json with 1,027 hostnames in eight industry strata. The first path was /.well-known/ai.txt, which is where the ai.txt Internet-Draft requires the file to be served, section 2.1 of draft-car-ai-txt-wellknown-00 being explicit that it MUST sit under /.well-known/. The second was /ai.txt at the site root, the older placement that predates the draft. That specification was read here when it appeared, including its 22 field names and the deny default it sets for training, and that post said plainly that it had measured nothing about how many sites publish one. This is that count. Every request went as LantadBot/1.0, the agent described on our bot page, with redirects followed and no JavaScript executed.

112 hostnames answered HTTP 200 at /.well-known/ai.txt and 115 at /ai.txt, for 134 distinct hostnames answering at one path or the other. Almost all of that is noise, for reasons the next section is about, and after the catch-alls were removed and every survivor was read by hand, five hostnames served a plain text file at an ai.txt path: sammylabs.com, uio.no, louvre.fr, freshworks.com and edx.org. Four of the five declare an AI usage preference. The fifth, edx.org, serves a site index in the style of an llms.txt file, 3,645 bytes of headings and links with no user-agent line, no allow or disallow rule and no policy field anywhere in it, so it declares nothing about AI usage at all and is counted as a file that is not a preference.

The four that do declare something do not point the same way, which is the part worth sitting with. uio.no serves 83 bytes at both paths, opening with the comment that no content may be crawled for AI purposes, then a wildcard user-agent and two disallow lines. freshworks.com writes four content signal fields, search: yes, ai-input: yes, ai-train: no and use: reference, so it refuses training while inviting retrieval, which is the same shape as the Content-Signal approach covered here before. louvre.fr serves a file whose own first comment names Spawning AI, and whose ten disallow lines are all image file extensions, from .bmp to .webp, above 45 allow lines including the root: an opt out of image datasets and an opt in to everything else. And sammylabs.com, the only one of the four that is permissive by design, states in its comment block that it grants permission to index, retrieve and train, adds that this declaration is express, and asks that where a national or regional regime conditions opt-in or opt-out on machine-readable signals, this file be treated as the definitive expression of those preferences for sammylabs.com.

So the file a European regulator names as a way of clearly opposing scraping is served, in a 1,419 site frame, by four sites, and the single most legally self-aware of the four is using it to say yes. That is not an argument against the draft. It is a statement about what a filter built on paragraph 37 would actually find if it ran this week, and anyone measuring their own AI visibility should know that the second file in that sentence is, for now, almost entirely theoretical.

Why a 200 at the path is not evidence that the file is there

The figure in the last section is four, and the raw count of hostnames answering HTTP 200 at an ai.txt path was 134. The distance between those two numbers is the whole methodological content of this post, and it is a trap that any published count of llms.txt, ai.txt, security.txt or any other well known file walks into unless it is specifically avoided.

A large share of the web answers 200 to requests for paths that do not exist. Single page applications serve their shell for any unmatched route. Content platforms route unknown paths to a soft 404 page that returns 200 with a body. Bot management systems return an interstitial. In every case the status line says success and the body is HTML, and a counter that keys on the status code records a file that was never there. So the check is to ask for something that certainly does not exist and see what happens. We requested /.well-known/lantad-control-7f3a9c2e.txt from each of the 134 candidates. 107 of them answered HTTP 200 to that too. Those 107 are not sites with an ai.txt file; they are sites that answer 200 to anything, and an uncorrected count would have reported them as adopters.

Of the 27 that did not answer 200 at the control path, 18 answered 404, 2 answered 403, 1 answered 405, 2 answered 500 and 4 did not answer at all. All 27 were then read by hand, which is what produced the five in the table above. 22 of the 27 were serving HTML: application shells on vercel.com, linear.app, postman.com and gap.com among others, a Korean government error page on seoul.go.kr, and in one case, hu-berlin.de, a page whose own title says it is making sure you are not a bot. None of the 22 contained a policy declaration of any kind. That hand read is not a formality. It is the step where a measurement either earns its number or does not, and it is the reason this post reports five files rather than 134 adopters.

Two things follow for anybody reading crawler statistics, including ours. The first is that a methodology which does not state its negative control cannot distinguish a file from a catch-all, so a published adoption rate for a well known file should be treated as an upper bound until it says how it excluded soft 200s. Our own methodology page carries the general version of this rule, and the specific version is above. The second is narrower and sharper: 107 of 134 is roughly four fifths, so on this evidence the error term in the naive measurement is several times larger than the signal. A survey reporting that 134 sites in a 1,419 site frame serve an ai.txt file, which is 9.4 percent and would read as early but real adoption, would be overstating the five files that exist by a factor of 27.

robots.txt names crawlers one at a time, so it is a poor yes or no

The draft's phrasing at paragraph 37 is websites which clearly oppose the scraping of their content, and at paragraph 64 the restrictions imposed by the scraped website. Both sentences treat the file as a position a site holds. A robots.txt file is not shaped like that. It is a set of groups, each naming one or more product tokens and carrying rules that apply to those tokens and to nobody else, and a site can refuse one company's crawler while admitting the next.

The counts, measured on 20 October 2026 across the same 1,419 hostnames: 1,146 answered HTTP 200 at /robots.txt, 37 of those served an HTML document at that address, and 1,089 parsed as a robots.txt carrying at least one user-agent, allow, disallow or sitemap line. 48 answered 404, 108 answered 403, 89 did not answer at all and 28 returned some other status. Of the 1,089 parseable files, 251 named at least one of the nine training crawler tokens that core/src/bots.ts classes with the purpose training, and 89 named all nine. Thirteen files carried a wildcard group whose rules disallow the root, which is the only configuration in the corpus that reads as opposition to every crawler rather than to a list of them: amsterdam.nl, seoul.go.kr, wa.gov, helsinki.fi, reuters.com, wsj.com, theregister.com, eluniversal.com.mx, sap.com, gmarket.co.kr, instacart.com, botcity.dev and coralvilleanimalhospital.com.

Set those two figures side by side and the difficulty is plain. 251 files express a view about at least one AI training crawler, and 13 express a view about all crawlers at once. Everything in between is a site that has opposed some named companies and not others, usually because it named the companies it had heard of. The per token counts run from 208 files naming GPTBot down to 144 naming Amazonbot, and the ordering tracks how long each token has been publicly documented more than anything a site could be said to hold a position about. GPTBot has been the most named token in every count run here, which is a fact about publicity rather than about preference. That pattern has shown up here repeatedly: a count of Applebot-Extended found 110 files naming Apple's training opt out and 61 of them leaving it allowed, and a count of CCBot found 15 files that blocked GPTBot while leaving CCBot alone. Neither of those is a policy. Both are a template plus whatever was in the news.

So the question the draft's test asks of a file is one the file is not built to answer. A site that disallows GPTBot and says nothing about Bytespider has clearly opposed one scraper, and whether that reads as clearly opposing the scraping of their content depends on who is asking. Something close to the stacked case in Example 5 does exist in this corpus, since 89 files name all nine training tokens and 13 close the door on everybody, but those are small minorities of 1,089. For the rest, the honest description is the one an earlier post here reached from a different direction, which is that for most sites the wildcard group is the whole of the AI policy, and it was written for search engines years before any of this. If you want to know which side of that line your own file falls on, the AI crawler list names the tokens and the tester resolves them per path.

What this post does not do is worth stating plainly, because the subject invites overreach. It does not measure CAPTCHA or robots authentication, two of the three measures the draft names, so the hero table marks them unmeasured rather than guessing. It does not assess any site's legal position, which is not ours to assess and would require facts about each site that no crawl can see. It does not report what any AI crawler actually did with any of these files, because nothing here observed one: every statement about crawler behaviour in this post comes from a published specification or a vendor's own documentation. It also treats a 403 at the robots.txt address as an unread file rather than as a refusal aimed at this crawler, which an earlier control on this corpus found is usually a decision about the network a request came from. And the 1,419 hostnames are a sampling frame committed to this repository for a different study rather than a sample of the web, so none of these proportions generalises past them.