The EU AI Act never writes robots.txt, and its Code of Practice does twice

What Article 53 actually obliges a model provider to do

The operative sentence is short. Article 53(1)(c) of the AI Act requires providers of general-purpose AI models to "put in place a policy to comply with Union law on copyright and related rights, and in particular to identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed pursuant to Article 4(3) of Directive (EU) 2019/790".

Three things in that sentence are worth separating. The obligation is to have a policy, not to achieve an outcome. The means are described as state-of-the-art technologies, which is a moving standard rather than a named one. And the thing to be identified is defined by cross-reference, so the AI Act itself never has to decide what a reservation looks like.

The reach is wider than the drafting suggests. Recital 106 states that any provider placing a general-purpose AI model on the Union market should comply with this obligation "regardless of the jurisdiction in which the copyright-relevant acts underpinning the training of those general-purpose AI models take place", and gives the reason as a level playing field, so that no provider gains an advantage in the Union market by applying lower copyright standards. A model trained entirely outside the EU is not outside the obligation if the model is placed on the EU market.

Article 53(2) exempts free and open-source models from the documentation duties in points (a) and (b), and recital 104 says in terms that the exemption "should not concern" the training-content summary or the copyright policy. So the copyright policy obligation survives an open-source release. That matters for anyone reasoning about which AI crawler traffic is covered, because the crawler fleets behind open-weight models are not carved out. What none of this does is tell a site owner what to publish. For that the Regulation hands off, and the next document is where the word machine-readable finally appears.

Where the words machine-readable come from

The cross-reference lands on Directive (EU) 2019/790, the copyright directive that predates the current wave of models by several years. Its Article 4 creates a text and data mining exception for reproductions and extractions of lawfully accessible works. Article 4(3) then attaches the condition that decides everything: the exception "shall apply on condition that the use of works and other subject matter referred to in that paragraph has not been expressly reserved by their rightholders in an appropriate manner, such as machine-readable means in the case of content made publicly available online".

That is the whole legal basis for the idea that a machine-readable file on your site carries weight. It is a condition on an exception, not a right to be granted, which is why the burden sits on the rightholder to express something rather than on the crawler to ask.

Recital 18 of the same Directive goes one step further and is the closest either instrument comes to naming a mechanism: for content made publicly available online, "it should only be considered appropriate to reserve those rights by the use of machine-readable means, including metadata and terms and conditions of a website or a service". Two examples, and robots.txt is neither of them. Metadata and terms and conditions are what the drafters had in mind in 2019.

This is the origin of a persistent confusion. The AI Act uses the phrase machine-readable four times in its own text, and not one of those four is about rights reservations: they concern the EU database, the CE marking, and the Article 50 duty to mark synthetic outputs as artificially generated. The machine-readable that matters for crawling is borrowed wholesale from a 2019 directive whose examples are page metadata and website terms, both of which sit outside the file most people are editing when they think about AI crawler access.

The Code of Practice is the document that names robots.txt

The European Commission published the General-Purpose AI Code of Practice on 10 July 2025 in three chapters, Transparency, Copyright, and Safety and Security. The Copyright Chapter, authored by Working Group 1 co-chair Alexander Peukert and vice-chair Céline Castets-Renard, is a six page document, and it is the first place in this chain where the file has a name. The chapter is hosted on the Commission's digital strategy site, which is not a host this blog is permitted to link, so the URL is here as plain text: https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai

Measure 1.3, headed "Identify and comply with rights reservations when crawling the World Wide Web", commits signatories "to employ web-crawlers that read and follow instructions expressed in accordance with the Robot Exclusion Protocol (robots.txt), as specified in the Internet Engineering Task Force (IETF) Request for Comments No. 9309, and any subsequent version of this Protocol". RFC 9309 is the same Standards Track document that decides what a 404 and a 503 on robots.txt mean, and it is now load bearing in a compliance instrument as well as in a crawler.

Measure 1.3 does not stop at the file. Point (1)(b) extends the commitment to "other appropriate machine-readable protocols to express rights reservations", giving asset-based or location-based metadata as the example, and conditioning it on adoption by a standardisation body or on being state-of-the-art and widely adopted. That is a live description of a gap rather than a settled rule, and it is the same gap that leaves the IETF AI preferences vocabulary undeployable on a real site today.

Two further points deserve attention from anyone who watches crawler behaviour. Point (4) commits signatories to make public information about "the web crawlers employed, their robots.txt features and other measures" and to provide automatic notification when it changes, which is the documentation a crawler registry depends on. Point (5) encourages signatories that also run a search engine to ensure that honouring a reservation for training does not lead to adverse effects on indexing, which is the separation between training and retrieval that crawler tokens with no user agent were invented to express.

One thing the Code is not is binding. Article 53(4) of the AI Act makes codes of practice a means to demonstrate compliance until a harmonised standard is published, and the Code's own objectives state that adherence "does not constitute conclusive evidence of compliance". A provider that does not sign has to demonstrate alternative adequate means. So robots.txt is not required by law; it is the mechanism the main compliance route names.

What changes on 2 August 2026

The dates are set by Article 113, and they are worth reading literally because the headline dates in circulation flatten a distinction the text makes. Article 113 states that the Regulation "shall apply from 2 August 2026", then carves out three exceptions: Chapters I and II from 2 February 2025; "Chapter III Section 4, Chapter V, Chapter VII and Chapter XII and Article 78" from 2 August 2025, "with the exception of Article 101"; and Article 6(1) with its corresponding obligations from 2 August 2027.

Chapter V is where Article 53 lives, so the copyright policy obligation has applied since 2 August 2025. Article 101 is expressly pulled out of that early start, which leaves it on the general date. Article 101 is the enforcement provision for this specific population: it empowers the Commission to impose on providers of general-purpose AI models fines "not exceeding 3 % of their annual total worldwide turnover in the preceding financial year or EUR 15 000 000, whichever is higher", where it finds the provider intentionally or negligently infringed the relevant provisions, failed to comply with an information request, failed to comply with a requested measure, or failed to give access to the model for evaluation.

So the shape of the change is narrow and specific. The duty to have a copyright policy is a year old. What arrives on 2 August 2026 is the Commission's power to fine for failing it. That is a change in consequence rather than a change in requirement, and it is the reason the date is being written about at all.

Nothing about that date changes a byte on your website. It does not create a new file to publish, a new header to send, or a new directive syntax to adopt. If your robots.txt already expresses what you want, the same file expresses it on 3 August. What changes is on the other side of the request, inside organisations that no external scan reaches, which is the point at which honest reporting of this story has to stop and hand over to what can actually be checked. Anyone reasoning about generative engine optimization on the strength of this date should be clear that it is a regulatory milestone, not a ranking event.

What an external scan can see, and what it cannot

This is the part where a measurement product has to be careful, because the temptation on a regulatory news day is to imply that a scan can report compliance. It cannot, and the reason is structural rather than a limitation anyone will engineer away.

What an external scan observes is one side of a conversation. It can fetch your robots.txt and parse it, evaluate the rules that apply to a named crawler token, and report which tokens are allowed and which are disallowed. It can fetch a page as a crawler and compare what the HTML carries against what a browser renders, which is what prose parity measures. It can read structured data and check whether a machine can identify the publisher. All of that is a property of bytes your server returns, so it is measurable by anybody, repeatedly, from outside.

What it cannot observe is the other side. Whether a given model provider fetched your robots.txt, parsed it the way RFC 9309 specifies, and honoured the result is an event inside their infrastructure. A scanner sees no trace of it. Neither, in practice, do your own logs with any certainty, because a user agent is a claim rather than an identity and confirming it requires a reverse lookup against published address ranges. The industry is building toward cryptographic answers, and what Web Bot Auth actually specifies is a signature on the crawler's request rather than anything a site publishes, so even that arrives on the request path and not in a scan of your pages.

The same limit applies to the two mechanisms recital 18 names. Terms and conditions are prose on a page: readable, but there is no agreed syntax that would let any scanner say a reservation has been validly expressed rather than merely written down. Asset-level metadata sits inside image and document files rather than in the HTML most scanners parse. So a tool that reported an EU rights reservation status would be inferring it, and Lantad does not carry that field. The honest scope is narrower and still useful: what you publish, what a crawler can read, and where the two disagree.

That reticence is the same rule that makes Lantad withhold a grade when a measurement failed rather than print a confident number. A regulation whose subject is somebody else's internal policy is not a thing to score a website on.

What to check on your own site this week

None of the above obliges a site owner to do anything. The obligations in Article 53 run to model providers, and a website has no compliance duty under it. What the chain does give you is a clear reason to know exactly what your own site currently says, because for the first time the main compliance route names the file you already have.

Start by reading it rather than assuming it. A robots.txt that was written for search engines in 2019 says nothing about the tokens that matter now, and a per-token evaluation is the only way to see the result rather than the intent: test robots.txt against each AI crawler and read the verdict for each one separately. The common surprise is not a rule that blocks too much. It is a file that was never updated and therefore allows everything, which is a decision nobody made.

Second, separate training from retrieval deliberately. Blocking a training crawler and blocking a retrieval crawler have different consequences, and only the second removes you from answers. If being cited matters, check what the platforms document about their retrieval fleets before you write a Disallow: the requirements for getting cited in ChatGPT and in Claude are published by their vendors and are not the same as the training tokens.

Third, check that allowing a crawler is enough. Access is only the first of two layers that decide whether AI can read your site, and a permissive robots.txt in front of a page that renders its text in the browser leaves a crawler with an empty document. Seeing what GPTBot actually receives settles that in one request, and it is the check most likely to change what you do next.

Finally, resist the urge to add a file because a regulation is in the news. The evidence on llms.txt is that publishing one changes nothing measurable about whether crawlers fetch it, and nothing in the AI Act, the Directive or the Code of Practice mentions it. The mechanisms those documents name are robots.txt, metadata and website terms. Two of those you already have, and the third is not a file you can add this afternoon.