A robots.txt edit does not take effect when you save it

What the experiment did, and where these numbers come from

The paper is Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study, by Taein Kim, Karstan Bock, Claire Luo, Amanda Liswood, Chloe Poroslay and Emily Wenger, most of them in the Department of Electrical and Computer Engineering at Duke University. It was first posted to arXiv on 27 May 2025 and revised to a second version on 23 October 2025, and it appears in the proceedings of the 2025 ACM Internet Measurement Conference, held in Madison, Wisconsin from 28 to 31 October 2025.

The dataset is anonymised web traffic from 36 websites run by a large private US university, covering roughly 3.9 million external requests between 12 February and 29 March 2025. Grouping those requests into sessions, where five minutes of inactivity ends one, reduces 3,914,096 rows to 761,956. Across the whole set there are 19,250 distinct user agent strings and 8,841 distinct autonomous system numbers, but only 405 of those user agents match a known bot signature. The authors say plainly what that ratio implies, which is that most automated traffic never announces itself in a form anyone can act on. That is the same problem described here in a user agent is a claim, not an identity, arriving from the opposite direction: not a false name, but no name at all.

The controlled part ran on one site chosen for its bot volume. Four versions of robots.txt were deployed in sequence, each for two weeks, giving eight weeks in total. The base version blocked three paths. Version one added a 30 second crawl delay. Version two restricted most bots to a single endpoint. Version three disallowed everything for most bots. Eight search engine crawlers were exempted throughout at the institution's request. Traffic held steady across the four deployments, at between 111,206 and 119,399 unique site visits and between 75 and 87 unique bot visitors, which is what makes the four periods comparable at all.

The bot mix in the wider dataset is worth noting before reading anything into per-vendor figures. YisouSpider and Applebot together drove about 30 percent of all traffic. GPTBot accounts for 1,225 hits, or 0.16 percent, and ClaudeBot for 684, or 0.09 percent. Those are small samples for the crawlers most people care about, on a university estate rather than a commercial site, and the paper does not present them as representative of the web. The token counting problem is separate again and covered in the crawler tokens that never appear in your logs: some robots.txt product tokens send no user agent of their own, so they cannot appear in a log study however large it is.

The AI crawlers complied better than the grey literature predicts

Before the uncomfortable finding, the convenient one, because leaving it out would misrepresent the paper. Ranked by weighted compliance across the three directives, SEO crawlers came first at 0.695, and the next two categories were AI search crawlers at 0.631 and AI assistants at 0.616. Both sit above the 0.420 average across every category. Search engine crawlers came in at 0.447, below both AI categories. Headless browsers, which is where a scraper running a browser without a display window lands, scored 0.155.

The authors flag this as surprising given how much has been written about AI bots ignoring robots.txt, and they name it as future work rather than explaining it away. The paper also records that Amazonbot, ClaudeBot and GPTBot each made notable efforts to comply with every directive, and that PerplexityBot, which had stated it will not respect robots.txt, nonetheless showed reasonably high compliance, which the authors read as either compliance by accident or as being more respectful than claimed. Running the other way, BrightEdge Crawler claims to respect robots.txt and complied poorly.

Compliance also fell as directives got stricter. Averaged across categories, the crawl delay directive drew 0.609, endpoint restriction 0.310 and disallow all 0.307. Crawl delay is the outlier and the paper is careful about why: a bot that was already slow complies with a 30 second delay without doing anything, so a high figure there is partly an artefact of default behaviour rather than evidence of obedience. That caveat matters more than it looks, because it is the mechanism behind the whole post: a bot can satisfy a directive it has never read.

None of this contradicts the divergence reported here previously between declared rules and served responses. 234 of 592 sites that ban GPTBot in robots.txt served it a 200 measured sites failing to enforce their own bans, and the sites that block AI crawlers are the ones with editors measured enforcement that was never declared. This paper measures a third thing again, which is the crawler's side of the same exchange. All three can be true at once because the file, the server and the crawler are three independent systems. What a site can actually address is narrower than the argument suggests: most vendors expose one lever, a point set out in six of the nine AI vendors we track publish one crawler token, and the current list of names worth writing down is enumerated in the AI crawler reference.

How often does a crawler actually re-read robots.txt

This is the part of the paper that changes what a site owner should expect from an edit. The authors first noticed that some bots complied with directives they had demonstrably never fetched, which raised the question of how often bots fetch the file at all. The counts are direct: 9 of 34 bots never fetched robots.txt during the crawl delay experiment, and 15 of 47 never fetched it during the endpoint access and disallow all experiments. Some of those complied anyway, which the paper treats as an open question with two candidate explanations, either compliance by coincidence or retrieval of the file from some cache the experiment could not see. It declines to claim which, and says investigating caching was out of scope.

To measure frequency rather than presence, the authors found other sites in their estate whose robots.txt carried meaningful restrictions and was publicly reachable. Only 3 of the 36 sites qualified. They then segmented each bot's access log into fixed windows of 12, 24, 48, 72 and 168 hours from that bot's first robots.txt fetch, and counted a bot as complying with a given window length only if every window of that length in the 40 day dataset contained a fetch. That is a strict definition, and it should be read as such: one missed window disqualifies a bot for that length.

The result is the headline. Scrapers, archivers and intelligence gatherers re-fetch within 12 hours consistently, if they fetch at all. AI assistants and AI search crawlers have the lowest re-check rates of any category, with fewer than 40 percent of them fetching robots.txt within a 168 hour window. A 168 hour window is a week. The categories whose output most directly feeds AI answers are the categories least likely to have looked at your file in the last seven days.

Two limits are worth stating alongside that. Three sites is a small base for the frequency analysis, and the paper says so. And the population it can see is the population that identifies itself, which excludes an entire class of client: seven of nine AI coding agents arrived without a product name reported agents presenting as HTTP libraries rather than products, and a study keyed on known bot signatures cannot count those at all. There is also a class of fetch that no re-read schedule would help with, because it is not decided by the wildcard group in the first place, which is the subject of fourteen Google agents that a robots.txt wildcard does not stop. Newer controls layered into the same file, such as the signal line examined in Cloudflare's Content-signal line asks, and the Disallow rules under it block, inherit exactly the same delivery problem: they only take effect when the file is next read.

The standard asks for 24 hours and cannot enforce it

There is a written expectation here, and it is much shorter than the measured behaviour. RFC 9309, the Robots Exclusion Protocol, published September 2022 on the standards track, addresses caching in section 2.4 in three sentences. Crawlers MAY cache the contents of the file. Crawlers MAY use standard cache control. Crawlers SHOULD NOT use the cached version for more than 24 hours, unless the file is unreachable. That is the whole provision, and the operative word is SHOULD NOT, which in specification language is a strong recommendation that a crawler may depart from with reason, not a requirement.

The largest crawler operator documents the same number. Google's robots.txt documentation, carrying Last updated 2026-07-08 UTC when read on 3 August 2026, states that Google generally caches the contents of a robots.txt file for up to 24 hours, but may cache it longer in situations where refreshing the cached version is not possible, and that the cache duration can be influenced by max-age Cache-Control headers. So the norm is not in dispute and it is not obscure. What the measurement shows is that the norm and the observed behaviour of AI-related bots are roughly a factor of seven apart, and that a meaningful share of bots do not participate in the norm at all.

This is worth being precise about, because the temptation is to read it as bad faith. It need not be. A crawler that visits a domain once a fortnight has no occasion to re-fetch robots.txt daily; the 24 hour figure is a ceiling on cache age, not a requirement to poll. The paper cannot separate a bot that caches too long from a bot that simply visits rarely, and it does not claim to. Either way the consequence for the site owner is identical: the rule you wrote today is not the rule being applied to you today.

The other half of the delivery problem is what happens when the fetch does occur but fails. The status code the server returns for robots.txt itself decides the outcome before any rule is parsed, and a 404 and a 503 mean opposite things to a conformant crawler, which is set out in a 404 on robots.txt allows every crawler, a 503 blocks them all. RFC 9309 also adds a clause directly relevant to a stale copy: if the file has been undefined for a reasonably long period, given as an example of 30 days, a crawler may either treat it as unavailable or continue to use a cached copy. A site that serves errors while it edits its rules can therefore keep an old policy in force for a month by specification. For anyone auditing this against a named crawler, the identity of the client is a separate question again, and the terms this site publishes about its own scanner are on the bot page.

What to do when a robots.txt edit needs to take effect

The practical reading is not that robots.txt is useless. It is that an edit to it is a request with an unknown delivery time, and anything that needs to be true immediately needs a second mechanism. Three things follow, and none of them require trusting a figure from a study of a university estate.

Treat the edit as the start of a wait, not the end of a task. If you have server logs, the fetch of robots.txt is itself a logged request, and the interval between those fetches per user agent is the only measurement of this that applies to your own domain. Nothing external can see it: a scan from outside observes what your file says and what your server returns, never when a particular crawler last read it. Knowing which group a token resolves to is a different check and a cheaper one, and resolving a real file against a specific token is what the robots.txt tester exists for. Reading the file top to bottom does not tell you which group binds, because a named group and a wildcard group can give the same file opposite meanings for two crawlers.

Second, separate the things robots.txt can do from the things only the server can do. A directive is advisory in both directions, and the paper's central finding is about latency rather than defiance, but latency and defiance produce the same server log. If a page must not be served to a given client starting now, that decision belongs at the origin or the edge, which is the layering argument in two layers decide whether AI can read your site. Robots.txt is the polite request; the response your server sends is the answer.

Third, and pointing the other way, most sites reading this are not trying to block anything. They are trying to be read, and for them the delay works in reverse: an edit that opens access is also on the crawler's schedule, so removing a Disallow line does not produce a visit. What decides the outcome then is whether the page returns usable text when a crawler does arrive, which is what what GPTBot sees fetches and what the gap named prose parity measures. A site can admit every AI crawler in its file and still return an empty shell. That failure needs no cache to explain it and no waiting to fix.

What this site contributes to the subject is a measurement of one domain at one moment rather than a population over time, and the boundaries of that are written down in the methodology. It resolves a real robots.txt against a registry of crawler tokens and reports what a crawler fetch returned, which feeds a component of an AI visibility score. It cannot tell you when any crawler last read your file, and any tool claiming otherwise is inferring rather than observing. Aggregate findings across scans go on the research page when the sample supports publishing them, and a reading of somebody else's experiment, which is what this post is, does not belong there.