Drupal AI crawlers: 79 of 86 robots.txt files named none, and 68 are Drupal's own file

Drupal AI crawlers: what the 86 files actually say

The method was deliberately narrow. One GET of https://hostname/ per corpus entry, sent as LantadBot/1.0 with redirects followed and no browser involved, then one GET of /robots.txt against the same hostname. A page counted as Drupal if its bytes carried any of three markers: a generator meta element naming Drupal, a reference to /core/misc/drupal.js or /core/assets/vendor/, or Drupal's own client side settings markers, being drupal-settings-json, Drupal.settings or a data-drupal- attribute. Fifty nine of the 86 carried the generator meta, so on two thirds of them the site says so itself and the other 27 were identified by machinery Drupal puts in the page.

Across the 86 files we searched for the 15 product tokens in this scanner's registry: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot. A token counts as named only when it heads a group, which is to say it appears on a User-agent line of its own, because that is the only position in which a product token does any work. Seven files named at least one. Seventy nine named none.

Then we evaluated each file against each of the 15 tokens at the site root, using the same parser a scan uses, and counted how many came back blocked. Seventy eight files allowed all 15. Eight blocked at least one. The eighth is worth pulling out now because it is the only site in the set that closes the door without naming anybody: helsinki.fi serves a five line robots.txt holding User-agent: *, Disallow: /, and a Sitemap line. That blocks all 15 tokens, and every other crawler, by a route that needs no maintenance and no vendor list.

The file lengths run from those five lines to 2,151 lines. Thirty seven of the 86 declare a Sitemap, which is lower than the 839 of 1,016 we counted across the whole corpus and is explained by the stock file, which carries no Sitemap line at all. Nothing here is a fault in Drupal. It is what a default looks like after it has been deployed a few thousand times and left alone.

Why 68 of the 86 files are the same file

Drupal ships a robots.txt in core, at the web root, and it opens with a comment block explaining what the file is for. That comment is distinctive enough to fingerprint. Sixty eight of the 86 files carry it verbatim, including the sentence that begins "This file is to prevent the crawling and indexing of certain parts of your site by web crawlers and spiders run by sites like Yahoo! and Google". Yahoo is still in the comment on 68 public institution websites in September 2026, which tells you roughly how often anybody opens the file.

Below the comment the shipped file is a single wildcard group. It allows crawlers to fetch stylesheets, scripts and images out of the framework directories, then disallows the framework directories themselves along with /admin/, /user/register, /user/password, /user/login, /user/logout, /search, /node/add/, /comment/reply/, /media/oembed, /README.txt and /web.config, each written twice so that the rule holds whether or not the site uses clean URLs. Seventy six of the 86 files carry that block. The current version, on the 11.x branch, is published at git.drupalcode.org/project/drupal/-/raw/11.x/robots.txt, which is written here as plain text because that host is not on this site's registered outbound list.

The block comes in two shapes and the difference dates the file. Drupal 7 allowed assets out of /misc/ and /modules/. Drupal 8 moved the framework into /core/ and the shipped file followed. Sixty nine of the 86 carry the /core/ form and seven carry the /misc/ form, with no file carrying both. So seven of these hostnames are serving a robots.txt whose asset paths belong to a major version that reached end of life, which says nothing about what version of Drupal is actually running and everything about the fact that the file was never touched.

What the shipped file has never contained, in any branch, is a crawler name. Not Googlebot, not Bingbot, and certainly not GPTBot. It is a file about paths, written before anybody needed a file about parties. That is the correct scope for a CMS to take and it is also why a Drupal site that has never been edited expresses no opinion about AI at all: under RFC 9309, a crawler that finds no group matching its own token obeys the wildcard group, and the wildcard group here grants the whole site apart from the administrative paths. An unstated position lands as an allow, which is the same place an empty file or a 404 lands by a different road.

Crawl-delay: 10 is a Drupal 7 line, and the standard never defined it

Twenty of the 86 files carry a Crawl-delay line and 16 of those read exactly Crawl-delay: 10. That is not a number 16 site owners arrived at independently. It is the line Drupal 7 core shipped directly beneath User-agent: *, and reading the four branches confirms where it went: it is present on 7.x and absent from 9.5.x, 10.5.x and 11.x. Drupal stopped shipping it, and 16 sites kept it.

Seven of those 16 are the interesting ones. They carry the /core/ asset block, which only the Drupal 8 and later file has, and the Crawl-delay: 10 line, which only the Drupal 7 file had. A file in that state was not left alone; it was merged, by somebody who took the new paths and kept the old line, probably without knowing it was a Drupal 7 artefact rather than a setting. That group includes justice.gov, london.gov.uk, gatech.edu and redhat.com.

The line does nothing on any of them. Crawl-delay appears nowhere in RFC 9309: the Robots Exclusion Protocol, published in September 2022, defines allow, disallow and user-agent, and there is not one occurrence of the string crawl-delay in the specification's text. Google's robots.txt documentation, carrying a last updated date of 31 August 2026, is blunter, listing the fields it supports and adding that other fields such as crawl-delay are not supported. OpenAI's crawler documentation, which documents OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User, does not mention crawl rate or crawl-delay anywhere on the page, and neither does Anthropic's crawler article.

None of that makes the line harmful. It is unread rather than dangerous, and a site owner who believes it is throttling anything has one wrong belief and no worse consequence. It matters because of what it stands for. A robots.txt gives its author no feedback of any kind, which is why a line can sit in a file for a decade doing nothing while the person who inherited it assumes it works. The same silence is what lets a group named for the wrong token match nothing and what lets fourteen Google agents walk past a wildcard group that their owner believed covered everything. Per token evaluation is the only thing that answers it, which is why we built a robots.txt tester that reports one verdict per crawler rather than a summary of the file.

The seven sites that named an AI crawler, and what each one chose

Seven files out of 86 addressed an AI crawler by name, and no two of them made the same decision. Counted by token, GPTBot leads with four appearances, then ClaudeBot, Meta-ExternalAgent and Bytespider on three each, anthropic-ai and Amazonbot on two, and OAI-SearchBot, Google-Extended, Applebot-Extended and CCBot on one apiece. PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User and ChatGPT-User were named by nobody. That ordering broadly tracks what we found across the wider corpus, where GPTBot appeared in 82 of 718 files and its search twin in 24.

Three of the seven wrote a single line about a single bot. fema.gov blocks Meta-ExternalAgent and nothing else. broadinstitute.org blocks ClaudeBot and nothing else. aha.org blocks Amazonbot and nothing else. Each of those files is otherwise the stock Drupal file with a long tail of path rules, and each names one company out of the nine this scanner tracks, which reads less like a policy than like a response to one crawler somebody noticed in a log.

unesco.org is the most complete file in the set and the only one that names nine tokens. Its comment header carries a Content Signals preamble, dated by the file itself as last updated 2026-08-24, setting out the search, ai-input, ai-train and use vocabulary in full before any rule appears. Below it, nine groups disallow the whole site to GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot. Evaluated per token, all nine come back blocked at the root. Six of our 15 are untouched by it, among them OAI-SearchBot and both Perplexity tokens.

uct.ac.za is the file worth reading twice, because it draws the one distinction most of this market claims to care about. GPTBot gets its own group with Disallow: /, as do Bytespider and several non AI crawlers. OAI-SearchBot gets a group too, and that group is not a block: it carries the same handful of path rules the site gives SemrushBot, so OAI-SearchBot is free at the root. A university has refused the training crawler and admitted the one that feeds answers with citations, which is exactly the position most site owners describe when asked what they want from AI visibility, and it is a position you can only express by naming tokens. rivm.nl, the Dutch public health institute, does the same thing more bluntly, naming five and blocking all five, and dukehealth.org blocks GPTBot alone.

The uneven part is where those seven sit. Three of the 18 healthcare hostnames named a token, against two of 22 in government and two of 33 in education. On numbers this small that is an observation rather than a trend, and it would take a much larger Drupal sample to say anything firmer. What it does suggest is that the decision travels by sector rather than by platform, which is what you would expect if it is driven by whoever is answering legal questions about content rather than by whoever maintains the site.

What this does not measure, including a gap in our own scanner

Start with the gap on this side of the fence, because it is the reason this post took a detour. Lantad's platform detector, in core/src/platform.ts, holds 15 signals: Next.js, Nuxt, Gatsby, SvelteKit, Remix, Astro, Shopify, WordPress, Wix, Squarespace, Webflow, Framer, Hugo, Jekyll and a generic single page app test. Drupal is not one of them. Every Drupal site in this corpus has been scanned by this product as a site on no recognised platform, which means the per stack advice a report can offer was never offered to any of them. The detection used above was written for this measurement and lives outside the scanner. That is a finding about our software, not about anybody else's, and it is the sort of thing that only surfaces when you go looking for a population nobody asked about.

Now the limits on the measurement itself. This is one request per hostname, on one day, from one network location, with redirects followed and no retry. A file that changed an hour later is recorded as it was when we asked, and a robots.txt edit does not take effect the moment it is saved for a crawler that has cached the previous copy. Nothing here observed an AI crawler. We read no access log and we hold no traffic sample, so every sentence above is about what a file says and none of them is about what any crawler did with it.

The two are not the same question and the gap between them is measurable. Across the wider corpus on 13 September 2026 we found 79 of 115 hostnames refusing a crawler their own robots.txt allowed, which is the edge and the origin overruling the file in the other direction. A permission in robots.txt is a statement of intent by the site owner, and there are at least two layers underneath it that can contradict that statement without telling anyone.

There is also a definitional limit worth stating plainly. "Named no AI crawler token" is not the same as "allows every AI crawler", even though on 78 of these 86 files the two coincided. A file can block a crawler without naming it, as helsinki.fi does, and a file can name a crawler and then grant it more than the wildcard group grants, because under the standard a named group replaces the wildcard rather than adding to it. Both cases are in this sample. The only way to tell which one you have is to evaluate per token at the specific path you care about, which is what what a crawler actually receives from your page and our scoring method are built around.

Finally, the sample. Eighty six hostnames identified as Drupal out of 701 that answered HTTP 200 is not a random sample of Drupal installations, it is the Drupal sites inside one editorially chosen corpus of large organisations, weighted heavily toward the public sector by construction. It says what those 86 files hold. It does not say what share of the world's Drupal sites name a crawler, and anybody quoting it as though it does will be wrong. Our own crawler's behaviour is documented for the same reason we publish the corpus counts: a measurement you cannot inspect is a claim.