BlogFindings
Robots.txt subdomains: 73 of 277 gave an AI crawler a different answer from the apex
A robots.txt file governs one host and nothing else, so the rule you wrote on example.com does not reach blog.example.com. Lantad resolved six content subdomain names under each of 1,027 hostnames on 18 September 2026 and found 277 pairs where a subdomain and its apex each served a robots.txt of its own. 73 of those pairs disagreed about at least one of the 15 AI crawler tokens this scanner evaluates, and 262 of the 277 named no AI crawler at all.
The specification half is settled and we quote it below. The deployed half is what we measured. On 18 September 2026 we took the 1,027 hostnames in this repository's committed industry corpus, resolved six content subdomain names under each one, blog, docs, shop, support, help and news, and requested /robots.txt from every name that resolved, as LantadBot/1.0 with redirects followed, a twenty second timeout and no browser involved. 717 apex files came back usable. 1,377 of the 6,162 subdomain names resolved in DNS and 386 of those served a robots.txt. 49 of the 386 were reached by a redirect that landed on another host, so they are not independent files, and a further 60 sit under a host whose own apex file never parsed and so have nothing to be compared against. That leaves 277 pairs, across 214 hostnames, where both sides of the comparison exist. Every access verdict below is computed with this repository's own parser, the same one behind our robots.txt tester, evaluated for the path / against each of the 15 AI crawler tokens the scanner knows. Nothing here reports what any crawler did, because no access log was read.
In short
- Robots.txt subdomains are scoped separately from the apex: Lantad compared 277 subdomain robots.txt files against their own apex file on 18 September 2026 and 73 of them, across 67 hostnames, answered at least one of the 15 AI crawler tokens differently.
- 262 of the 277 subdomain files named no AI crawler token at all, against 64 apex files in the same set that named at least one. The policy is written at the apex and does not travel.
- On 40 of the 73 divergent pairs the apex refused a crawler the subdomain allowed, and on 34 the subdomain was the stricter of the two. One host, support.sap.com, ran both ways at once across four tokens.
- cnn.com names 14 of the 15 tokens and disallows GPTBot at the root, while help.cnn.com serves User-agent colon star with Allow colon slash. forbes.com, nature.com and unesco.org show the same shape on their help, shop and support hosts.
- Google's robots.txt specification, carrying Last updated 2026-08-31 UTC, states that a robots.txt on a subdomain is only valid for that subdomain, and that a 4xx response other than 429 is read as no crawl restrictions at all.
| Step | Count | What it leaves |
|---|---|---|
| Hostnames in the industry corpus | 1,027 | The sampling frame, unchanged since 3 August 2026 |
| Apex /robots.txt usable | 717 | 155 answered 403, 51 answered 503, 34 sent HTML under a 200 |
| Hosts excluded by the wildcard DNS control | 63 | Their subdomains prove nothing about separate administration |
| Subdomain names that resolved | 1,377 | Of 6,162 probed, so most content subdomains do not exist |
| Subdomain /robots.txt that answered | 386 | 222 answered 503, 143 sent HTML, 93 answered 404 |
| Of those, redirected to another host | 49 | Borrowing the apex file, which RFC 9309 permits |
| Own file, under an apex that parsed | 277 | The comparable set, across 214 hostnames |
| Disagreed with the apex on an AI token | 73 | 26.4 percent, across 67 hostnames |
Does robots.txt apply to subdomains?
No. A robots.txt file is scoped to one host and carries no authority over any other, which means the rule protecting example.com says nothing whatsoever about blog.example.com. This is not a grey area or an implementation quirk, and it is worth reading the two primary sources rather than taking anyone's summary of them.
RFC 9309, the Robots Exclusion Protocol, published by the IETF in September 2022 on the standards track, defines the location in section 2.3: the rules MUST be accessible in a file named /robots.txt in the top-level path of the service, and, as per RFC 3986, the URI of the robots.txt file is scheme colon, optional double slash authority, slash robots.txt. The authority is the host. The RFC's own two examples are https://www.example.com/robots.txt and ftp://ftp.example.com/robots.txt, which are different authorities and therefore different files. One honest caveat: the word subdomain does not appear anywhere in RFC 9309. The scoping is carried entirely by that word authority, and a reader who does not already know that RFC 3986 folds the host into the authority can finish the document without noticing the consequence.
Google's robots.txt specification, which carries Last updated 2026-08-31 UTC, spells out what the RFC leaves implicit, and its table of worked examples is the clearest statement of the rule anywhere. Of a file at https://example.com/robots.txt it says: this is the general case, it's not valid for other subdomains, protocols, or port numbers, and it's valid for all files in all subdirectories on the same host, protocol, and port number. It lists https://other.example.com/ as explicitly not covered. Of a file at https://www.example.com/robots.txt it says, verbatim, that a robots.txt on a subdomain is only valid for that subdomain, and then rules out https://example.com/, https://shop.www.example.com/ and https://www.shop.example.com/ in the same breath. The page also notes that crawlers don't check for robots.txt files in subdirectories, so moving the file is not an option either.
The one place the boundary bends is a redirect, and it bends in a direction that helps. RFC 9309 section 2.3.1.2 says crawlers SHOULD follow at least five consecutive redirects, even across authorities, and that a file reached that way MUST be fetched, parsed, and its rules followed in the context of the initial authority. A subdomain can therefore borrow the apex file deliberately by redirecting to it. 49 of the 386 subdomain files we fetched did exactly that and we counted them separately, because a borrowed file is agreement rather than a coincidence. This is the same mechanism we measured when 122 of 391 robots.txt requests answered with a cross-host redirect, and it is the only supported way to make one file speak for two hosts.
| File location | Valid for | Not valid for |
|---|---|---|
| https://example.com/robots.txt | example.com and all its subdirectories | other.example.com, http://, port 8181 |
| https://www.example.com/robots.txt | www.example.com only | example.com, shop.www.example.com |
| https://example.com/folder/robots.txt | Nothing: not a valid robots.txt file | Crawlers do not look in subdirectories |
| https://example.com:8181/robots.txt | Content served on port 8181 | example.com on the default port |
| A 4xx response other than 429 | Read as no crawl restrictions | Absence of a file is not a refusal |
What 277 subdomain files answered
73 of the 277 disagreed with their own apex about at least one of the 15 tokens, which is 26.4 percent, and they are spread across 67 of the 214 hostnames that had a comparable subdomain at all. That headline number is the least interesting thing in the run, though, because the shape underneath it is lopsided in a way that says what is actually happening.
24 of the 73 disagreed on all 15 tokens at once. That is a complete reversal rather than a difference of detail: one host admits every AI crawler in the registry and the other refuses all of them. At the other end, 13 of the 73 differed on exactly one token. The middle is thin, and the reason is mechanical. Most of these files contain no AI crawler rules whatever, so the verdict for all 15 tokens is decided by whatever the wildcard group says, and when two wildcard groups differ they differ for everything simultaneously.
Per token, the counts track how widely each name has been adopted rather than anything about the crawlers themselves. Bytespider diverged on 55 pairs, CCBot on 53 and GPTBot on 52, while Perplexity-User diverged on 30, Claude-User on 32 and Claude-SearchBot on 32. The older and more widely copied a token is, the more likely it is to appear in an apex file and be absent from a subdomain file, which is the same adoption gradient we found when 1,004 robots.txt files named 2,209 distinct tokens between them. The registry of tokens the scanner evaluates is published on our AI crawler list.
By subdomain name, support led with 18 divergent of 60 comparable, then help with 17 of 56 and shop with 16 of 49. The three that most often hold a company's actual prose came last: news diverged on 9 of 39, docs on 8 of 40 and blog on only 5 of 33. By sector, news sites were worst in proportion at 13 of 21, education next at 13 of 47, and SaaS the best at 16 of 103 despite contributing the most pairs. A blog subdomain is usually run by the same team on the same platform as the main site. A help centre usually is not, and that distinction turns out to predict the result better than the industry does. It is the same split that shows up in how many files declare a sitemap: the parts of a site that marketing owns behave differently from the parts a vendor operates.
The AI policy is written at the apex and does not travel
262 of the 277 subdomain files named no AI crawler token at all. Fifteen named one or more. In the same set, 64 apex files named at least one, and on 58 of those 64 pairs the apex named a token and the subdomain named none. Only nine pairs ran the other way, with the subdomain carrying an AI rule its apex lacked. Whatever effort has gone into AI crawler policy over the last two years has gone into one file per organisation, and that file sits at the front door.
On 40 of those 58, the difference in naming produced a difference in outcome. cnn.com names 14 of the 15 tokens and gives GPTBot a Disallow of slash. help.cnn.com, a separate host, answers with a User-agent colon star group and Allow colon slash, so GPTBot is admitted. forbes.com names 12 tokens and refuses GPTBot at the root, while help.forbes.com names none and has no rule that touches it. nature.com names 11 and refuses GPTBot, while both shop.nature.com and support.nature.com admit it. unesco.org names nine and refuses GPTBot, and its shop and support hosts admit it. cambridge.org is the narrowest version of the same thing: it names exactly one token, ChatGPT-User, and disallows it, and help.cambridge.org does not name it, so on that host ChatGPT-User is allowed.
The pattern in those names is not carelessness so much as ownership. A help centre or a storefront is very often a hosted product with a robots.txt the vendor wrote, and the person who added a careful fourteen-token block to the main site had no file to edit on the other host. We are describing what the files say, not what anyone intended, and we cannot see intent from the outside. Some of these splits will be deliberate. What can be said without guessing is that an organisation that believes it has one AI crawler policy has, on this evidence, as many as it has hosts.
The consequence depends entirely on which way the site wanted it. If the intent was to refuse training crawlers, the most blocked of which is GPTBot at 82 of 718 files, then the refusal has a hole in it wherever a subdomain answers differently. If the intent was to be read and cited, and a support host quietly refuses everything, then the pages most likely to answer a real question are the ones missing. Either way the failure is silent, and it is the same silence as a rule that never named the crawler it was aimed at, which is how 559 of 581 pages GPTBot lost were closed by a wildcard. Our guidance for anyone working through this deliberately is on the page about getting cited by ChatGPT, and the standing caution that a user agent string is a claim rather than an identity applies to every token named above.
| Apex | Tokens named at apex | Subdomain | Tokens named there | Verdict differs on |
|---|---|---|---|---|
| cnn.com | 14 | help.cnn.com | 0 | 14 tokens |
| forbes.com | 12 | help.forbes.com | 0 | 9 tokens |
| nature.com | 11 | shop.nature.com | 0 | 11 tokens |
| nature.com | 11 | support.nature.com | 0 | 11 tokens |
| unesco.org | 9 | shop.unesco.org | 0 | 9 tokens |
| cambridge.org | 1 | help.cambridge.org | 0 | ChatGPT-User |
34 subdomains were stricter than the apex, not looser
The direction most people expect is a leaky block, and it is the more common one here: 40 of the 73 divergent pairs had an apex refusing a crawler its subdomain admitted. But 34 ran the other way, and those are the ones worth a site owner's attention, because nobody goes looking for content they are accidentally hiding.
news.cam.ac.uk serves a User-agent colon star group with Disallow colon slash, and then two explicit exceptions, LinkedInBot and facebookexternalhit, each with an empty Disallow. The apex at cam.ac.uk has no such restriction. So the University of Cambridge's news office is readable by two social preview bots and closed to all 15 AI crawlers, while the university's main site is open to everything. That file reads like a press-office convention from before AI crawlers existed, which is exactly how a rule written for one problem ends up deciding a different one.
docs.temu.com is the sharpest case in the set. Its apex names nine AI tokens and allows GPTBot, and the docs host answers every crawler with a wildcard Disallow of slash, which flips 14 of the 15 tokens because the apex had already refused Bytespider. The documentation is the part of that domain a question is most likely to have a real answer in, and it is the part that is closed. news.ovhcloud.com, shop.sage.com, blog.bayzat.com, news.fattureincloud.it and news.bergfreunde.de all show the same shape: a permissive apex and a content subdomain that refuses everyone through the wildcard group. github.com is the mildest, where docs, support and help all diverge from the apex on exactly one token, Bytespider, and agree on the other 14.
One host managed both directions at once. support.sap.com is stricter than sap.com on OAI-SearchBot, Claude-SearchBot and Claude-User, and looser on anthropic-ai, which is what happens when two files name overlapping but not identical sets of tokens and each fills the gaps with a different wildcard rule. It is the only pair of the 73 that does this, and it is a good illustration of why reading a single file tells you less than people assume. This is the same class of problem as a site that refuses a crawler its robots.txt allows, and whether the pages behind any of these hosts would have been worth citing is a separate question that prose parity measures and this run did not. The broader question of whether refusing training costs anything in search is one we looked at separately when 52 of 54 sites refusing Google-Extended still allowed Googlebot.
-
Agreed on all 15 tokens204 pairs The subdomain answers every AI crawler exactly as the apex does -
Apex refused, subdomain allowed40 pairs A block written at the front door with a hole in it -
Subdomain refused, apex allowed34 pairs Content closed without anyone deciding to close it -
Both directions at once1 pair support.sap.com, across four tokens -
Differed on all 15 at once24 pairs A wildcard group disagreeing, not an AI rule -
Differed on exactly one token13 pairs Usually one name present in one file and absent in the other
What the controls threw out, and one run we discarded
Two controls decide whether any of the above means anything, and both removed data rather than adding it.
The first is wildcard DNS. A host that resolves every name under it, including one nobody has ever registered, is not telling you that it runs a separate docs property. So every hostname in the corpus was also asked to resolve a nonsense subdomain, lantad-scope-control-9f3a, and the 63 hosts that resolved it were excluded from the comparison entirely along with all their subdomains. That is six percent of the frame removed for being unable to answer the question. 1password.com is one of them, and it is a good example of why the control is not optional: its apex and its docs host really do carry different rules, but with a wildcard in place we cannot distinguish a deliberately separate property from a catch-all that answers anything.
The second is the redirect rule quoted earlier. 49 of the 386 subdomain files that answered were reached by a redirect that landed on a different host, usually the apex or its www form. Those are not independent files and counting them as agreement would have inflated the agreeing side, so they sit outside the 277. Separately, 93 subdomains answered 404 for robots.txt. Under Google's specification a 4xx other than 429 is treated as if a valid robots.txt file didn't exist, which means Google assumes that there are no crawl restrictions, so those hosts are wide open regardless of what their apex says. They are not in the 277 either, because there is no file to compare, and their exclusion makes the divergence figure conservative rather than generous. The difference between a missing file and a refusing one is the subject of why a 404 and a 503 are opposites.
The run itself needs one disclosure. The first full pass at this measurement issued every probe over HTTP at high concurrency, exhausted the outbound connection pool of the machine it ran on, and returned transport errors for 904 of the 1,027 hostnames. Those errors are indistinguishable in shape from a site refusing a connection, and had they been believed the post would have reported a corpus that mostly refuses to answer, which is false. The pass was thrown away and the measurement rebuilt to resolve names in DNS first and issue HTTP only for names that exist, at a twelfth of the concurrency. The figures here are from that second pass, where 717 apex files parsed, a number in line with the 700 to 744 this corpus has returned on other recent runs. Our standing method notes are on the methodology page and what our crawler sends is documented on the bot page.
Flow: 1,027 corpus hosts (200 and not HTML) to 717 apex files parsed; 1,027 corpus hosts (control resolved) to 63 excluded: wildcard DNS; 717 apex files parsed (six names each) to 1,377 subdomains resolved; 1,377 subdomains resolved (200 and not HTML) to 386 served robots.txt; 386 served robots.txt (landed elsewhere) to 49 redirected off-host; 386 served robots.txt (own file) to 277 comparable pairs; 277 comparable pairs (verdict differs) to 73 disagreed.
What to check on your own hosts
The check is short and it does not need a tool. List the hosts that serve your content, which is not the same as listing your sites: the apex, the www form if it answers separately, and every subdomain carrying documentation, help, news, a blog, a store or a developer portal. Request /robots.txt from each one and compare the files rather than assuming they are the same file. Most of the divergence in this run would have been visible in thirty seconds of looking, and none of it was visible from the apex alone.
Three specific things are worth looking for, in this order. First, a subdomain returning 404 for robots.txt, which is 93 of the hosts we probed and means no restrictions at all, whatever the apex says. Second, a subdomain whose file exists but names no AI crawler, which is 262 of the 277 here: its verdict for every token is decided by the wildcard group, so read that group carefully because it is doing all the work. Third, a subdomain on a vendor platform, because the file will be the vendor's default and changing it may not be something the dashboard exposes. If it is not editable, a redirect to the apex file is the supported alternative, and the RFC language quoted above is what makes it work.
Then decide what you actually want, which is the part no measurement settles. If the goal is to be read and cited, the useful question is whether the pages behind these hosts say anything a crawler can extract, and what GPTBot sees answers that for one page at a time. If the goal is to refuse, the rule has to be repeated on every host, because there is no inheritance to rely on and the specification is explicit that there is not. Our broader survey work is collected on the crawlability study, and the question of how many files write rules for the whole site rather than a path is covered in 89 of 145 files ruled only on the whole site.
One last thing this run did not do, stated plainly so nobody has to infer it. We did not fetch a single page behind any of these hosts, we did not read an access log, and we have no evidence about whether any AI crawler ever requested any of these files or honoured what it found. We measured what 277 files say when a parser reads them on one day from one network location. What a crawler does with that is a separate claim and we are not making it here, which is the same boundary we hold in the guidance on getting cited by Claude.
- List every host, not every site The apex, www, and each of docs, help, support, shop, news and blog that answers
- Look for a 404 on /robots.txt 93 subdomains here. Read as no crawl restrictions, whatever the apex says
- Check whether the file names any AI token 262 of 277 named none, so the wildcard group decides all 15 verdicts
- Read the wildcard group on each host 24 pairs differed on all 15 tokens purely because their star groups differed
- Identify vendor-operated hosts help and support diverged most, at 17 of 56 and 18 of 60
- Redirect rather than duplicate where you cannot edit RFC 9309 section 2.3.1.2 makes a cross-authority redirect binding on the initial authority
Lantad
Published .
Somebody asking this question has usually just written an AI crawler rule and wants to know how far it reaches. They added a Disallow for GPTBot to the file at the root of their domain, and then remembered that the documentation lives on a docs host, the help centre is on a vendor's platform, and the press releases go out from a news host. The question is whether one file covered all of that. It has a precise answer in the specification, and a separate answer in what real sites have actually deployed, and those two are not the same thing.
Common questions
Does robots.txt apply to subdomains?
No. A robots.txt file is valid only for the host it is served from. Google's robots.txt specification, last updated 2026-08-31 UTC, states that a robots.txt on a subdomain is only valid for that subdomain, and lists https://other.example.com/ as not covered by a file at https://example.com/robots.txt. RFC 9309 carries the same rule through the URI it defines for the file, which includes the authority.
How often do a site's subdomain and apex rules actually disagree?
On 73 of 277 comparable pairs, or 26.4 percent, measured by Lantad on 18 September 2026 across 214 hostnames drawn from a 1,027 host corpus. 40 of the 73 had an apex that refused a crawler the subdomain allowed, 34 had the subdomain as the stricter of the two, and one host ran both ways at once across four tokens.
Why do so few subdomains name AI crawlers?
262 of the 277 subdomain files in this run named no AI crawler token at all, against 64 apex files in the same set that named at least one. Help centres, storefronts and support portals are frequently hosted products carrying a vendor's default robots.txt, so the person who wrote a careful block on the main site had no file to edit there.
Can one robots.txt file cover two hosts?
Only by redirect. RFC 9309 section 2.3.1.2 says crawlers should follow at least five consecutive redirects, even across authorities, and that a file reached that way must be fetched, parsed and its rules followed in the context of the initial authority. 49 of the 386 subdomain files Lantad fetched on 18 September 2026 were reached this way. Copying the file to each host is the alternative, and it then has to be maintained in each place.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
Prose Parity
Prose Parity is Lantad's metric for the share of a page's rendered, human-visible text that is present in the raw HTML an AI crawler receives, where 100% means a crawler sees everything a human visitor sees.
Blocked by robots.txt: 559 of the 581 pages GPTBot lost were closed by a rule that never named it
Lantad requested the home page and the robots.txt of 1,027 hostnames on 17 September 2026 and evaluated every same host link on those home pages against the scanner's robots parser. On the 563 sites where GPTBot could fetch the home page, 581 of the 58,813 linked paths were disallowed to it. 559 of those 581 were decided by the wildcard group, and only 22, spread over nine sites, by a group that names GPTBot at all.
Crawler user agent: 1,004 robots.txt files named 2,209 tokens, and 120 name one Anthropic no longer documents
A user-agent line in robots.txt is an exact product token, and a token that matches nothing is a rule that does nothing. Lantad requested /robots.txt once from each of 1,419 hostnames on 17 September 2026. 1,004 files parsed into at least one group and named 2,209 distinct tokens between them, 1,098 of which appear in a single file, 813 of which contain a character RFC 9309 does not let a crawler put in its own name, and 120 of which name anthropic-ai or Claude-Web, neither of which appears on Anthropic's crawler page.
robots.txt redirect: 122 of 391 sites sent one, and all 122 crossed hosts
Lantad requested /robots.txt once from each of 392 hostnames on 10 September 2026 with redirect following switched off. 391 answered, 122 of them with a 3xx, and every one of those 122 Location headers pointed at a hostname other than the one asked. RFC 9309 says to follow such a redirect and apply the file it reaches to the original authority.
