# Check for robots.txt: three of six sites served a different file > Fetched on 2 September 2026 across the 14 hostnames belonging to the six real sites this scanner captured on 15 July 2026, one request per hostname returned eight distinct files. On three of the six sites the second hostname did not serve the file that governs the captured page, and on one of those it returned nothing at all. - Canonical page: https://lantad.co/blog/check-for-robots-txt-fourteen-hostnames-eight-files - This file: https://lantad.co/blog/check-for-robots-txt-fourteen-hostnames-eight-files.md - Last substantive update: 2026-09-02 ## Key facts - **Published:** 2026-09-02 - **Category:** Findings - **Author:** Lantad - **Length:** 3269 words - **Takeaway 1:** Fetched on 2 September 2026, the 14 hostnames belonging to the six real sites this scanner captured on 15 July 2026 returned eight distinct robots.txt files, and on three of those six sites the second hostname did not serve the file that governs the captured page. - **Takeaway 2:** A check for robots.txt covers one authority and nothing else: RFC 9309 section 2.3 places the file at scheme://authority/robots.txt, so example.com, www.example.com and shop.example.com are three separate documents that are free to disagree. - **Takeaway 3:** Requesting gymshark.com/robots.txt on 2 September 2026 returned a 301 to us.checkout.gymshark.com/robots.txt and a 3,656 byte file carrying 2 groups, while www.gymshark.com served its own 3,123 byte file carrying 6 groups and a different Sitemap line. - **Takeaway 4:** Across those eight files, 46 user-agent lines named a specific crawler and 38 distinct names appeared, and none of them was one of the 15 AI crawler tokens this scanner tracks, so every AI token fell to the wildcard group and Lantad's shipped matcher returned allowed at every origin on 2 September 2026. ## Summary A check for robots.txt is usually one request. Type the domain, append /robots.txt, read what comes back, and decide whether the site is open to AI crawlers. That request is well formed and the answer is true, and it is true about less than most people assume. The file is scoped to a single authority, so what you read belongs to the exact hostname you typed and says nothing about the other hostnames the same brand serves pages from. So the question was asked 14 times. On 2 September 2026 every plausible hostname for the six real sites this scanner captured on 15 July 2026 was sent one request for /robots.txt, and every response was parsed with the parser that ships in this repository, the same code behind [the robots.txt tester on this site](https://lantad.co/tools/robots-txt-tester). Eight distinct files came back. Three of the six sites answered with a document other than the one governing the page that was captured, one hostname did not answer at all, and one brand turned out to be running two unrelated files. The second half of the run then asked whether any of that changed what an [AI crawler](https://lantad.co/glossary/ai-crawler) is permitted to fetch, and the answer there was no, which is the more useful of the two findings and the one this post ends on. ## What does a check for robots.txt actually cover? One authority. That is the whole scope, and the standard is explicit about it. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the Robots Exclusion Protocol, published September 2022 on the Standards Track, says in section 2.3 that the rules MUST be accessible in a file named /robots.txt, all lowercase, in the top-level path of the service, and then gives the URI form as scheme, colon, optional double slash and authority, then /robots.txt. The example it prints is https://www.example.com/robots.txt. An authority is not a brand and it is not a registrable domain. Under RFC 3986, which RFC 9309 defers to here, it is the host together with the scheme and the port. That makes example.com and www.example.com two authorities and therefore two files, with no requirement anywhere that they agree, resemble each other, or both exist. Add shop.example.com, docs.example.com and a checkout host operated by a payments vendor and you have five files, five owners in practice, and five separate answers to the question you thought you had asked once. The same section carries a requirement that is easy to read past and matters more than the URI form. Section 2.3.1.2 says crawlers SHOULD follow at least five consecutive redirects, even across authorities, and then adds that if a robots.txt file is reached within five consecutive redirects, the file MUST be fetched, parsed, and its rules followed in the context of the initial authority. A redirect moves where the bytes live. It does not move which hostname those bytes govern. That single sentence explains the strangest row in the run below. Two neighbouring behaviours are already covered on this site and are worth keeping distinct from this one, because all three get called checking robots.txt. What the server's status code means is a separate question, answered by the rule that [a 404 on robots.txt allows every crawler and a 503 blocks them all](https://lantad.co/blog/robots-txt-404-and-503-are-opposites). What a rule inside the file matches is a third, and the pattern grammar has its own surprises, including that [a trailing asterisk changed nothing in 168 comparisons](https://lantad.co/blog/wildcard-robots-txt-trailing-star-changed-nothing). This post is only about which file you got. Our own crawler's rules are published at [the LantadBot page](https://lantad.co/bot) and live at one authority like everybody else's. ## Fourteen hostnames returned eight distinct files The six sites are the real pages held as golden fixtures in this repository, captured on 15 July 2026: www.allbirds.com, astro.build, www.gymshark.com, developer.mozilla.org, webflow.com and en.wikipedia.org. For each one, the run asked the hostname that actually served the captured page and then the obvious alternative a person would try, which is the apex where the page lives on www and the www form where the page lives on the apex. Wikipedia and Mozilla got a third request, since both are brands whose content lives on a subdomain and whose registrable domain is a separate site. Fourteen requests produced 13 bodies and 8 distinct files by SHA-256. Every body that arrived came back as Internet Media Type text/plain, which is what section 2.3 requires, so no site in the set failed the format requirement. Sizes ranged from 66 bytes at www.mozilla.org to 28,275 bytes at en.wikipedia.org, a spread of more than four hundred to one across six sites that are all real businesses with real traffic. Three of the six were internally consistent and one check would have been enough for them. Allbirds redirects the apex to www and serves the same 5,495 byte file. Webflow does the reverse, redirecting www to the apex for the same 279 byte file. Wikipedia sends wikipedia.org to www.wikipedia.org and www.wikipedia.org on to en.wikipedia.org, and all three requests end at the same 28,275 byte document. In each case the two hostnames are two authorities that happen to agree, which is a property of that site's configuration rather than anything the standard promises. The other three did not agree. Requesting www.astro.build produced no response at all, because no address record resolved for that hostname and the connection was never established; astro.build itself answers normally with a 120 byte file. Requesting mozilla.org lands on www.mozilla.org and a 66 byte file with two Disallow rules, while developer.mozilla.org, where the captured documentation page lives, serves its own 119 byte file with three different Disallow rules and a Sitemap line the other does not have. And Gymshark, covered next, runs two files that have visibly diverged. That pattern of one domain resolving into many is the same one behind the crawl surface figures in the post on [985 links found and 200,712 URLs declared](https://lantad.co/blog/scan-a-website-for-all-pages-985-linked-200712-declared), and it is why [our methodology](https://lantad.co/methodology) fixes the authority before it fixes anything else. The wider habit of leaning on one file is visible in the field too: [the catch-all user agent appears in 77.04 percent of desktop robots.txt files](https://lantad.co/blog/robots-txt-names-the-wildcard-not-the-ai-crawler), so most files are written once, broadly, and rarely revisited. What the site finds across real pages rather than constructed ones is collected in [the crawlability study](https://lantad.co/research/crawlability-study). ## One brand, two robots.txt files, neither aware of the other Gymshark is the clearest case in the set and it is not a misconfiguration. Requesting https://gymshark.com/robots.txt on 2 September 2026 returned a 301 to https://us.checkout.gymshark.com/robots.txt, and following it produced a 3,656 byte file. Requesting https://www.gymshark.com/robots.txt returned 200 and a different 3,123 byte file. The captured storefront page lives on www. The apex redirects to a checkout host on a different subdomain, so a person checking the shortest form of the brand's domain reads a file belonging to the payment flow. Parsed, the two documents are not variations on one policy. The www file carries 6 groups and 108 rules and names six crawlers by token, being adsbot-google, nutch, ahrefsbot, ahrefssiteaudit, mj12bot and pinterest. The checkout file carries 2 groups and 85 rules and names one, adsbot-google. The Sitemap lines point at different hosts, www.gymshark.com/sitemap.xml in one and us.checkout.gymshark.com/sitemap.xml in the other, which means the two files also disagree about where the site's URLs are declared, a distinction that matters given that [a sitemap tells a crawler where pages are and not that they changed](https://lantad.co/blog/sitemap-lastmod-is-an-assertion). Both files are internally reasonable. Neither describes the other's host. This shape is common wherever a hosted platform owns part of the domain. A storefront on one host, a checkout or account subdomain on a vendor's infrastructure, a docs site on a third: each authority ships whatever robots.txt its platform generates, and the site owner often only ever edits one of them. The per-stack version of that problem, including what a platform writes for you and what it does not, is the subject of [the Shopify guide](https://lantad.co/fix/shopify). The same split explains why an edit made in one place can appear to do nothing, though the more common reason for that is timing rather than scope, and [a robots.txt edit does not take effect when you save it](https://lantad.co/blog/when-a-robots-txt-edit-reaches-a-crawler). Webflow's file is worth naming for the opposite reason. Its 279 bytes carry a Content-Signal line, reading ai-train=yes, search=yes, ai-input=yes, placed above five Disallow rules. That is a declaration about use rather than access, and the two do not interact in the way people expect: [the Content-Signal line asks and the Disallow lines under it block](https://lantad.co/blog/content-signal-asks-disallow-blocks). It is on exactly one of the eight files here. To resolve a specific URL on a specific host against whatever file actually governs it, [what GPTBot sees](https://lantad.co/tools/what-gptbot-sees) does the authority selection rather than leaving it to a guess about which hostname to type. ## A redirect moves the file, not the authority it governs Six of the 14 first responses were a 301, so redirects are the normal case rather than the exception, and the standard has a precise answer for what they mean. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) section 2.3.1.2 permits them explicitly, tells crawlers to follow at least five consecutive hops even across authorities, and then fixes the interpretation: a file reached inside that budget MUST have its rules followed in the context of the initial authority. Beyond five hops a crawler MAY treat the file as unavailable. Wikipedia exercises that budget in the open. Asking wikipedia.org returns a 301 to www.wikipedia.org, which returns a second 301 to en.wikipedia.org, which serves the 28,275 byte file. Two consecutive hops across three authorities, comfortably inside the five the specification asks for, and a crawler that had started at wikipedia.org would apply the English Wikipedia rules to wikipedia.org URLs. The scheme is scoped the same way, which is why the run also checked it: all six page hosts answered a plain http request for /robots.txt with a 301 to the https form, and Webflow's redirect target spelled the port out as webflow.com:443, which is the authority written in full. Read back through the Gymshark result, the rule has a consequence that is not obvious. A crawler that asks gymshark.com follows the 301 and applies the checkout host's 85 rules to gymshark.com URLs, because that is what the initial authority clause requires. A crawler that asks www.gymshark.com applies the storefront's 108 rules to www.gymshark.com URLs. Both crawlers are behaving correctly. They are enforcing two different policies on two hostnames that a customer experiences as one shop, and neither file is wrong, because neither was ever asked about the other. Redirects at the file level are a smaller version of a pattern this site keeps meeting, which is that an identifier a person treats as stable resolves differently depending on who resolves it and when. Page level redirects and self-reference produced their own finding in the post where [five of five canonical tags pointed at themselves](https://lantad.co/blog/canonical-tags-and-ai-crawlers-five-of-five-self-referential). The fix in both cases is the same and it is unglamorous: name the exact authority you measured, and do not let one lookup stand in for a set of them. ## Did any of it change what an AI crawler may fetch? No, and that is the part worth publishing. The second half of the run took the 15 AI crawler tokens in this repository's registry, being GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot, and evaluated each one against the captured page's path using the shipped matcher, once with the file from the page's own host and once with the file the second hostname served. Every token came back allowed, at every origin, on every site. Zero verdicts differed. The reason is in the files rather than in the matcher. Across all eight documents, 46 user-agent lines named a specific crawler and 38 distinct names appeared, from mj12bot and ahrefsbot to teleportpro, wget and semrushbot. Not one of them is an AI crawler token. Every AI token in the registry therefore selected the catch-all group, and the catch-all group on these six sites does not disallow the captured page. Wikipedia writes 33 groups and 462 rules, names 33 crawler tokens between them, and still addresses no AI crawler by name. The directory of the agents this scanner does track is at [the AI crawler list](https://lantad.co/tools/ai-crawlers). So the honest summary of the whole run is two-sided. Which hostname you check decides which file you read, and on half these sites it decided which sitemap was declared, how many groups existed and which crawlers were named at all. It did not decide anything about AI access, because none of the files legislated on AI access in the first place. Someone who checked the wrong hostname would have reached the right conclusion about GPTBot by luck, and would have been wrong about everything else in the file. What this run does not establish is worth stating plainly. Six sites are not a sample and no percentage here describes the web. The requests carried no crawler identity, so this measures what those origins serve to an ordinary client and not what they serve to a named bot, a distinction that matters because [robots.txt allowed all 60 AI crawler probes and 11 responses still came back different](https://lantad.co/blog/robots-txt-allowed-sixty-probes-eleven-differed), and because of 592 sites that banned GPTBot in robots.txt, [234 served it a 200 anyway](https://lantad.co/blog/gptbot-bans-served-a-200-anyway). Nor does a permission observed today survive a rename: a group heading that stops matching leaves [stale robots rules behind a renamed crawler token](https://lantad.co/blog/renamed-crawler-tokens-leave-stale-robots-rules), and matching at all depends on the token being one a vendor publishes, which [6 of 15 crawler tokens do for their request User-Agent](https://lantad.co/blog/gptbot-detection-six-of-fifteen-tokens-publish-a-ua). Reading the correct file is the first step of [AI visibility](https://lantad.co/glossary/ai-visibility) and nowhere near the last. ## Questions and answers **Does robots.txt on example.com cover www.example.com?** No. RFC 9309 section 2.3 places the file at scheme://authority/robots.txt, and the authority is the host, scheme and port together, so example.com and www.example.com are two separate files that may differ. They often agree because the apex redirects to www or the reverse, but that is a configuration choice rather than a rule. On 2 September 2026 Allbirds and Webflow agreed across both forms, and Gymshark did not: gymshark.com redirected to a checkout subdomain serving a 3,656 byte file while www.gymshark.com served its own 3,123 byte file. **Do subdomains need their own robots.txt?** Yes, if you want rules to apply to them. Each subdomain is a separate authority and a crawler fetching a URL on shop.example.com will request shop.example.com/robots.txt, not the file at the apex. In the 2 September 2026 run, developer.mozilla.org served a 119 byte file with three Disallow rules and its own Sitemap line while www.mozilla.org served an unrelated 66 byte file with two Disallow rules and no sitemap, which is two policies for one organisation because they are two authorities. **What happens when robots.txt redirects to another hostname?** The crawler follows it and then applies the rules to the hostname it started from. RFC 9309 section 2.3.1.2 says crawlers SHOULD follow at least five consecutive redirects, even across authorities, and that a file reached within five redirects MUST be fetched, parsed, and its rules followed in the context of the initial authority. Beyond five hops a crawler MAY treat the file as unavailable. Asking wikipedia.org on 2 September 2026 took two hops through www.wikipedia.org to reach the file at en.wikipedia.org. **How many hostnames should I test on my own site?** Every authority that serves pages you want read, which is more than one for most sites. Test the apex and the www form, then every subdomain carrying content, including any docs, help, blog, careers or store host, and any subdomain a platform vendor operates on your behalf. Check the scheme too: on 2 September 2026 all six page hosts in this run redirected a plain http request for /robots.txt to the https form, but that redirect is a configuration rather than a guarantee. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot