Robots.txt sitemap: 839 of 1,016 files declare one, and a site-wide block covers it on 92 of 93

What does the robots.txt sitemap line actually do?

It is not part of the protocol. RFC 9309, the Standards Track document that defines the Robots Exclusion Protocol, gives the sitemap line one short subsection titled Other Records, and the wording is deliberately loose: crawlers MAY interpret other records that are not part of the robots.txt protocol, for example Sitemaps, and crawlers MAY be lenient when interpreting other records. Everything the specification requires of a compliant crawler concerns user-agent, allow and disallow. Sitemap is a convention that the protocol agrees not to break.

The second half of that subsection is the sentence this post turns on. Parsing of other records MUST NOT interfere with the parsing of explicitly defined records, and the specification gives its own example: a Sitemaps record MUST NOT terminate a group. So a Sitemap line written in the middle of a block of Disallow rules does not end that block, and it does not join it either. It is simply not a member of any group, which means no user-agent line can narrow it, widen it, or address it at all.

Google says the same thing in the operational register. Google's robots.txt documentation, carrying Last updated 2026-08-31 UTC, lists exactly four supported fields, being user-agent, allow, disallow and sitemap, and then states that the sitemap field is not tied to any specific user agent and may be followed by all crawlers, provided it is not disallowed for crawling. The same page requires the value to be a fully qualified URL, including the protocol and host.

That final clause, provided it is not disallowed for crawling, is the whole reason this had to be measured rather than reasoned out. The line is unscoped, but the address it names is an ordinary URL on an ordinary host, and an ordinary Disallow rule can match it like any other path. Whether a refusal reaches the map therefore depends on what the refusal says, not on what the sitemap line says, and that is a property of each individual file. If you want to see the per token verdicts for your own file before reading further, the robots.txt tester runs the same parser these figures came from.

How many robots.txt files declare a sitemap?

Most of them. Of the 1,419 hostnames asked, 29 failed at the network layer, 289 answered with a status other than 200, and 38 answered 200 with a content type that was not text/plain, which RFC 9309 requires. That leaves 1,063 usable responses, of which 1,016 parsed into at least one user-agent group. 839 of those 1,016 declare at least one sitemap, which is 82.6 percent.

The count of lines is larger than the count of files by a wide margin: 3,090 Sitemap lines across the 839. The distribution is heavily skewed. 621 files declare exactly one, which is the shape a content management system writes by default. 51 declare ten or more. The largest is booking.com, whose 39,491 byte file carries 434 Sitemap lines, beginning with an airport index. ucl.ac.uk declares 206 and airbaltic.com 110. At that size the robots.txt file has stopped being an access control document and become a directory, which is a reasonable thing for it to be and worth noticing, because a directory is the opposite of a restriction.

This matters for anything downstream that reads the file. A sitemap is the only place most sites state, in one machine readable document, which URLs they consider canonical and current. It is also routinely overread: a sitemap tells a crawler where pages are and not that they changed, and an earlier measurement found that the links on a home page and the URLs in a published sitemap disagree by three orders of magnitude on the same site. For AI visibility the practical consequence is narrower and more useful: if your pages are not in the map you publish, the cheapest discovery path any crawler has does not lead to them.

Does blocking an AI crawler also block the sitemap?

On this sample, almost always, and not because anybody planned it. 108 of the 1,016 files refuse at least one of the 15 AI crawler tokens this scanner evaluates at the site root, and 93 of those 108 also declare a sitemap. Those 93 are the files where the question bites. For each of them we took every crawler the file refuses at the root, took every sitemap address the file declares on the same host, and asked the parser whether that crawler may fetch that address.

On 92 of the 93, the answer was no for every combination. The reason is unglamorous. A site that refuses a crawler almost always does it with Disallow: / under a named token, and a sitemap URL is a path on the same host like any other, so the refusal swallows it. An earlier count found that 89 of 145 files ruled only on the whole site rather than on any narrower path, and that blunt habit is exactly what closes this gap. The protection is real and it is a side effect.

The single exception is instructive because it shows what the mechanism does when a file is more careful. washingtonpost.com refuses eight of the 15 tokens at the root, including ClaudeBot, PerplexityBot, CCBot, Bytespider, Amazonbot, anthropic-ai, Applebot-Extended and Meta-ExternalAgent, and each of those groups carries Disallow: / followed by Allow: /creativegroup/. The file declares seven sitemaps, and the seventh sits at /creativegroup/sitemaps/sitemap.xml. Seven of those eight refused tokens may fetch it, by the file's own rules, read on 17 September 2026. Nothing here says any of them did, and we did not ask them: the crawler registry records what each vendor documents, and OpenAI's crawler documentation is the kind of source that settles what a token means, not what a fetch did. This is a reading of one file, not an observation of traffic.

The second gap is structural rather than accidental. 12 of the 93 declare at least one sitemap whose authority is not the authority that served the robots.txt file, and a robots.txt file governs its own host only. notion.com names sitemaps.notion.com, figma.com names help.figma.com, rivm.nl names rivm.openrepository.com, and cambridge.org names www.cambridge.org, which the protocol treats as a separate service with its own file. In each of those cases the refusal written on one host has no authority over the address it advertises on another, and the second host's own robots.txt decides. That is not a mistake, but it is a thing to check rather than assume, and it is the same class of surprise as robots.txt redirects crossing hosts at the fetch layer.

Why the sitemap line looks scoped when it is not

Position is the reason this is confusing, and position is measurable. Of the 839 files declaring a sitemap, 766 put their first Sitemap line after their first user-agent line. Only 73 put it before any group opens, which is the only placement that looks the way the field actually behaves. So on more than nine in ten of these files the reader's eye sees a Sitemap line sitting inside a block of rules, indented by nothing but convention, and infers a scope that the parser never applies.

nature.com is a clean illustration and a short file, 2,721 bytes over 98 lines. Its four Sitemap lines are at lines 2, 3, 4 and 5, before any group. Sixty three lines further down, at line 68, comes User-Agent: GPTBot, and at line 69, Disallow: /. Read top to bottom, that file publishes its maps and then closes the door, and the two acts have no relationship in the protocol at all. The door is closed because Disallow: / matches /sitemap.xml, not because the Sitemap lines came earlier.

The same indifference to layout is what makes robots.txt errors quiet. RFC 9309 requires a crawler to pick one group by matching its own product token, then to obey that group and no other, and it never reads whitespace, ordering or intent. A file can be beautifully arranged and wrong, which is how a single mistyped character can turn a whole Disallow group off with nothing reporting the difference. Anyone treating robots.txt as a lever in generative engine optimization work should be reading it the way a parser does rather than the way a document reads.

There is one more consequence worth stating plainly, because it cuts the other way. RFC 9309's security considerations note that listing paths in robots.txt exposes them publicly and makes them discoverable. A Sitemap line is the strongest version of that: it is not a hint about one path, it is an invitation to the index. A file that refuses every AI crawler and publishes a sitemap has still published the sitemap to everything that reads the file, including clients that ignore the rules entirely.

Do the sitemaps a robots.txt declares actually answer?

Publishing an address is not the same as serving one, so we asked. The first sitemap declared by each of the 839 files was requested once on 17 September 2026, as LantadBot, with redirects followed and the same twenty second timeout. 793 returned HTTP 200. 46 did not, and the breakdown is the interesting part: 23 returned HTTP 403, nine returned 404, four returned 503, two returned 202, one returned 401, one returned 429, and six did not complete, of which three could not be requested at all because the declared value was a path rather than an absolute URL and three timed out.

The 23 refusals are the finding here. A 403 on a sitemap address means a site is refusing, to an identified scanner, the one document it published in robots.txt specifically so that automated clients would fetch it. amsterdam.nl, imf.org, oecd.org, science.org, jhu.edu and futurelearn.com are among them. We cannot say from outside whether that is a bot management rule, a geography rule or a rate limit, and we did not retry, so treat it as one observation of one request rather than a verdict on those sites. What it is not is an edge case: 23 of 839 is 2.7 percent, and the whole value of the sitemap convention rests on the address being fetchable by exactly the kind of client that was refused.

Status is not content either. Of the 793 that answered 200, 439 opened with a sitemapindex element and 338 with a urlset, which together is 777 responses that are recognisably a sitemap. Nine returned a gzip or octet-stream body that this pass did not decompress, and those are very probably valid compressed sitemaps that we simply did not open. Four returned HTML, among them daytona.io, walmart.com, regions.com and seattle.gov. Three returned XML that was not a sitemap at all: an RSS feed at eluniversal.com.mx, an FAQ document at visitdubai.com, and an HTML preloader fragment served with a text/xml content type at magicalkenya.com.

That last group is the familiar failure of any convention where the status code and the payload are allowed to disagree, and it is the same reason a 404 on robots.txt allows every crawler while a 503 blocks them all matters more than it sounds. A crawler asking for a map and receiving a styled error page gets HTTP 200 and no URLs, and nothing in that exchange reports a problem to anybody. Checking it is one request, and it is the same discipline as fetching your own page under a crawler user agent rather than trusting that the server treats every client alike.

What to check in your own robots.txt

Four checks come straight out of the numbers above, and all four are one HTTP request each. None of them needs a vendor, a dashboard or an account, which is deliberate: the point of publishing these figures is that anyone can reproduce the method on their own host, and the methodology page states what this scanner does and does not do before it does it.

First, fetch your sitemap address as a client rather than in a browser tab. 23 of 839 sites refused it to an identified scanner and 20 more returned 404, 503 or a body that was not a sitemap. A browser is the one client least likely to reproduce that, because bot management rules are written to let browsers through. Second, confirm the value is a fully qualified URL. Three files in this sample wrote a bare path, which Google's documentation does not accept, and the cost of that mistake is total rather than partial.

Third, check which host the address names. 31 files in the full sample point at a different authority once www and the apex are treated as one site, and 12 of the 93 refusing files point somewhere their own rules cannot reach. If your sitemap lives on a subdomain or a storage bucket, the rules that govern it are written in that host's robots.txt and not in yours. Fourth, decide whether the mismatch is intentional. Publishing a map to a crawler you have refused is a coherent position if the map is meant for search engines and the refusal is aimed at training, and the two are separable: our research page keeps the measured work in one place, our own bot page states what LantadBot does with what it reads, and the platform guides such as how to get cited by ChatGPT treat discovery and permission as different problems because they are.

What this post does not show is worth naming too. No crawler was observed. No access log was read. Every hostname was asked once, from one network location, on one day, and the corpus is a stratified convenience sample from this repository's two committed seed files rather than a random sample of the web, so it supports statements about these 1,016 files and nothing wider. The parser is this repository's own implementation of RFC 9309, and where a real crawler's implementation differs, the real crawler is right and we are describing a document rather than predicting behaviour. That distinction is the same one that makes llms.txt worth measuring rather than assuming, and it is the only honest way to read a file that nobody is obliged to obey.