BlogFindings
Sitemap blocked by robots.txt: 2,022 of 467,786 advertised URLs were closed to Googlebot, and none of them on a news site
A sitemap is a list of URLs a site wants fetched and robots.txt is a list a crawler may not fetch, and nothing asks the two files to agree. Lantad read 474,786 URLs out of 2,216 sitemap files on 12 November 2026 and tested each one against the robots.txt of the host it sits on. 2,022 URLs on 53 of 756 sites were disallowed to Googlebot by their own site, 703 sites contradicted themselves nowhere, and the sites that refuse AI crawlers on purpose were not the sites that did this by accident.
So this run went looking for the disagreement, one layer deeper than an earlier measurement of the sitemap line itself, which asked whether the map's own address survives a block and not what the map contains. Every request below was sent as LantadBot/1.0, the agent described on the bot page, with HTTP redirects followed and a twenty second timeout. The corpus is the stratified sampling frame this repository commits for its crawlability study: 392 hostnames across ten platform strata and 1,027 across eight industry strata, 1,419 in total.
In short
- Lantad read 474,786 URLs from 2,216 sitemap files on 12 November 2026 and evaluated 467,786 of them against the robots.txt of the host each URL actually sits on: 2,022 URLs across 53 of 756 sites were disallowed to Googlebot by their own site's rules.
- 703 of those 756 sites advertised nothing their own robots.txt refused, so a sitemap blocked by robots.txt is a minority condition on 12 November 2026 rather than the normal state of the web.
- Five sites held 1,568 of the 2,022 blocked URLs on 12 November 2026: canva.com at 923 under a Disallow of /ar/, texas.gov at 202, wa.gov at 171, bbva.com at 149 and santafe.edu at 123.
- GPTBot was refused 46,756 of the same URLs across 109 sites on 12 November 2026, but 44,995 of those came from the 60 sites that refuse GPTBot at the site root, which is a deliberate block and not a contradiction.
- News sites were 38 of the 60 deliberate GPTBot refusals and 0 of the 53 sites that blocked their own advertised URLs to Googlebot, measured 12 November 2026.
What robots.txt says
- User-agent: *
- Disallow: /ar/
- applies to every crawler
- no exception for Googlebot
What the sitemap says
- 923 URLs beginning /ar/
- inside the first 1,000 read
- canva.com/ar/, /ar/apps/, /ar/help/
- offered to every crawler
What does a sitemap blocked by robots.txt actually mean?
It means a crawler was handed an address and a rule forbidding that address, and the rule wins. Nothing in a sitemap grants access. RFC 9309, the robots exclusion protocol, decides the outcome on its own terms: its section 2.2.2 says the most specific match found must be used and that the most specific match is the match that has the most octets. A sitemap entry is not a match of any kind, so it never enters that comparison. It is a suggestion about crawling made in a file that has no say over crawling.
Which robots.txt applies is the part that is easy to get wrong, and getting it wrong is how a measurement like this produces a number that is too large. RFC 9309 places the rules at /robots.txt in the top-level path of the service and writes the address as scheme, authority and path, so the file governs one authority and not a brand. An apex and its www are two authorities and can serve two different files, which this corpus has already shown on 73 of 277 subdomains. A first pass at this measurement read the apex file and judged www URLs against it, which charged helsinki.fi with a thousand self-blocked URLs it had not committed: the apex of that site answers with a site-wide Disallow and www.helsinki.fi answers with an ordinary permissive file. Every figure below instead fetches robots.txt for the exact host each URL sits on, which took 1,966 robots.txt files to cover 1,419 sites.
The rest of the method is ordinary. Of the 1,419 hosts, 1,108 answered robots.txt with a 2xx status and 1,100 of those a plain 200; 223 answered a 4xx, which RFC 9309 reads as imposing no restrictions, and 57 answered a 5xx, which the same specification reads as a complete disallow, a split this blog has measured as the sharpest asymmetry in the standard; 31 failed at the transport. 30 of the 2xx responses were HTML documents rather than rules files and 1,028 parsed into at least one user-agent group. 878 of those declared at least one sitemap, across 3,171 Sitemap lines.
Flow: Sitemap lists a URL (no standing) to Most specific match wins; robots.txt for that host (rules only) to Most specific match wins; Most specific match wins (no rule matches) to Crawler may fetch; Most specific match wins (Disallow matches) to Crawler must not fetch.
How many sites advertise URLs their own robots.txt refuses?
Opening 2,314 sitemap addresses returned 2,216 files and 98 that did not deliver, which is the same order of failure a dedicated run found when it measured how many declared sitemaps answer at all. Those files held 474,786 URLs. 115,163 sat on the same host as the robots.txt that declared them and 352,807 on another host of the same site, almost all of them the www twin. 6,808 belonged to another site entirely and 184 on three sites had no usable robots.txt for their own host, leaving 467,786 URLs evaluated across 756 sites.
2,022 of those URLs were disallowed to Googlebot by the robots.txt of the host they sit on, spread across 53 sites. The other 703 sites contradicted themselves nowhere, which is the first thing worth saying plainly: this is a real defect with a small blast radius, not a condition of the web. Of the 53, nineteen blocked exactly one URL and nineteen blocked ten or more. Only two blocked anything with a site-wide Disallow of the root, and the other 51 did it with a path rule, which is why the total stays in the low thousands rather than the hundreds of thousands.
The distribution across the corpus strata is the part that was not predictable. Finance was the worst affected at 12 of 77 sites, followed by ecommerce at 8 of 51 and software as a service at 7 of 102, while news was 0 of 59. Documentation sites carried the highest rate on a small base at 3 of 15. Three sites did it through a rule in a group naming Googlebot specifically rather than through the wildcard: embl.org, decathlon.fr and bancolombia.com, so on those three the refusal is not a side effect of a wildcard but a line that names Google. On the wildcard side the rule often predates the map, and a Disallow in a wildcard group reaches every crawler that has no group of its own, so one line closes the whole field at once.
| Corpus stratum | Sites measured | Sites blocking their own URLs | Share |
|---|---|---|---|
| finance | 77 | 12 | 15.6 percent |
| ecommerce | 51 | 8 | 15.7 percent |
| software as a service | 102 | 7 | 6.9 percent |
| healthcare | 51 | 6 | 11.8 percent |
| travel | 51 | 5 | 9.8 percent |
| government | 39 | 5 | 12.8 percent |
| documentation | 15 | 3 | 20.0 percent |
| education | 42 | 3 | 7.1 percent |
| Shopify storefronts | 33 | 2 | 6.1 percent |
| Wix and Squarespace | 54 | 1 | 1.9 percent |
| SaaS marketing | 26 | 1 | 3.8 percent |
| news | 59 | 0 | 0 percent |
The five sites that blocked a hundred or more of their own URLs
1,568 of the 2,022 URLs came from five sites, and each one is a different way for the two files to end up disagreeing. All five rules below were re-read directly at their source on 12 November 2026 after the sweep, and all five were unchanged.
canva.com is the largest and the cleanest to explain. Its robots.txt carries a wildcard group disallowing a list of locale paths, /ar/ among them alongside /zh_cn/, /ar_ae/ and several more, while its sitemap advertises 923 URLs beginning /ar/ inside the first thousand read. texas.gov disallows /government-services-directory/*/ and /texas-state-agencies-departments/*/ to every crawler and then advertises 202 of its 303 URLs under exactly those two prefixes, in English and Spanish, so two thirds of what the state publishes to crawlers is closed to them. wa.gov is the blunt case: its file is a wildcard group with a Disallow of the root, two named exemptions for a site auditing bot and a content management bot, and a Sitemap line, so all 171 of the URLs it advertises are refused to everything that is not one of those two tools. bbva.com disallows four suffix patterns including */todays-number and */todays-news and advertises 149 pages matching them. santafe.edu disallows *.pdf and */videos/play, then lists 30 PDFs and 80 video pages.
The pattern underneath the five is not carelessness about crawling, it is two artefacts maintained on different clocks. A rule gets added to stop a crawler reaching something awkward, a staging path, a redirect handler, a locale nobody finished. The sitemap is generated from the content management system and knows nothing about the rule. Below the top five the same shapes repeat at smaller scale, and they read as a list of things meant to stay out of an index: 86 URLs under /external-redirects/ on rcgp.org.uk, 38 under /ppc/ on fasthosts.co.uk, 28 under /archive/ on nsw.gov.au, 25 under /Test/ on uchicago.edu including pages named Nobel-Prize-Test-Page and TEST-Academics, 16 under /authors/ on scalekit.com, 9 account pages on publix.com and three under /admin/ on langhamhotels.com. A sitemap that offers a crawler an admin path and a set of test pages suggests a generator reading the content tree without consulting the rules file, which is also how a sitemap ends up disagreeing with the server about when a page last changed.
| Site | URLs blocked | URLs read | Rule that closed them | Group holding the rule |
|---|---|---|---|---|
| canva.com | 923 | 1,000 | Disallow: /ar/ | wildcard |
| texas.gov | 202 | 303 | two directory prefixes, English and Spanish | wildcard |
| wa.gov | 171 | 171 | Disallow: / | wildcard |
| bbva.com | 149 | 1,000 | four suffix patterns | wildcard |
| santafe.edu | 123 | 874 | file type rules, largest *.pdf and */videos/play | wildcard |
GPTBot was refused 46,756 of them, and most of that was deliberate
Running the same 467,786 URLs against the same files for AI crawler tokens returns a number twenty three times the Googlebot one, and reporting it without the split would be the dishonest version of this post. GPTBot was disallowed 46,756 of the URLs on 109 sites and ClaudeBot 41,369 on 103. But 44,995 of GPTBot's total comes from the 60 sites that refuse it at the site root, where the refusal is the entire point and the sitemap is aimed at Google. That is a decision, not a defect, and it is the finding the earlier run on the sitemap line already reached from the other direction.
The two populations barely overlap, and the way they differ is the most interesting thing this run produced. News sites were 38 of those 60 deliberate GPTBot refusals and 0 of the 53 accidental Googlebot blocks. The sites with editors, which this corpus has repeatedly found to be the ones that block AI crawlers on purpose, do it cleanly: one rule at the root, no contradiction, the map still serving the search engine they want. The sites that blocked their own advertised URLs were banks, shops and government portals, where the rule that did the blocking names no AI crawler and is closing pages the generator keeps offering. 59 sites blocked sitemap URLs to GPTBot and not to Googlebot, and 50 blocked some to both.
Strip the root refusals out and 49 sites were left refusing GPTBot URLs by a narrower rule, and one of them is worth naming because it is the honest middle case. openstax.org writes a group for GPTBot alone and disallows /books/ inside it, then advertises 653 /books/ URLs in its sitemap. The textbooks are the thing the rule protects, the block is specific rather than a blanket refusal, and the sitemap still hands GPTBot a complete index of the thing it has just been refused. Nothing in either file is wrong on its own, which is the condition this whole post is about, and it is a different question from whether the two crawlers get different answers out of the same file.
Sixteen robots.txt files advertised somebody else's sitemap
6,808 of the URLs read belonged to a site other than the one whose robots.txt declared them, across 16 sites, and the reason this matters is that no rule on the declaring site governs them. The sitemap protocol at sitemaps.org is explicit that the location of a sitemap file determines the set of URLs that can be included in it, that all URLs listed must use the same protocol, and that they must sit on the same host as the sitemap. Google's own guide to building a sitemap, carrying a last updated date of 8 July 2026, puts the operational half the same way: for each individual sitemap file, include only URLs from that particular site, and a sitemap affects only descendants of the parent directory unless it is submitted through Search Console.
Two of the 16 are plainly domain migrations left unfinished in the rules file. greggs.co.uk declares a sitemap at www.greggs.com, a different registrable domain, and every one of the 1,000 URLs read from it is a greggs.com address. workweave.dev declares one at weaveos.com, which is the same company under a later name. In both cases the old host is still answering, still publishing a robots.txt, and still pointing crawlers at a map whose contents it has no authority over, so the rules a visitor would reasonably expect to apply are not the rules that apply. Two more sites put the map itself off-host rather than the URLs: santafe.edu declares its sitemap on an Amazon S3 bucket, and embl.org declares five sitemaps over plain HTTP from an HTTPS site, which is the same scheme mismatch a run on absolute and relative internal links found pinned into page markup.
seoul.go.kr is the case that only the per-host method can see at all. Its sitemap advertises URLs across three of its own subdomains, www, event and org, and each of those three serves its own robots.txt with a site-wide Disallow in the wildcard group. Seven URLs, three different files, one brand. Judged against the apex file it would have looked clean, and judged against each host it is three separate refusals, which is the whole argument for fetching the rules per authority rather than per company.
-
greggs.co.ukMap on another domain robots.txt declares a sitemap at www.greggs.com, and all 1,000 URLs read were greggs.com addresses. -
workweave.devMap on another domain Declares a sitemap at weaveos.com, the same company under a later name, holding 1,000 URLs it does not govern. -
santafe.eduMap on object storage Declares its sitemap from an Amazon S3 bucket rather than its own host, while its own rules disallow PDFs the map lists. -
embl.orgMap over plain HTTP Declares five sitemap addresses over http from a site served over https, a scheme the protocol asks to match.
What this did not measure, and what to check on your own site
Four limits, and the first is the one that moves the headline. Every count here is a floor, because the sweep stopped at 1,000 URLs per site, three declared sitemaps per site and three children per sitemap index to two levels of nesting; 364 of the 756 sites hit one of those caps, and 28 of the 53 Googlebot-blocked sites were among them, so their individual totals are partial too. Second, no crawler was observed. This reads sites rather than server logs, so nothing here says whether Googlebot, GPTBot or anything else requested these URLs, honoured the rule or indexed the page anyway, and the figures describe what the two files instruct rather than what any company did. Third, each file was read once, so a transient rule and a standing one look identical, and the 161 hosts that refused this scanner its robots.txt may serve it to another client. Fourth, the corpus is a stratified sampling frame for crawlability rather than a census, so the stratum shares show direction and not rates.
One thing is deliberately not a recommendation: deleting rules. A Disallow that catches a sitemap URL is not evidence the rule is wrong. On most of the 53 the rule is the considered artefact and the sitemap is the one generated without thinking, and the fix is to stop advertising the path rather than to open it. Which of the two files a site meant is not observable from outside, and this measurement does not guess.
The check itself is short and needs nothing installed. Fetch your own robots.txt for the exact host your sitemap URLs use, not the apex if the map lists www, and read the groups rather than assuming the wildcard is the only one that applies; the robots.txt tester resolves a path per crawler token and will show which group won and why, and what GPTBot sees shows what comes back for a single URL. Then take twenty addresses out of your own sitemap, including some from any locale, archive, print or test tree, and resolve each one. A rule carrying a wildcard or a suffix is where the surprises were in this corpus, and resolution is genuinely non-obvious there: an Allow line changed no verdict on 343 of 716 files, a trailing star changed nothing in 168 comparisons, and a raw non-ASCII path bound nothing once encoded. If the two files disagree, pick the one you meant. An inventory of pages nobody is allowed to fetch does not improve AI visibility, and it is not what the scoring method reads either.
resolve sitemap URL against its own host
- sitemap declares www.uchicago.edu/Test/Nobel-Prize-Test-Page listed
- fetch https://www.uchicago.edu/robots.txt 200
- select group for Googlebot wildcard
- match path against rules Disallow: /Test/
- verdict disallowed
Lantad
Published .
Two files on a site answer the same question and are allowed to disagree. A sitemap is a list of addresses the owner wants fetched. A robots.txt is a list of addresses a crawler is told not to fetch. They are written at different times, usually by different systems, and neither the robots exclusion standard nor the sitemap protocol contains a line asking them to match. Google's own robots.txt documentation says the plain part out loud: the sitemap field is not tied to any specific user agent and may be followed by all crawlers, provided it is not disallowed for crawling. The invitation and the refusal can sit in the same file, four lines apart.
Common questions
What does it mean when a sitemap is blocked by robots.txt?
It means the site published the URL as one it wants fetched and separately published a rule forbidding that fetch, and the rule decides. A sitemap entry grants no access: RFC 9309 resolves a request by matching robots.txt rules against the path and taking the most specific match, and a sitemap line is never one of those rules. The crawler is left holding an address it has been told not to request.
Does a sitemap override robots.txt?
No. Nothing in a sitemap can open a path that robots.txt closes, and Google's robots.txt documentation says the sitemap field may be followed by all crawlers only provided it is not itself disallowed. Lantad measured 2,022 URLs on 53 sites on 12 November 2026 where the two files disagreed, and in every case the rule is what a compliant crawler obeys.
How many sites block URLs their own sitemap advertises?
53 of the 756 sites that yielded an evaluable URL on 12 November 2026, covering 2,022 of 467,786 URLs. 703 sites contradicted themselves nowhere. Nineteen of the 53 blocked exactly one URL and five blocked a hundred or more, so the condition is both uncommon and heavily concentrated.
Can a robots.txt declare a sitemap on a different domain?
It can, and 16 sites in this corpus did on 12 November 2026, covering 6,808 URLs. The sitemap protocol asks that a sitemap's URLs share its host and scheme, and Google's guide to building one asks that each file contain only URLs from that particular site. The practical consequence is that the declaring site's rules do not govern those URLs at all, so checking them against the wrong robots.txt gives the wrong answer.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
Robots.txt sitemap: 839 of 1,016 files declare one, and a site-wide block covers it on 92 of 93
A Sitemap line is not scoped to any user-agent group, which suggests a site could refuse an AI crawler and still hand it a full URL inventory in the same file. Lantad requested /robots.txt once from each of 1,419 hostnames on 17 September 2026 and then fetched the first sitemap each file declared. 1,016 files parsed into at least one group, 839 declared a sitemap, 108 refuse at least one of the 15 AI crawler tokens this scanner evaluates, and on 92 of the 93 refusing files that also publish a map, the refusal already covers the map.
Sitemap validator: 44 of 871 declared sitemaps delivered no file, and none of the rest broke the 50,000 URL limit
Lantad requested /robots.txt from all 1,419 hostnames in this repository's two committed corpus seed files on 1 October 2026, then fetched the first sitemap each file declared. 1,104 answered HTTP 200 and 871 of those declared a sitemap. 827 of the declared sitemaps delivered a body, 38 answered a non-200 status, one timed out, and five named a value that is not an absolute URL.
Blocked by robots.txt: 559 of the 581 pages GPTBot lost were closed by a rule that never named it
Lantad requested the home page and the robots.txt of 1,027 hostnames on 17 September 2026 and evaluated every same host link on those home pages against the scanner's robots parser. On the 563 sites where GPTBot could fetch the home page, 581 of the 58,813 linked paths were disallowed to it. 559 of those 581 were decided by the wildcard group, and only 22, spread over nine sites, by a group that names GPTBot at all.
