BlogFindings
Blocked by robots.txt: 559 of the 581 pages GPTBot lost were closed by a rule that never named it
Lantad requested the home page and the robots.txt of 1,027 hostnames on 17 September 2026 and evaluated every same host link on those home pages against the scanner's robots parser. On the 563 sites where GPTBot could fetch the home page, 581 of the 58,813 linked paths were disallowed to it. 559 of those 581 were decided by the wildcard group, and only 22, spread over nine sites, by a group that names GPTBot at all.
The method is narrow on purpose. One GET of https://hostname/robots.txt and one GET of https://hostname/ per entry in this repository's industry corpus, both sent as LantadBot/1.0 with redirects followed and no browser involved. Every same host link in the home page HTML was collected, deduplicated by path and query string, and evaluated against the file with the same parser a scan uses. Of the 678 hostnames that answered 200 to both requests, 53 served a home page whose raw bytes held no same host link and six served a robots.txt of zero bytes, which leaves 619 analysed here. That parser follows RFC 9309, and the rule that matters most here is its group selection rule: the robots exclusion protocol, published by the IETF in September 2022, says a crawler obeys the group matching its product token and falls back to the group headed with a star only when no group names it. So a site that never names GPTBot hands GPTBot whatever it wrote for everyone.
In short
- Blocked by robots.txt is rarer than the argument about AI crawlers suggests: of 58,813 same host links measured across 563 hostnames on 17 September 2026, 581 were disallowed to GPTBot, and 450 of the 563 sites disallowed none of them.
- 559 of those 581 blocks were decided by the wildcard group, the one headed User-agent colon star, which was written before any AI crawler existed and names none of them.
- Only 22 blocked paths, on nine hostnames, were decided by a group that names GPTBot, and every one of those nine groups was written to welcome AI crawlers rather than to exclude them.
- On 535 of the 544 hostnames that allow all 15 registry tokens at the site root, all 15 tokens lose exactly the same set of linked pages, because the rule doing the work is shared.
- Ecommerce is the exception: 28 of 50 ecommerce hostnames disallowed at least one linked page against 8 of 68 in government, and 197 of the 311 ecommerce blocks were query string URLs from faceted navigation.
| Stage | Hostnames | What it means |
|---|---|---|
| In the industry corpus | 1,027 | The sampling frame, unchanged since 3 August 2026 |
| Answered 200 on /robots.txt | 744 | 160 answered 403 and 42 answered 503 |
| Also served an HTML home page | 678 | 703 home pages were 200 and HTML in total |
| Carried a link and a non empty robots.txt | 619 | 53 home pages carried no link in the raw bytes, 6 files were zero bytes |
| Allowed GPTBot at the site root | 563 | 56 disallowed it outright |
| Of those, disallowed no linked page | 450 | The crawler lost nothing it could see |
What does blocked by robots.txt actually cost an AI crawler?
Across the 563 hostnames that let GPTBot fetch the home page, those home pages carried 58,813 distinct same host paths between them, a median of 74 per site and a mean of 104, which is the same first hop view we used when counting how many pages a site links against how many it declares. Evaluating each of those paths against its own site's file, 581 came back disallowed. That is just under one per cent of everything the home pages point at, and it is concentrated: 113 sites accounted for all 581, and the other 450 disallowed nothing at all.
The distribution inside those 113 matters more than the headline. Eighty three of them lost under five per cent of their linked paths, 27 lost between five and twenty per cent, one lost between twenty and fifty, and two lost half or more. The two at the top are not what a reader would guess. clevelandclinic.org served a home page with exactly one same host link in the raw HTML, /search, and its file disallows /search, which is a 100 per cent loss of a single link rather than a site closing itself. reliancedigital.in lost 26 of 34, and 18 of those 26 fell to one rule, a disallow on any URL carrying an internal_source query parameter.
This is a narrower question than whether a crawler is allowed on the site, which we measured separately across 145 files and which almost always resolves at the root. It is also narrower than whether the server will answer the crawler at all, where 79 of 115 hostnames refused a request their own robots.txt permits. Between those two sits the question that decides most of a site's AI visibility in practice: once a crawler is in, how much of what the site links to can it actually take? On this corpus, on this day, the answer is almost all of it.
Two things are worth saying plainly about that number before anyone relaxes. The 58,813 paths are what the home page links, not what the site contains, so this is a measurement of the first hop and nothing deeper. And a path that robots.txt permits can still be refused by the server, or answered with a page that says nothing. Permission is the cheapest of the gates, and it is the only one measured here.
Flow: 1,027 corpus hostnames (283 did not) to 744 served robots.txt 200; 744 served robots.txt 200 (125 dropped) to 619 also linked a page; 619 also linked a page (56 blocked at /) to 563 allow GPTBot at /; 563 allow GPTBot at / to 58,232 paths allowed; 563 allow GPTBot at / to 581 paths disallowed.
559 of the 581 blocks came from a group that names no crawler
For every disallowed path the parser reports which group decided it, so the 581 can be attributed rather than guessed at. 559 of them were decided by the group headed User-agent colon star. Twenty two were decided by a group naming GPTBot. There is no third category, because a group either names the token or it does not.
At the level of sites the split is the same shape. Of the 113 hostnames that blocked at least one linked page, 104 did it entirely through the wildcard group and nine through a group that names GPTBot. Those 104 sites did not make a decision about AI. They wrote a robots.txt for search engines, in most cases years ago, and RFC 9309 hands that file to every crawler that arrives without a group of its own. It is the same mechanism we found running Drupal's install base, where 79 of 86 sites were governed by a file the CMS shipped and nobody edited, and the same reason the wildcard group is what most AI crawlers are actually reading.
The practical consequence is that the rules an AI crawler obeys on most sites were chosen against a different threat model. A disallow on /search exists because search result pages are thin, duplicated and infinite, which is a crawl budget argument for a search engine index. Whether the same page is worth withholding from a model answering a question is a different question that nobody asked, because the file predates it.
That is also why counting which tokens appear in robots.txt files measures intent rather than effect. A token in a file is a decision somebody made. A wildcard rule reaching an AI crawler is a decision nobody made, and on this corpus the second outnumbers the first by more than twenty five to one at the level of individual pages.
What kind of page is behind the 581 closed doors
The 581 blocks were produced by 173 distinct rule patterns, which is itself the finding: this is not one industry convention repeated, it is hundreds of individual housekeeping decisions. The single most common pattern is a plain disallow on /search, which accounted for 44 blocked paths, followed by a marks and spencer rule on sale listing paths at 29 and a rule on any URL carrying a sortBy parameter at 28.
Sorting the 581 by the shape of the rule that caught them is more useful than sorting by the rule text. 176 were caught by a pattern containing a question mark, which is to say a rule written about query strings rather than about pages. 188 were caught by a pattern using a star or a bracket wildcard without a question mark. The remaining 217 fell to a plain prefix such as /search or /cart, with no wildcard at all. A separate and cruder cut, counting patterns whose text contains one of search, cart, checkout, account, login, user, profile, register, wishlist, order or auth, puts 141 of the 581 in the category of search pages, carts, accounts, logins and registration flows.
Read together those cuts say the same thing. What robots.txt takes away from an AI crawler on this corpus is mostly machinery: the search box, the basket, the account area, and the near infinite surface that faceted navigation generates. It is the same material a search engine was never meant to index either. Only a minority of the 581 look like editorial content, and the clearest examples are language variants and archive listings rather than articles.
There is a real cost hidden in the query string half, and it falls on sites whose products live behind parameters. If the only path to a product is a filtered listing URL, a rule written to stop a search engine crawling ten thousand permutations of that listing also stops a model reaching the ten products that matter. We saw the opposite failure on Shopify, where 38 stores served 1,112 products as open JSON that nobody had thought about at all.
| Pattern | Paths blocked | Shape |
|---|---|---|
| /search | 44 | Plain prefix |
| /*/offers/sale/*/* | 29 | Wildcard path |
| /*sortBy= | 28 | Query string |
| /website/templates*?screen= | 20 | Query string |
| /*facetValueFilter | 19 | Wildcard path |
| /*?gag= | 19 | Query string |
| /*?internal_source=* | 18 | Query string |
| *?freeText=* | 13 | Query string |
| /search? | 12 | Query string |
| /*?fl= | 12 | Query string |
Why ecommerce is the exception at 28 of 50 sites
The corpus is filed into eight editorial categories, and the rate at which a site disallows one of its own linked pages to GPTBot varies more than fourfold between them. Government is the lowest at 8 of 68 hostnames, followed by education at 12 of 88 and travel at 10 of 60. Ecommerce is the outlier at 28 of 50, which is 56 per cent against a corpus rate of 20 per cent.
The explanation is in the rule shapes rather than in any hostility to AI. Ecommerce home pages on this corpus carried 11,176 same host links across the sites where GPTBot is allowed at the root, and 311 of those were disallowed. 197 of the 311 carried a query string. A catalogue with facets generates a combinatorial number of URLs, every one of them a real page that returns 200, and the standard defence has been a robots.txt rule against the parameter. That rule was correct when the only crawler that mattered indexed pages for a ranked list. It is applied unchanged to a crawler whose job is to answer a question about a product.
None of this shows a lost citation, and this measurement cannot show one. What it shows is that the population most likely to have parameter rules is also the population whose content most often sits behind a parameter, which is a structural collision rather than a policy. A merchant who wants to know what survives it can compare the two readings directly with the robots.txt tester against a product listing URL rather than the home page, because the home page is the one URL this collision never touches.
The news category is the smallest here at 25 usable hostnames and should be read with that in mind. Five of the 25 blocked at least one linked page. That is a rate of 20 per cent on a base too small to carry a decimal point, and it is reported because leaving it out would be selective rather than because it supports anything.
On 535 of 544 sites all 15 AI crawlers lost exactly the same pages
This scanner evaluates 15 product tokens: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot. On 544 of the 619 usable hostnames every one of those 15 is allowed at the site root. On 535 of those 544, all 15 lose exactly the same set of linked paths, character for character. Nine sites differentiate.
That near total agreement is the wildcard finding restated from the other end. If the rule doing the work names nobody, then every crawler that fails to match a group gets the identical answer, and the elaborate differences between vendor purposes collapse to one outcome. OpenAI's own documentation separates GPTBot for model training from OAI-SearchBot for search, and states of ChatGPT-User that because the action is initiated by a user, robots.txt rules may not apply. That distinction is real in the vendor's policy and invisible in 535 of these 544 files.
Where sites do differentiate is at the root, not in the paths, and there the pattern is the one already visible in the corpus: training tokens are refused and search tokens are let through. GPTBot is disallowed at the root on 56 of the 619, CCBot on 52, Bytespider on 49 and ClaudeBot on 48, against OAI-SearchBot on 20, Perplexity-User on 19, and Claude-SearchBot and Claude-User on 17 each. The same split at larger scale is why GPTBot is the most blocked AI crawler on this corpus, and it is a genuine decision, made at the root, by people who typed the names.
Ten hostnames disallow all 15 tokens at the root: amsterdam.nl, wa.gov, helsinki.fi, scielo.org, dr.dk, news24.com, nbcnews.com, lapresse.ca, nzherald.co.nz and rappler.com. Six of the ten are news organisations. None of the ten needed to name a single AI crawler to do it, because a site wide disallow in the wildcard group covers every crawler that has ever existed and every one that has not been invented yet.
| Token | Blocked at / | Purpose in the registry |
|---|---|---|
| GPTBot | 56 | Training |
| CCBot | 52 | Training |
| Bytespider | 49 | Training |
| ClaudeBot | 48 | Training |
| anthropic-ai | 42 | Training |
| Google-Extended | 41 | Training |
| ChatGPT-User | 33 | User agent |
| PerplexityBot | 32 | Search |
| OAI-SearchBot | 20 | Search |
| Claude-SearchBot | 17 | Search |
The nine sites whose rules did name GPTBot wrote them to let it in
Twenty two of the 581 blocks came from a group naming GPTBot, on nine hostnames: canonical.com, sap.com, github.com, sonarsource.com, commbank.com.au, westwing.de, bestbuy.com, gap.com and dailymaverick.co.za. Opening all nine files produced the most surprising result in this measurement. Not one of those groups was written to keep AI crawlers out. Every one is an allow list with operational exceptions.
canonical.com heads a group with fourteen AI tokens under a comment reading "Includes: OpenAI, Perplexity, Anthropic, and Apple (crawling and live search)", then allows /blog/, /documentation/, /knowledge/ and ten more content trees, points at llms.txt and a markdown format parameter, and disallows author pages, tag pages, archives, /search, /navigation and /contact-us under a heading that says "Block Noise (Preserve context window for technical content)". Those last three are exactly the paths our scan recorded as blocked there. The site is curating what a model reads, not refusing it.
commbank.com.au heads a seventeen token group with a comment that states the intent outright: "Explicit ALLOW on the owned authority corpus. If a model is going to answer best home loan in Australia, we want it answering from us." github.com opens its file with a group headed by GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot, gives it a crawl delay of one second and twelve Allow lines where the wildcard group has one, and then repeats that group's 57 Disallow lines byte for byte. Nine lines later it opens a second group for Bytespider alone and gives it a bare Disallow: /, which is the clearest statement of preference anywhere in this corpus. dailymaverick.co.za heads its group "AI training crawlers and control tokens", allows the root, and disallows the WordPress plugin directory, an admin path and an account path.
gap.com is worth separating because it is the one case where the distinction is cosmetic. Its group header lists ten AI tokens and then User-agent colon star, all before any rule, so under RFC 9309 that is one group serving the AI crawlers and everything else identically. The single blocked path there, /my-account/, is not an AI decision either. Nine sites out of 619 wrote AI specific rules, and the rules they wrote were invitations.
canonical.com/robots.txt, lines 54 to 104, read 17 September 2026
- User-agent: GPTBot first of 14 tokens heading one group
- # Nudge toward Markdown endpoints site comment
- Allow: /*?format=md allow
- Allow: /documentation/ allow
- # Block Noise (Preserve context window for technical content) site comment
- Disallow: /navigation one of the 22 named blocks
- Disallow: /search one of the 22 named blocks
- Crawl-delay: 2 not supported by every vendor
What this measurement does not show
This is one request pair per hostname from one network location on one day, and everything in it follows from that. It is a reading of files and markup, not of behaviour. No access log was read, no AI crawler was observed requesting any of these sites, and nothing here reports what a crawler did or what any model answered. A disallowed path is a path a compliant crawler should not fetch, which is a claim about a text file rather than about traffic.
Three specific limits are worth naming. The link set is what the raw HTML carried, so a home page that builds its navigation in JavaScript contributes fewer paths, which is why 57 of the 703 HTML home pages we read carried no same host link at all and why the median of 74 links per site is a floor rather than an estimate. We have measured that gap directly elsewhere, and on the pages where we did, rendering added no new crawl paths. Second, 283 hostnames never returned a robots.txt at 200, most of them with a 403, so the sites most aggressive about bot traffic are systematically missing from the denominator and the true block rate is probably understated. Third, this measures the first hop only: pages two clicks deep are not in the 58,813.
What the measurement does support is narrow and, we think, useful. On this corpus, being blocked by robots.txt is uncommon, it is concentrated in machinery rather than content, and it is overwhelmingly an inherited rule rather than a choice. If you want to know which of your own pages sit behind one, the file is the cheapest thing on your site to check, and our reading of what GPTBot gets starts there. Two adjacent checks are worth running at the same time, because they fail independently: whether noindex is doing what you think for AI crawlers, and whether your sitemap is reachable under your own rules.
Our own file is at lantad.co/bot, the scoring and the fetch rules behind every figure above are written up in the methodology, and the registry of tokens we evaluate is listed on the AI crawlers page. We would rather publish the unexciting version of this finding than the alarming one. The alarming version, that robots.txt is quietly starving answer engines of the web, is not what 581 blocked paths out of 58,813 look like.
-
Rules as servedMeasured 744 robots.txt responses read at 200 on 17 September 2026 -
Paths a crawler may fetchMeasured 58,813 home page links evaluated with the scanner's RFC 9309 parser -
What a crawler actually fetchedNot measured No access log was read and no AI crawler was observed -
Pages beyond the first hopNot measured Only same host links present in the home page HTML were evaluated -
Links added by JavaScriptNot measured Raw bytes only, which is why 57 of 703 HTML home pages carried no link -
Effect on any citationNot measured No model was asked anything in this run
Lantad
Published .
A site owner who reads that an AI crawler was blocked by robots.txt usually pictures a decision: somebody sat down, typed a crawler's name, and shut it out. On 17 September 2026 we went looking for that decision on 1,027 hostnames and found almost none of it. What we found instead was a file written for search engines years ago, quietly deciding what an AI crawler can read today, and doing it through a group that names nobody.
Common questions
What does blocked by robots.txt mean for an AI crawler?
It means a Disallow rule in the site's robots.txt matches the URL for that crawler's product token, so a compliant crawler should not fetch it. It is not a lock and it is not enforcement: the server still answers anyone who ignores the file. On the 563 hostnames measured here on 17 September 2026 where GPTBot could fetch the home page, 581 of 58,813 linked paths were disallowed to it.
Does a robots.txt rule have to name GPTBot to block it?
No, and usually it does not. RFC 9309 says a crawler obeys the group headed with a star when no group names its product token, so a file written years ago for search engines governs every AI crawler that arrives. 559 of the 581 blocks measured on 17 September 2026 came from that wildcard group.
Which pages do robots.txt rules most often take away from AI crawlers?
Site machinery rather than articles. The most common single pattern in this measurement was a plain disallow on /search, accounting for 44 blocked paths, and 176 of the 581 blocks came from rules written about query strings. 141 of the 581 were matched by a rule whose pattern text contains one of search, cart, checkout, account, login, user, profile, register, wishlist, order or auth.
Do different AI crawlers get different answers from the same robots.txt?
Rarely at the page level and more often at the root. On 535 of the 544 hostnames that allow all 15 tokens in this scanner's registry at the site root, every one of those 15 lost exactly the same set of linked pages. At the root the picture differs: GPTBot was disallowed on 56 hostnames against 20 for OAI-SearchBot.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
GPTBot is named in 4.5 percent of robots.txt files, the wildcard in 77
The 2025 Web Almanac, written from HTTP Archive's July 2025 crawl of 16,213,084 websites and published on 15 January 2026, reports the catch-all user agent in 77.04 percent of desktop robots.txt files and gptbot in 4.5 percent. Google's advertising crawler is named more than twice as often as any AI crawler.
Most blocked AI crawler: GPTBot in 82 of 718 robots.txt files, and its search twin in 24
Lantad requested /robots.txt once from each of 1,027 hostnames across eight industry sectors on 12 September 2026, and 718 of the responses parsed as robots.txt. GPTBot was disallowed at the site root by 82 of them, seven more than the fourth placed token. The wider gap is inside each vendor: 59 of the 718 refused GPTBot while leaving OAI-SearchBot allowed.
Allow AI crawlers in robots.txt: 89 of 145 files ruled only on the whole site
Lantad requested /robots.txt once from each of 1,027 hostnames on 13 September 2026. 706 responses parsed into at least one group, and 145 of those name at least one of the 15 AI crawler tokens this scanner evaluates. On 89 of the 145, every rule in those groups governs the whole site. Fifty six write at least one path rule, and on 29 of the 145 deleting the AI rules entirely would have changed no verdict we tested.
Crawler user agent: 1,004 robots.txt files named 2,209 tokens, and 120 name one Anthropic no longer documents
A user-agent line in robots.txt is an exact product token, and a token that matches nothing is a rule that does nothing. Lantad requested /robots.txt once from each of 1,419 hostnames on 17 September 2026. 1,004 files parsed into at least one group and named 2,209 distinct tokens between them, 1,098 of which appear in a single file, 813 of which contain a character RFC 9309 does not let a crawler put in its own name, and 120 of which name anthropic-ai or Claude-Web, neither of which appears on Anthropic's crawler page.
