Robots.txt blocked resources: 78 of 925 home pages disallowed a script or stylesheet they load

What are robots.txt blocked resources?

They are the files a page depends on, refused to a crawler by the page's own robots.txt while the page itself stays open. The sequence is worth setting out exactly, because the order is what makes the failure invisible. A crawler resolves the host, fetches robots.txt, evaluates the page URL against it, fetches the page if the verdict allows, and then, if it renders at all, evaluates every URL the document references and fetches the ones it is still allowed. The second evaluation is the one nobody tests. A site owner checks that the home page is allowed, sees that it is, and stops.

Google documents both halves of this in its own words. The introduction to robots.txt, last updated 2025-12-10 UTC, says you can use a robots.txt file to block resource files such as unimportant image, script, or style files, if you think that pages loaded without these resources will not be significantly affected by the loss, and then adds the condition that matters: if the absence of these resources make the page harder for Google's crawler to understand the page, do not block them, or else Google will not do a good job of analyzing pages that depend on those resources. The permission and the warning are one sentence apart, and the word carrying the whole weight is "unimportant".

The enforcement is stated even more plainly. Google's JavaScript SEO basics, last updated 2026-03-04 UTC, says that if robots.txt marks a URL as disallowed then Googlebot skips making an HTTP request to this URL and skips the URL, and that Google Search will not render JavaScript from blocked files or on blocked pages. There is no partial fetch and no warning in the serving path. The file is simply not there, and the renderer proceeds without it.

The reason this belongs on a site that measures AI crawlers rather than on a classic technical SEO checklist is the path from Google's renderer into an AI answer. Google's page on AI features in Search, last updated 2025-12-10 UTC, states that to be eligible as a supporting link in AI Overviews or AI Mode a page must be indexed and eligible to be shown in Google Search with a snippet, and that robots.txt directives for Googlebot are the control site owners have over how their sites are crawled for Search. The index is the intermediary, so anything that degrades what Googlebot renders degrades the entry to AI Overviews by the same amount and through the same mechanism.

78 of 925 home pages disallowed a script or stylesheet they load

The method first, because the denominators carry the result. Every one of the 1,419 hostnames in the committed corpus received one request for robots.txt and one for the home page, both as LantadBot with redirects followed and no JavaScript executed, which is the same request shape described on our bot page. 32 home page requests failed outright and 1,090 answered HTTP 200 with HTML. Of those 1,090, 997 served something that parsed as a robots.txt file; 48 answered in the 400 to 499 range, which RFC 9309 section 2.3.1.3 says leaves a crawler free to access any resource on the server, 40 answered the request with a web page rather than a text file, and 4 answered 5xx, which the same specification says a crawler must read as complete disallow.

Those 997 are the population, because a site with no enforceable file cannot block anything. 72 of them disallow GPTBot the home page outright, which is its own subject and one we have measured separately: 559 of the 581 pages GPTBot lost were closed by a rule that never named it. Excluding those leaves 925 home pages that GPTBot may fetch, and those are the pages where a blocked resource is a real loss rather than a footnote to a page the crawler was never getting.

Across the 925 the markup named 15,893 script and stylesheet URLs on the page's own host and 7,555 on other hosts. 713 of the 925 pages carry at least one same-host reference; 198 carry only third-party ones. Evaluating each same-host URL, path and query string together, against its own site's file with the shipped evaluator produced 495 disallowed references on 78 distinct hosts: 332 scripts and 163 stylesheets.

The distribution is the useful part. 11 of the 78 lost every same-host script and stylesheet the page references, 13 lost every stylesheet and 12 lost every script. The rest lost a handful out of many, which in practice means an analytics tag or a consent script rather than the layout. One stratum dominates the count and does so for a single reason covered two sections down: 31 of the 78 are direct-to-consumer stores on Shopify, with 11 software-as-a-service sites, 9 other ecommerce sites and 8 financial services sites behind them.

The rules that did it were not written about scripts

Only two of the deciding patterns name a file extension at all. The rest are directory prefixes and query-string catchalls aimed at something else entirely, which is the mechanism this whole finding rests on: nobody in the corpus set out to hide a stylesheet.

The single most common deciding pattern was Disallow: /*?*, which blocked 73 references, and 71 of those 73 are on one site, so read it as one severe case rather than a widespread one. It is a reasonable rule with a clear purpose, which is to keep a crawler out of faceted and parameterised URLs, and it is indifferent to what the parameter is for. nsw.gov.au carries it, and it also carries the Drupal convention of explicitly permitting resources, with lines reading Allow: /core/*.css? and Allow: /profiles/*.js? among others. Every one of the 44 stylesheet and 27 script references on its home page carries a cache-busting query string, served from /sites/default/files/css/ and /libraries/dcs/assets/, which those Allow lines do not cover, so all 71 are refused. The file both anticipated the problem and failed to catch it.

mcgill.ca shows the same collision resolved by pattern length rather than intent. Its file carries sixteen Allow lines of the form Allow: */themes/*.css$ and Allow: */modules/*.js?, and then, further down the same wildcard group, Disallow: /*.css$ and Disallow: /*.js$ inside a long list of file extensions that is mostly archives and images. For a stylesheet at /root/css/, the narrow Allow does not match and the blanket Disallow does, so all four of its same-host stylesheets go. Longest match wins is the documented rule, our own robots.txt validator measurement found 232 of 1,059 files carrying a defect of some kind, and this class of self-contradiction is the one that is hardest to see by reading.

snyk.io is the cleanest case in the corpus and worth quoting because the intent is legible. Its file opens one wildcard group with Allow: / and a Cloudflare Content-Signal line reading ai-train=no, search=yes, ai-input=yes, a mechanism we measured when one of ten Cloudflare endpoints read it. A second wildcard group then lists Disallow: /*_buildManifest.js$, Disallow: /*_middlewareManifest.js$ and Disallow: /*_ssgManifest.js$, which are three specific Next.js build artefacts, and immediately after them Disallow: /*.js$. Three targeted rules and one that generalises to every script on the site, including all 31 chunk URLs its own home page loads from /_next/static/. That stack is the one where this costs most, since the payload is where the content is: we have measured Next.js home pages sending 15 times more payload than prose, and a reader on that framework can check the specifics against our Next.js guide.

The AI-specific reading of all this does not survive contact with the data. 482 of the 495 blocked references were decided by a group whose token is the wildcard, and 13 by a group naming GPTBot, on sonarsource.com, sap.com, otto.de and commbank.com.au. In all 13 the wildcard group in the same file carries a rule that blocks the same path, so even those are not AI-specific. commbank.com.au is the clearest: it opens with a group naming seventeen AI crawler tokens including GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, and the rule that takes its scripts is Disallow: /etc/, which its wildcard group repeats verbatim. The sites are not singling out AI. They are blocking a directory, and the scripts live in it.

A platform default blocked a platform script on 37 stores

37 of the 78 affected hosts were blocked by a rule matching /checkouts/, and on every one of them the blocked file is the same: /checkouts/internal/preloads.js, loaded by the storefront theme on the home page. Shopify's default robots.txt disallows /checkouts/ and /*/checkouts/, for good reasons that have nothing to do with assets, and Shopify's storefront serves a preload script from inside that path. Neither decision is wrong on its own. Together they mean that a merchant who has edited nothing ships a home page with a script their own file refuses, and the merchant cannot see it by reading either part.

The same pattern repeats one host out. Shopify serves the same preload from shop.app, and shop.app's robots.txt disallows /checkouts/internal explicitly, which accounts for 39 of the third-party blocked references in the next section. So the file is refused on both the store's host and the platform's host, by two separate rules, to every crawler in the wildcard group.

Worth recording for its own sake: shop.app's robots.txt opens with a framed comment block addressed to automated clients, stating that checkouts are for humans, that automated scraping or any end-to-end flow completing payment without a final human review step is not permitted, and that legitimate integrators must use the official Checkout Kit. It names a contact address for bot operators. A comment has no standing under RFC 9309 and no parser acts on it, so it is a policy written where only a reader will find it, which is a choice several large sites are now making.

How much this costs a store depends on what the script does, and the honest answer for a preload helper is probably very little: it is not the file that renders the product copy. The reason it is in this post is that it is the clearest available example of the mechanism, a platform default colliding with a platform asset path at a scale no individual merchant chose. We have measured the opposite side of the same stack, where 38 Shopify stores served 1,112 products as open JSON, and the pairing says something about platform defaults generally. They are the single highest leverage place to fix a problem like this, and the single easiest place for one to persist unnoticed for years.

183 home pages loaded a blocked resource from a host they do not control

The 7,555 third-party references on those 925 pages point at 712 distinct hosts, and the file governing each one belongs to whoever owns that host. Fetching all 712 returned 278 that parsed as a robots.txt, 382 in the 400 to 499 range, which imposes no restriction, 28 that answered with a web page, 15 that answered 5xx and 9 that failed. That leaves 2,449 references that could be evaluated against a real file, and 330 of them, on 183 home pages and 90 resource hosts, were disallowed.

The leading cases are not obscure. static.linear.app serves linear.app its stylesheets and scripts and answers robots.txt with two lines, User-agent: * and Disallow: /, which accounts for 51 blocked references on one page. js.stripe.com does the same and blocks 12 across several sites. rum.hlx.page, an Adobe telemetry endpoint, blocks 20. cdn.e-devlet.gov.tr serves the Turkish government's shared theme to turkiye.gov.tr and blocks 8 of its stylesheets.

One of them is the only genuinely AI-specific asymmetry in the whole measurement, and it is on a third-party host rather than any publisher's own. stats.wp.com, Automattic's statistics endpoint, disallows everything for the wildcard group and then opens one exception: User-agent: Googlebot followed by Allow: /. Googlebot gets the script. GPTBot does not. That is 22 blocked references across the WordPress sites in the corpus that load it, and it is a deliberate per-crawler decision rather than a collision, which makes it the exception that shows how rare the intentional version is.

For a site owner the consequence is a scoping problem rather than a bug to fix. A rule in your own file is yours to change. A Disallow: / on the CDN subdomain your framework ships with, or on your payment provider's script host, is not, and no amount of auditing your own robots.txt will surface it. This is also where prose parity and generative engine optimization part company with ordinary crawl auditing, because the input to the answer is assembled from hosts the publisher never enumerated. The one caveat on the figures above: the 15 resource hosts answering 5xx were not stable. A serial retry of six of them returned a different status on three, assets.squarespace.com answering 307 and dnnapi.com answering 404, so that bucket is reported as an observation and is excluded from the 330.

Googlebot loses more of this than GPTBot does

Running the same evaluation with four tokens settles who the blocked resources actually cost, and the answer is the opposite of what the subject suggests. With the Googlebot token, 2 of the 997 sites disallow the home page rather than 72, so 995 pages are in scope, and across them 592 references on 85 hosts are disallowed. ClaudeBot sees 518 on 79, PerplexityBot 550 on 81, GPTBot 495 on 78. The ordering is driven almost entirely by how many home pages each token is allowed to reach in the first place: a crawler barred from the page never gets far enough to lose its stylesheet.

That matters because of which of those four renders. Google says in its own documentation that it will not render JavaScript from blocked files, which is a statement about a renderer that exists. For the AI crawlers the picture is thinner, and we measured it directly: of nine operators, only 2 of 9 say either way whether their crawlers render JavaScript. A crawler that reads the delivered bytes and stops, which is what most of them document and what this measurement itself did, loses nothing at all when a stylesheet is refused. The cost is real and it is concentrated on the rendering path, which runs through Google's index and into AI Overviews and AI Mode.

So the honest version of this finding has two halves. If your site depends on JavaScript to put its words on the page, a blocked script is the difference between prose and an empty container, and we have measured both ends of that: JavaScript supplied 7.6 percent of the prose and all of it on 11 of 271 pages, and text in hydration JSON scored the same as no text at all. If your prose is in the HTML as served, a blocked stylesheet costs you very little, because the extractor was never going to apply it. The measurement above counts exposure, not damage, and the two are only the same number on a site that renders to produce its content. A reader on React or Framer is in the first group more often than they think.

The wildcard result is the finding with the longest reach, though. 482 of 495 blocked references were decided by a rule aimed at nobody in particular, which is the same shape as our earlier result that a trailing star changed nothing in 168 comparisons and that 1,004 robots.txt files named 2,209 crawler tokens while the rules that bite came from the group naming none. Adding an AI crawler token to a file whose wildcard group already blocks your asset directory changes nothing about this.

What this measurement does not show

The limits are worth stating in full, because several of them are large enough to change how the headline should be read.

It counts references, not bytes or effects. A page losing one analytics script and a page losing all 44 of its stylesheets both appear in the 78, and only the per-host table separates them. It covers one URL per hostname, the home page, so a site that blocks resources only on its product templates is not counted and a site whose home page is unusually clean may still have the defect elsewhere. It reads the markup as delivered with no JavaScript executed, so any resource injected by a script is invisible to it, which almost certainly undercounts: the pages most likely to load assets dynamically are the pages most likely to depend on them. It evaluates script src and stylesheet link elements only, not images, fonts, preloads or modulepreloads, so the real number of refused subresources is higher than 495 by some margin I have not measured.

It also evaluates a verdict rather than observing a crawler. No crawler was watched obeying or ignoring these rules; the figures say what a compliant client would do with each file, computed with the shipped evaluator, which is the same logic behind our robots.txt tester and documented in our methodology. Compliance is itself variable, and a non-compliant crawler would fetch the stylesheet and none of this would apply to it.

The checkable version on your own site needs no tooling. Open your home page source, list every script src and stylesheet href on your own host, and run each path, query string included, through your own robots.txt. Then do the same for the third-party hosts, which is the half nobody does. A scan at what GPTBot sees will show the request log for the page itself, and the AI crawler reference lists the tokens worth testing against.

One last thing this does not show, and it is the thing the corpus cannot answer: whether any of the 78 sites would prefer the current behaviour if they knew. A rule that blocks a build directory to keep junk out of an index is doing something an author wanted. The collision with the asset path is a side effect, and the fix is a narrower rule rather than a deleted one. That is a different kind of defect from the ones in our structured data work, where the markup is simply absent. Here the instruction is present, correct for its purpose, and wrong about one thing nobody checked.