Crawl budget advice ends in a Disallow line, and AI crawlers read it too

What Google's crawl budget guide actually defines

The definition is one sentence and it is more restrictive than the term's popular use. Google's crawl budget guide, carrying Last updated 2026-07-22 UTC, states that Google defines a site's crawl budget as the set of URLs that Google can and wants to crawl. Two words in that sentence are doing the work. Can is capacity, and wants is demand, and the guide treats them as separate quantities that combine into one outcome.

Capacity is described as a crawl capacity limit, also called hostload, and the guide defines it as a limit on the total amount of time your server spends holding connections open for Google, factoring in both the number of parallel connections and their duration. This is a politeness ceiling rather than a quota. The guide names two things that move it: crawl health, where consistent responses and stable latency raise the limit while slowdowns and 5xx or 429 errors lower it, and Google's own crawling limits, which the guide frames as finite resources that have to be prioritised across the whole web. Neither of those is a number a publisher is given. They are described qualitatively, and the guide offers no figure for what any site's limit is.

Demand is the other half, and the guide is clear that it is crawler specific. For Googlebot it names three influenceable factors: perceived inventory, where duplicate or unnecessary URLs waste crawling time that would otherwise go to real pages, popularity, where more popular URLs are crawled more frequently, and staleness, where Google's systems recrawl in order to detect change. The combination matters more than either part. The guide's own framing is that capacity and demand together produce the budget, so a site with plenty of spare capacity and low demand simply gets crawled less, and no amount of server headroom changes that.

What that definition does not include is anything about whether the page a crawler fetches is readable when it arrives. Crawl budget is about which URLs get requested and how often. Whether the response to that request contains the text a human would see is a separate property, which is what prose parity measures and what our scoring method sets out. The two are easy to conflate because both are described loosely as crawlability. They fail independently, and a site can have abundant crawl budget and no readable content, or the reverse.

Who the guide says should read it

The guide opens by turning most readers away, and this is the part least often quoted. It states that if your site does not have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you do not need to read the guide. That is an unusually direct disclaimer for vendor documentation, and it is the sentence to hold on to when someone proposes a crawl budget project.

The thresholds it gives are specific. The guide addresses large sites, which it puts at more than 1 million unique pages with content changing about once a week, and medium or larger sites, which it puts at more than 10,000 unique pages with content changing daily. It adds a third audience defined by a symptom rather than a size: sites with a large proportion of URLs reported in Search Console as Discovered, currently not indexed. Those numbers are worth setting against an ordinary commercial site. A storefront with a few thousand products and a blog is nowhere near either threshold, and the guide is telling that site plainly that the document is not for it.

This matters for the AI visibility conversation because the crawl budget framing is often imported wholesale into it. The argument runs that a dozen AI crawlers are now fetching pages, so budget is tighter and pruning is more urgent. The first half of that is a real observation about traffic volume and cost. The second half does not follow from anything in this guide, because the budget the guide describes is Google's alone. Anthropic's or OpenAI's crawlers consume your server's capacity, which is a hosting concern worth taking seriously, but they do not draw down a shared allowance that Googlebot is also spending. No published Google documentation describes such a pool, and none of the vendor crawler documentation we have read describes one either.

There is a related trap in reasoning about how quickly a crawler learns a page changed. Speeding up discovery and improving what a crawler can read are different problems, which we set out when IndexNow's own documentation showed that a 200 only means received, and the same separation applies here. Crawl budget is about the rate of requests. It says nothing about the content of the responses, and a site that solves a rate problem it did not have will see nothing change in its AI visibility.

The remedy is a robots.txt rule, and robots.txt is not addressed to Googlebot alone

The guide's best practices are mostly about URL inventory, and the first of them is the one with reach beyond Google. It says to block crawling of URLs using robots.txt, on the reasoning that some pages might be important to users but you do not necessarily want them to appear on Google surfaces or get reprocessed by Google's systems. It also, notably, tells you not to use noindex for this purpose, because Google will still request the page and then drop it once it sees the noindex meta tag or header, wasting crawling time. The advice is internally consistent: if the goal is to stop the fetch, only the file that is read before the fetch can do it.

That is exactly why the advice does not stay inside Google. Robots.txt is fetched by every crawler that intends to honour it, and the rules in it are selected by product token, not by vendor intent. RFC 9309, published on the Standards Track in September 2022, sets out the matching procedure. A crawler looks for groups matching its own product token, and if more than one group matches, the matching groups' rules must be combined. If no matching group exists, the specification states that crawlers MUST obey the group with a user-agent line with the * value. Only if no group matches the token and there is no wildcard group, or there are no groups at all, do no rules apply.

Follow that through with a concrete case. A site adds Disallow lines under User-agent: * to keep faceted search URLs and internal endpoints out of Googlebot's queue. GPTBot, ClaudeBot and PerplexityBot arrive, find no group naming them, and are therefore required by the specification to obey the same wildcard group. The lines were written about Googlebot's crawl capacity and they are now the access policy for every AI crawler that reads the file. Nothing in the guide warns about this, and nothing should be expected to, because the guide is Google's documentation of Google's behaviour. The overlap is a property of the file, not an oversight in the page.

The direction of the resulting mistake depends on the site. If the URLs being pruned are genuinely low value, blocking everything from them is fine and arguably desirable. If the pruning is broader, and it often is when a site is trying to reduce crawl volume in general, then pages a site would very much like an answer engine to read can end up behind the same rule. Working out which crawler is affected by which line is a per token question, which is what our robots.txt tester resolves, and the set of tokens worth testing is listed on our AI crawler reference. Getting this wrong is not exotic. We have written about how a renamed crawler token leaves a robots.txt group matching nothing, which is the same matching procedure producing the opposite surprise.

Which crawlers the guide names, and which it does not

Read on 12 August 2026, the crawl budget guide names six Google crawlers and fetchers in the course of explaining how demand is calculated and which traffic counts: Googlebot, AdsBot, the Google Shopping crawler, Feedfetcher, Google Read Aloud and APIs-Google. Google-Extended, the token Google documents for controlling use of content in its generative AI products, does not appear on the page. Neither does any crawler belonging to another vendor, which is entirely as it should be, since Google does not document other companies' crawlers.

The absence is worth stating because of what gets built on top of it. We are aware of no first party document, from Google or from any other vendor, that defines a crawl budget spanning several companies' crawlers. Vendor crawler documentation in this space tends to be thin on operational limits generally, which we found when Googlebot's documented 2MB fetch ceiling turned out to have no counterpart in three AI crawler documentation pages read on 10 August 2026. Where a vendor's documentation is silent on a limit, the honest position is that the behaviour is unspecified, not that it matches Google's.

That silence also complicates the standard advice to treat training and search crawlers differently, because acting on it requires two published names per vendor. When we counted the registry on 1 August 2026, six of the nine AI vendors we track published exactly one crawler token, which means for most vendors there is no way to write a rule that separates a training fetch from a search fetch. A crawl budget cleanup expressed as a wildcard Disallow therefore hits both purposes at once for those vendors, with no mechanism available to distinguish them.

One further asymmetry is worth naming, and it runs the other way. Google documents nine user triggered fetchers that generally ignore robots.txt rules, and five special case crawlers that ignore the global user agent group, which together are fourteen Google agents a robots.txt wildcard does not stop. So the file is not uniformly obeyed even by one vendor, and reasoning about it as a single switch is wrong in both directions. That is the practical case for testing a robots.txt file per token rather than reading it as prose, and for keeping a record of which tokens a site intends to admit, which is what our crawler policy page does for this site's own scanner.

What to check before you add the Disallow line

None of this is an argument against pruning a crawl. It is an argument for knowing the blast radius of the instrument before picking it up, and the checks are cheap. The first is whether the guide applies to you at all, which its own opening paragraph settles in about thirty seconds against your page count and your publishing rate. If your pages are crawled the day you publish them, the guide says to stop reading, and taking it at its word saves the rest of the work.

The second check is what a proposed rule does to each token you care about, rather than to Googlebot alone. A rule under a wildcard group is an access decision for every crawler without a group of its own, so the question to answer before deploying is which of the tokens you want reading the site would newly be excluded. Tools help here, but so does simply reading the file with the matching procedure in mind. Our what GPTBot sees view answers the narrower question of what one named crawler receives for one URL, which is the fastest way to confirm that a rule did what you expected rather than what you intended.

The third check is whether robots.txt is even the right instrument for the goal. Google's own robots.txt introduction, last updated 2025-12-10 UTC, states that robots.txt is not a mechanism for keeping a web page out of Google, and that a page disallowed in robots.txt can still be indexed if it is linked from other sites, with the URL and information such as anchor text still capable of appearing in results. RFC 9309 makes the same point in stronger terms for every crawler, stating flatly that these rules are not a form of access authorization. If the requirement is that content must not be retrievable, robots.txt does not deliver it and never claimed to.

Where the crawl budget guide is unambiguously useful is in the parts of its advice that are about your server rather than about other people's crawlers. Returning a 404 or 410 for permanently removed pages, which the guide describes as a strong signal not to crawl a URL again, is good hygiene for every client. Supporting 304 Not Modified, so an unchanged page can be answered without resending the body, reduces load for everyone that asks. Unlike a wildcard Disallow, neither has a downside for a crawler you would have been happy to serve. Status codes carry protocol meaning in their own right, and the sharpest example is the robots.txt file itself, where a 404 allows every crawler and a 503 blocks them all under RFC 9309. For anyone whose actual goal is generative engine optimisation rather than crawl scheduling, that is where the effort belongs, alongside the content questions covered on our research page and in getting cited in Google AI Overviews.

It is worth being explicit about what Lantad can and cannot say here, since this site sells measurement. We have not measured Google's crawl scheduling, we hold no data on any site's crawl capacity limit, and nothing above is presented as our finding. What we do hold is capture data on what crawlers are permitted to fetch. In the six real site captures stored on 15 July 2026, covering 15 crawler tokens each, all 90 token and site pairs were recorded as allowed by robots.txt, and only one of the six sites carried a rule matching any of those tokens at all. Those captures are described in what a crawler actually meets on a real storefront. Six sites is a small sample and it proves nothing about the web, but it is a useful reminder of the starting position: on most sites the AI crawler tokens are not named anywhere in robots.txt, so whatever the wildcard group says is what they get.