# Googlebot vs GPTBot: 81 of 1,008 robots.txt files allowed one and refused the other > Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 17 October 2026, as LantadBot with no JavaScript executed, and parsed the 1,008 that resolved into at least one user-agent group. 81 allowed Googlebot at the site root and refused GPTBot, and none did the reverse. Beyond the root, 639 paths across 40 hostnames were open to GPTBot and closed to Googlebot, and every divergence in the corpus came from a file that names one of the two tokens. - Canonical page: https://lantad.co/blog/googlebot-vs-gptbot-81-of-1008-files-differed - This file: https://lantad.co/blog/googlebot-vs-gptbot-81-of-1008-files-differed.md - Last substantive update: 2026-10-17 ## Key facts - **Published:** 2026-10-17 - **Category:** Findings - **Author:** Lantad - **Length:** 3845 words - **Takeaway 1:** Googlebot vs GPTBot is two questions in one file, not one: of the 1,008 robots.txt files Lantad parsed on 17 October 2026, 81 allowed Googlebot at the site root and refused GPTBot, 925 allowed both, 2 refused both, and not one refused Googlebot while allowing GPTBot. - **Takeaway 2:** The 794 files that name neither token gave the two crawlers the same verdict on every path those files name. All 11,548 differing file and path pairs came from the 214 files that name Googlebot, GPTBot or both, so naming a crawler is what creates the difference. - **Takeaway 3:** 639 paths across 40 hostnames were closed to Googlebot and open to GPTBot on 17 October 2026, including /admin on mejuri.com and 46 paths on bestbuy.com. Google's robots.txt documentation, last updated 2026-08-31, states that user agent specific groups and global groups are not combined, so a group naming GPTBot replaces the wildcard group rather than adding to it. - **Takeaway 4:** 58 of the 81 files refusing GPTBot at the root left OAI-SearchBot allowed there. OpenAI's bots documentation, read on 17 October 2026, describes GPTBot as the crawler whose content may train its models and OAI-SearchBot as the crawler that surfaces sites in ChatGPT search, governed by separate robots.txt rules. - **Takeaway 5:** Writing a crawler its own group costs that crawler the file's housekeeping: 87 of the 172 files naming GPTBot gave its group fewer rules than their own wildcard group, 85 gave it exactly one rule, and 22 of those single rules were a bare Allow for the whole site. ## Summary Googlebot vs GPTBot reads like a question about two crawlers, and inside a robots.txt file it is a question about two groups. A file can say one thing to Google's search crawler and a different thing to OpenAI's training crawler, and the difference does not have to be deliberate: group selection is winner takes all, so the moment a file writes a group for one of them, that crawler stops reading every other group in the file. This post counts how often the two verdicts part company across a fixed corpus, in which direction, and what the open paths turn out to be. On 17 October 2026 this scanner requested the robots.txt of all 1,419 hostnames in the two committed corpus seed files in this repository, as LantadBot/1.0 with redirects followed, a fifteen second timeout and no JavaScript executed, and parsed the 1,008 files that resolved into at least one user-agent group. Every verdict below comes from evaluateRobots in core/src/robots.ts, the code that ships in the product, evaluated once for the token googlebot and once for gptbot. Separately, [Google's robots.txt documentation](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt), [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) and [OpenAI's bots documentation](https://developers.openai.com/api/docs/bots) were opened and read the same day. What follows is the root comparison, the paths where the two answers differ, the mechanism that produces them, and what a reader can check in their own file. The vocabulary for the tokens involved sits on the [AI crawler reference](https://lantad.co/tools/ai-crawlers). ## What was measured on 17 October 2026, and what was not The population is the 1,419 hostnames in the two committed corpus seed files in this repository: worker/seeds/corpus-seeds-platform.json with 392 hostnames across ten platform strata, and worker/seeds/corpus-seeds-industry.json with 1,027 across eight industry strata. Each hostname was asked once for its robots.txt at the apex, from one network location, as the user agent described on the [bot page](https://lantad.co/bot). No request was ever sent under Googlebot's user agent or GPTBot's, so nothing here observes either crawler. Every verdict in this post is what the file instructs, computed from the parsed file. The funnel decides every proportion below. 1,094 hostnames answered HTTP 200 and 8 answered 202, giving 1,102 in the 2xx range. 50 of those served an HTML page at the robots.txt address, a defect this corpus has [carried before](https://lantad.co/blog/robots-txt-validator-232-of-1059-carried-a-defect), and they were excluded. 44 more parsed into no user-agent group at all, usually a file holding only a Sitemap line or a comment, and a file with no group cannot decide anything, so those went too. That leaves 1,008. Of the rest, 178 answered 403, 50 answered 404, 33 answered 503, 5 answered 429, 3 answered 401, one each answered 406, 418 and 498, and 45 requests failed before a status arrived, 23 of them by timeout. A 403 on the file is a firmer refusal than any rule inside it, the asymmetry behind [92 of 1,056 sites refusing GPTBot the file itself](https://lantad.co/blog/can-gptbot-read-your-robots-txt-92-of-1056). Two comparisons were run on each of the 1,008 files. The first is the site root, the path slash, which is the blunt version of the question. The second is every path the file's own rules name, with a wildcard expanded to one literal segment and a trailing dollar anchor dropped, which gives 59,341 file and path pairs across the corpus, an average of 59 per file. That second set is a file's own rule surface rather than a site's real URL space: it says where a file's instructions disagree, not how many live pages are affected, and a file naming one long path and a file naming four hundred contribute very differently to the totals. Token matching is case insensitive, which this corpus confirmed from the other direction when [84 mis-cased crawler tokens all bound anyway](https://lantad.co/blog/is-robots-txt-case-sensitive-84-tokens-bound-9-pages-open). Three limits apply throughout. One URL per hostname was requested at the apex, so subdomains are outside all of it, and [73 of 277 gave an AI crawler a different answer from their apex](https://lantad.co/blog/robots-txt-subdomains-73-of-277-differ) when that was measured directly. The 1,419 hostnames are two committed sampling frames rather than a sample of the web, so no proportion generalises beyond them. And a separate fetch of the same corpus a few hours earlier the same day, for [the CCBot count published this morning](https://lantad.co/blog/ccbot-robots-txt-140-of-1013-named-it), parsed 1,013 files rather than 1,008: 403, 429 and timeout outcomes move between runs, so counts from two runs of the same corpus differ by a handful of hosts and the funnel above is the one that produced the figures in this post. ## Googlebot vs GPTBot: 81 files answered the site root differently 925 of the 1,008 files leave the root open to both tokens, which is the ordinary case and worth stating before anything else: on nine files in ten this comparison has no finding at all. 81 allow Googlebot and refuse GPTBot. Two refuse both, wa.gov and helsinki.fi, each through a wildcard Disallow for the whole site. None refuses Googlebot while allowing GPTBot. That one empty cell is the most compressed version of the result: in this corpus the search crawler is never the one shut out. 77 of the 81 refuse GPTBot through a group that names it, which is a decision somebody made and wrote down. The other 4 are the interesting shape, because nobody wrote GPTBot anywhere: the wildcard group carries a Disallow for the whole site and Googlebot has its own group that escapes it. instacart.com is the clearest of them, with Allow for the whole site under a Googlebot group at line 7 and Disallow for the whole site under the wildcard at line 531, and the same pattern appears on amsterdam.nl, scielo.org and coralvilleanimalhospital.com. These are sites whose files read as a search allowlist rather than an AI refusal, and the AI refusal is what they deliver. It is the same quiet mechanism behind [559 of 581 pages lost to a rule that never named the crawler](https://lantad.co/blog/blocked-by-robots-txt-559-of-581-by-a-wildcard). Where the 81 sit by stratum is the pattern every crawler count in this corpus has produced. 39 of the 65 news files refuse GPTBot while allowing Googlebot, against 42 spread across the other 943: 9 of 103 education files, 8 of 84 healthcare, 6 of 80 ecommerce, 4 of 19 static documentation sites, 3 of 119 SaaS, 3 of 78 government, 2 each of 89 finance, 76 travel and 32 WordPress sites, and one each across local media, SaaS marketing and single page app startups. Five strata produced none at all: 54 Wix and Squarespace sites, 35 Bubble and no code sites, 32 Shopify stores, 28 Framer sites and 18 Webflow sites. That matches the [42 of 77 blocking files that were news sites](https://lantad.co/blog/ai-crawler-statistics-77-of-706-blocked-42-were-news) when blocking was counted by category, and it means the headline proportion is carried by one stratum. The refusal is also narrower than it looks. 58 of the 81 files that close the root to GPTBot leave OAI-SearchBot allowed at that same root, and 37 of the 81 name OAI-SearchBot somewhere. OpenAI's bots documentation, read on 17 October 2026 and carrying no date of its own, states that OAI-SearchBot and GPTBot are controlled by separate robots.txt rules so a site can allow search while disallowing training, that OAI-SearchBot surfaces websites in ChatGPT's search features, and that disallowing GPTBot signals that a site's content should not be used to train its generative models. On 58 of these 81 sites the configuration says exactly that: no training, search allowed. This blog has counted the same split on the user triggered agent, where [38 files refused GPTBot and let ChatGPT-User in](https://lantad.co/blog/block-chatgpt-38-files-refuse-gptbot-allow-chatgpt-user). ## 639 paths were open to GPTBot and closed to Googlebot The root is the blunt question, and below it the comparison changes direction. Across the 59,341 file and path pairs the files' own rules name, 134 of the 1,008 files answer googlebot and gptbot differently somewhere, on 11,548 pairs in total. 10,909 of those pairs favour Googlebot, but that number is mostly the root block repeating itself: a file whose GPTBot group says Disallow for the whole site disagrees with its Googlebot verdict on every path it names, so one file with four hundred rules contributes four hundred pairs. 639 pairs, across 40 hostnames, run the other way, and those are the ones nobody configured on purpose. Restrict the comparison to the 925 files where both tokens are allowed the root, and the direction flips outright. 53 of those 925 still differ somewhere, on 933 pairs, and 630 of the 933 favour GPTBot against 303 favouring Googlebot. In other words, once the blunt refusals are set aside, a file that treats the two crawlers differently is twice as likely to be giving the AI crawler more than the search crawler, not less. 94 files diverge only in Googlebot's favour, 17 only in GPTBot's, and 23 in both directions at once. The 639 paths are not content. Sorted by what the path looks like, 196 are search or query string patterns, 30 are API or JSON endpoints, 30 are cart, checkout, order or account paths, 22 are test, staging or preview paths, 16 are CMS internals, 10 are administrative paths, and 335 are ordinary application routes such as /dashboard, /logout, /oauth and /home on single page apps. These are the paths a site closed to Google for crawl hygiene, and they are open to GPTBot by accident. time.com contributes 79 of them, 73 of those query string patterns, all under a Googlebot group the wildcard group does not repeat. embl.org contributes 52, bestbuy.com 46, magicmind.com 44, visitdubai.com 38 and mejuri.com 35, and the tail runs through coursera.org, zapier.com, canonical.com, decathlon.fr, geico.com, commbank.com.au, capitecbank.co.za and langhamhotels.com. What a site loses here is not access control, because robots.txt was never that: anything disallowed is still served to anyone who asks, as this corpus showed when [234 of 592 sites banning GPTBot served it a 200 anyway](https://lantad.co/blog/gptbot-bans-served-a-200-anyway). What it loses is crawl hygiene. The reason /search and /collections with a sort parameter are disallowed for Googlebot is that they are infinite and duplicative, and the reason an answer engine should not be reading /cart or /dashboard is the same one. A site that spent years tuning those rules for search has, on these 40 hostnames, handed the untuned version to the AI crawler. The [robots.txt tester](https://lantad.co/tools/robots-txt-tester) runs the same per-token evaluation against a single file, and [what GPTBot sees](https://lantad.co/tools/what-gptbot-sees) shows the page level consequence. ## Why naming a crawler detaches it from your wildcard rules None of this is a crawler misbehaving, and none of it is ambiguous in the standard. RFC 9309, published in September 2022, says in section 2.2.1 that a crawler must use case insensitive matching to find the group that matches its product token and then obey the rules of that group, that where several groups name the same token their rules must be combined into one group, and that if no matching group exists the crawler must obey the group whose user-agent line is the wildcard, if one is present. Google's robots.txt documentation, carrying Last updated 2026-08-31 UTC, states the consequence in one sentence: user agent specific groups and global groups are not combined. It adds that a crawler picks the group with the most specific user agent that matches it and that other groups are ignored. Section 2.2.2 of the RFC then decides rules within the selected group by the most specific match, defined as the match with the most octets. So a group is a replacement, not an addition, and that is the whole finding. Every divergence in this corpus is produced by naming. The 794 files that name neither googlebot nor gptbot gave the two tokens the same verdict on every path they name, without a single exception, because both fell to the same wildcard group. All 11,548 differing pairs came from the 214 files that name one or both. Nine of the 1,008 files carry no wildcard group at all, and 918 carry a wildcard group with at least one rule in it, which is the rule set a named group walks away from. The cost of walking away is measurable. Of the 172 files naming GPTBot, 87 give its group fewer rules than their own wildcard group, 71 the same number and 14 more. 85 give it exactly one rule, and 22 of those single rules are a bare Allow for the whole site. mejuri.com is the pattern in two lines: a wildcard group with 44 rules covering /admin, /cart, /checkout, /search and the Shopify collection filter traps that [this blog has read on 38 stores](https://lantad.co/blog/shopify-ai-crawlers-1112-products-as-open-json), and then a GPTBot group holding Allow for the whole site and nothing else. Compared against an unnamed crawler reading the same file, naming GPTBot made it looser on 439 paths across 28 files and tighter on 10,623 paths across 83 files, and on 2 files it opened a root the wildcard closes. The effect is not specific to GPTBot, and the honest way to say it is that Google benefits from it more often than any AI crawler does. Of the 70 files naming googlebot, 46 make Googlebot looser than an unnamed crawler, on 424 paths, and 10 of those open a root the wildcard closes. nhs.uk is the sharpest case in the corpus: its wildcard group carries a long list of disallowed paths including coronavirus booking and review surfaces, and its Googlebot group carries exactly one Disallow, for a loading spinner image, so Googlebot is admitted to 31 paths every other crawler is refused. The same arithmetic holds for ClaudeBot, named in 160 files and loosened in 26 of them on 383 paths, PerplexityBot, named in 102 and loosened in 29 on 617 paths, and CCBot, named in 139 and loosened in 15 on 393 paths. Writing a token into a file is the act that detaches it, whichever token it is, which is why [64 files that hand Applebot to their Googlebot group](https://lantad.co/blog/applebot-robots-txt-64-files-defer-to-googlebot) change the answer for Apple's crawler, and why [fourteen Google agents a wildcard does not stop](https://lantad.co/blog/google-fetchers-ignore-the-robots-txt-wildcard) is the same story told about one vendor. ## What to check in your own robots.txt The practical reading of all this is short. If your file has no group naming an AI crawler, the two crawlers in this comparison are getting identical treatment, and 794 of the 1,008 files here prove the point by producing no divergence at all. If your file does name one, then that group is the entire instruction set for that crawler, and every rule you keep outside it has stopped applying to it. Both halves of that are easy to check and neither needs a tool: read your wildcard group, read your named group, and ask whether you meant the second to be a replacement for the first. Three checks follow from the corpus. First, copy your wildcard hygiene rules into any crawler group you write, because the 639 pairs above are almost entirely housekeeping rules that one group had and another did not. bestbuy.com is the instructive case precisely because it tried: its file carries a deliberate block of AI tokens with its own list of blockouts, and AI crawlers are still looser than the wildcard on 46 paths, because the wildcard group grew entries the AI block never received. A copied rule set drifts, so the check is periodic rather than one off. Second, know which token you have actually ruled on. A GPTBot rule is not an OpenAI rule: 58 of the 81 files refusing GPTBot at the root leave OAI-SearchBot allowed, and whether that is right depends entirely on whether the objection was to training or to being read. The same applies in every direction across the registry, which is why [2,209 crawler tokens across 1,004 files](https://lantad.co/blog/crawler-user-agent-2209-tokens-in-1004-files) included 120 naming an agent Anthropic no longer documents, and why [89 of 145 files allowing AI crawlers ruled only on the whole site](https://lantad.co/blog/allow-ai-crawlers-robots-txt-89-of-145-whole-site). Third, treat the file as an instruction rather than an outcome. Everything in this post is what 1,008 files say, computed from the parsed text by the shipped evaluator. It is not what any crawler did, because no crawler was observed here and no request impersonated one, and a file that refuses a crawler cannot stop it: [79 of 115 sites refused a crawler their own robots.txt allows](https://lantad.co/blog/is-my-site-blocking-ai-crawlers-79-of-115) at the network layer instead, which the file never mentions. The gap between the rule and the behaviour is the reason this scanner reports both, and the method behind every figure here is on the [methodology page](https://lantad.co/methodology). ## Questions and answers **Is Googlebot the same as GPTBot in robots.txt?** No. They are separate product tokens and a rule written for one does not bind the other. Of the 1,008 robots.txt files Lantad parsed on 17 October 2026, 81 allowed Googlebot at the site root and refused GPTBot, and 639 paths across 40 hostnames were closed to Googlebot and open to GPTBot. **Does a group naming GPTBot inherit my wildcard rules?** No. Google's robots.txt documentation, last updated 2026-08-31 UTC, states that user agent specific groups and global groups are not combined, and that other groups are ignored once a crawler has selected its group. RFC 9309 section 2.2.1 says a crawler obeys the wildcard group only if no group matches its own token, so a named group replaces the wildcard rather than adding to it. **Does blocking GPTBot keep my site out of ChatGPT?** Not on its own. OpenAI's bots documentation, read on 17 October 2026, describes GPTBot as the crawler whose content may be used to train its models and OAI-SearchBot as the crawler that surfaces sites in ChatGPT's search features, controlled by separate robots.txt rules. 58 of the 81 files refusing GPTBot at the root in this corpus left OAI-SearchBot allowed there. **Which crawler do sites block more often, Googlebot or GPTBot?** GPTBot, by a wide margin, in this corpus. 81 of the 1,008 files refused GPTBot at the root while allowing Googlebot, 2 refused both, and no file refused Googlebot while allowing GPTBot. 39 of the 81 are news sites, so the proportion is carried by one stratum rather than spread across the corpus. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot