# GoogleOther robots.txt: 63 of 1,015 files named it, and 43 that refused Google-Extended left it allowed > Lantad requested robots.txt once from each of the 1,419 hostnames in this repository's two committed corpus seed files on 21 October 2026, as LantadBot with no JavaScript executed. 1,015 parsed into at least one user-agent group. 63 named GoogleOther, the crawler Google documents as generic and available to various product teams, and 27 of the 1,015 disallowed it at the root, against 68 that disallowed Google-Extended. 43 files closed the root to Google-Extended and left GoogleOther open. - Canonical page: https://lantad.co/blog/googleother-robots-txt-63-of-1015-named-it - This file: https://lantad.co/blog/googleother-robots-txt-63-of-1015-named-it.md - Last substantive update: 2026-10-21 ## Key facts - **Published:** 2026-10-21 - **Category:** Findings - **Author:** Lantad - **Length:** 3950 words - **Takeaway 1:** A GoogleOther robots.txt rule is rarer than a rule for any named AI crawler in this corpus: 63 of the 1,015 robots.txt files Lantad parsed on 21 October 2026 named GoogleOther, against 173 naming GPTBot, 163 naming ClaudeBot and 148 naming Google-Extended. - **Takeaway 2:** Google's common crawlers documentation, carrying Last updated 2026-10-15 UTC and read on 21 October 2026, describes GoogleOther as the generic crawler that may be used by various product teams for fetching publicly accessible content from sites, and says it may be used for one-off crawls for internal research and development. The page names no product and does not say what the fetched content is used for. - **Takeaway 3:** 43 of the 1,015 files disallowed Google-Extended at the root on 21 October 2026 and left GoogleOther allowed there. The two are separate tokens, so a rule written for one selects no group for the other. - **Takeaway 4:** 27 of the 1,015 files closed the root to GoogleOther: 18 through a group naming the token, every one of them with an exact Disallow: /, and 9 more through a wildcard group that never mentions Google. All 18 left Googlebot allowed at the root. - **Takeaway 5:** 39 of the 64 files naming a GoogleOther token carry the default Squarespace robots.txt, where 29 crawler tokens and the wildcard head one group, and all 39 leave the root open to GoogleOther. ## Summary Google runs more than one crawler, and the robots.txt conversation has settled on two of them. Googlebot is the one everybody writes rules for, and Google-Extended is the one the AI training argument turned into a household token. GoogleOther is the third name on the same documentation page, it answers to its own token, and almost nobody addresses it. That is worth counting rather than assuming, because an [AI crawler](https://lantad.co/glossary/ai-crawler) policy assembled from the token list that circulates in blog posts inherits whatever that list leaves out. On 21 October 2026 this scanner requested the robots.txt of all 1,419 hostnames in the two committed corpus seed files in this repository, as LantadBot/1.0 with redirects followed and no JavaScript executed, and parsed the 1,015 that resolved into at least one user-agent group. Google's crawler documentation was opened and read the same day. What follows is how many of those files name GoogleOther, how many of them actually close anything to it, and what the files that carefully refuse Google-Extended do about the crawler sitting next to it in Google's own table. The token list behind the measurement is the one published on the [AI crawler reference](https://lantad.co/tools/ai-crawlers). ## What does a GoogleOther robots.txt rule control, and what does it not? Google documents its crawlers in three groups, and GoogleOther sits in the first of them, the common crawlers. The entry is short and it is the whole basis for any decision about the token. [Google's common crawlers page states that GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), and adds that it may be used for one-off crawls for internal research and development. The page carries Last updated 2026-10-15 UTC and was read on 21 October 2026. It gives the robots.txt token as GoogleOther, and it says of the common crawlers that they always obey robots.txt rules when crawling automatically. Read that entry for what it does not contain. It names no product. It does not say which teams, which systems, or what the fetched content becomes once it arrives. Compare the specificity Google offers elsewhere: Google-Extended is the token the company points at for AI training and grounding, and its own AI features documentation, carrying Last updated 2025-12-10 UTC, sends a reader who wants to limit AI training and grounding in some of Google's other systems to read more about Google-Extended. That page does not mention GoogleOther at all. So the honest statement about GoogleOther is narrow: it is a Google crawler with its own token, documented as generic and internal, and nothing on Google's pages ties it to a named product or to AI training. This post does not claim it feeds any model, because Google does not say so. That narrowness is the reason the token is interesting rather than a reason to ignore it. A publisher deciding what Google may fetch has a documented control for search, a documented control for AI training and grounding, and a third crawler whose purpose is described only as internal research and development. An earlier post here measured what happens to the middle one when [52 of 54 sites refusing Google-Extended still allowed Googlebot](https://lantad.co/blog/does-blocking-ai-training-affect-search-rankings-52-of-54), which is the split working as designed. The question this post asks is what those same careful files do about the crawler with no stated purpose, and the answer turns out to be almost nothing. For the engines where Google's retrieval does carry citations, the standing guide is the [page on getting cited in AI Overviews](https://lantad.co/how-to-get-cited/google-ai-overviews), and the opt-out there is a different mechanism again, as this blog found when it established that [the AI Overviews opt-out does not live on your site](https://lantad.co/blog/ai-overviews-opt-out-does-not-live-on-your-site). ## What was measured on 21 October 2026, and what was not The population is the 1,419 hostnames in the two committed corpus seed files in this repository: worker/seeds/corpus-seeds-platform.json with 392 hostnames across ten platform strata, and worker/seeds/corpus-seeds-industry.json with 1,027 across eight industry strata. Each hostname was asked once for https://hostname/robots.txt, as LantadBot/1.0, from one network location, with redirects followed and a twenty second timeout. That user agent is the one described on the [bot page](https://lantad.co/bot), and it was never altered to impersonate a crawler belonging to anybody else. The funnel sets every proportion below, so it is worth stating in full. 1,098 hostnames answered HTTP 200 and 7 answered 202, giving 1,105 in the 2xx range. 41 of those served an HTML document at the robots.txt address rather than a robots file, a defect this blog has [counted across the same corpus before](https://lantad.co/blog/robots-txt-validator-232-of-1059-carried-a-defect), and they were excluded. 49 more parsed into no user-agent group at all, usually a file holding only a Sitemap line or a comment, and a file with no group can name no token, so those were excluded too. That leaves 1,015, the denominator for every file count in this post. Of the rest, 162 hostnames answered 403, 61 answered 503, 50 answered 404, 4 answered 429 and 2 answered 401, with four single instances of other statuses, and 31 requests failed before any status arrived, 10 of them by timeout and 20 with no DNS answer. Parsing and evaluation used parseRobotsTxt and evaluateRobots from core/src/robots.ts, the code that ships in the product rather than a script written for this post. A file counts as naming a token when a user-agent line lowercases to that exact string. A root verdict is the answer evaluateRobots returns for the path slash, which is the strongest form of the question, since a file can leave the root open and still close most of what matters. The approach is the one set out on the [methodology page](https://lantad.co/methodology) and used for the [crawlability study](https://lantad.co/research/crawlability-study). Three limits apply to all of it. No Google crawler was observed doing anything, because there are no server logs here, so every statement about what GoogleOther does or is for is a statement about what Google publishes. One URL per hostname was requested, at the apex, so nothing here covers subdomains, where [73 of 277 gave an AI crawler a different answer from their apex](https://lantad.co/blog/robots-txt-subdomains-73-of-277-differ). And these 1,419 front doors come from two committed sampling frames built for another purpose, not from the web, so no proportion here generalises past them. ## 63 files named GoogleOther, and 27 closed the root to it 63 of the 1,015 files named GoogleOther, 43 named GoogleOther-Image and 43 named GoogleOther-Video. 64 named at least one of the three, the extra file being postman.com, which names a variant without naming the bare token. For scale on the same files, 173 named GPTBot, 163 named ClaudeBot, 148 named Google-Extended, 144 named CCBot, 133 named Bytespider, 118 named Applebot-Extended, 114 named Amazonbot, 101 named PerplexityBot and 72 named Googlebot. GoogleOther is therefore named less often than every AI crawler token this series has counted, including the ones belonging to vendors with a fraction of Google's traffic: it sits well below [140 of 1,013 files naming CCBot](https://lantad.co/blog/ccbot-robots-txt-140-of-1013-named-it) and below [110 of 1,014 naming Applebot-Extended](https://lantad.co/blog/applebot-extended-110-of-1014-robots-txt-files-named-it), measured on the same corpus days earlier. 27 of the 1,015 files disallowed GoogleOther at the site root. 18 of those 27 name the token themselves and the other 9 are caught by a wildcard group that never mentions Google: coralvilleanimalhospital.com, amsterdam.nl, wa.gov, helsinki.fi, sciencedirect.com, scielo.org, eluniversal.com.mx, theregister.com and instacart.com all write a blanket Disallow under the wildcard and shut out everything unnamed. The refusals that do name the token are blunt. Every one of the 18 was decided by an exact Disallow: /, not one by a narrower pattern, so no site in this corpus closed the root to GoogleOther and then reopened a section of it. The gap between 63 names and 27 refusals is the finding, and it is not the gap you would guess. 46 of the 64 files naming a GoogleOther token leave the root open to it. Only two of those 46 wrote a rule that looks like a considered position on the crawler: botcity.dev heads a GoogleOther group with Allow: / inside a file that sorts crawlers into groups by whether they return traffic, and gmarket.co.kr gives GoogleOther a single narrow Disallow covering one path prefix and leaves the rest open. The other 44 are files where the name is present and decides nothing, which is a different failure from not writing the rule at all, and one that is invisible to a reader skimming the file. Against the whole corpus the raw count of distinct user-agent strings is large and mostly inert, which this blog measured when it found [2,209 crawler tokens across 1,004 files](https://lantad.co/blog/crawler-user-agent-2209-tokens-in-1004-files). ## 43 files refused Google-Extended at the root and left GoogleOther allowed This is the number the post exists for. 68 of the 1,015 files disallowed Google-Extended at the root. 43 of those 68 left GoogleOther allowed at the root on the same day, in the same file. Put the other way round, a site in this corpus that has decided Google may not use its content for AI training is, about two times in three, still open to the Google crawler whose documented purpose is not stated. The mechanism is the group selection rule, and it is not a quirk. [RFC 9309 defines a robots.txt group as one or more user-agent lines followed by rules](https://www.rfc-editor.org/rfc/rfc9309.html), requires a crawler to obey the group whose user-agent value matches it most specifically, and falls back to the group headed by the wildcard when nothing matches. [Google's robots.txt documentation says the same and adds that rules for multiple user agents are grouped by repeating user-agent lines](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt). So a Disallow written under User-agent: Google-Extended is selected by exactly one token. GoogleOther does not match it, is not a prefix of it, and falls through to the wildcard group, which in most of these files permits the root. Nothing is malfunctioning: the file says what it says, and it says nothing about GoogleOther. The same arithmetic runs in the other direction and shows how little the token list in circulation covers. 211 of the 1,015 files name at least one of thirteen widely circulated AI crawler tokens. 147 of those 211 name no GoogleOther token at all, and 142 of the 147 leave GoogleOther allowed at the root. A file can enumerate GPTBot, ClaudeBot, CCBot, Bytespider, Applebot-Extended, Amazonbot and Google-Extended, and still hand Google's generic crawler the wildcard's permissions, because the enumeration is a copy of a list rather than a reading of a vendor's documentation. The inverse of this problem, where the name is present and the position in the file drains it of effect, was the subject of the count showing that [on 43 files the ClaudeBot line sits in the wildcard's own group](https://lantad.co/blog/claudebot-robots-txt-159-of-1014-named-it). Both failures produce a file that reads like a policy. Divergence between two tokens from one vendor is itself routine in this corpus, as the finding that [81 of 1,008 files allowed one of Googlebot and GPTBot and refused the other](https://lantad.co/blog/googlebot-vs-gptbot-81-of-1008-files-differed) established. ## Who refused it, and what the same files kept open The 18 files that disallow GoogleOther under a group naming the token are not a random slice of the corpus. Ten of the eighteen are news or media sites: cnn.com, bostonglobe.com, slate.com, theglobeandmail.com, statnews.com, chronicle.com, rappler.com, nation.africa and eplocalnews.org, with eluniversal.com.mx and theregister.com arriving at the same outcome through a wildcard instead. Four are research or cultural institutions, britishmuseum.org, nhm.ac.uk, nature.com and congress.gov. Two are healthcare bodies, cma.ca and petermac.org. The remaining three are amazon.com, villagedentaldtc.com and lamplightdev.com. This is the publisher-and-institution pattern that runs through every blocking measurement on this corpus, most recently when [41 of 69 news robots.txt files disallowed GPTBot and 2 of 97 finance files did](https://lantad.co/blog/which-industries-block-ai-crawlers-41-of-69-news-files). Inside the news stratum the contrast with Google-Extended is sharp. 68 news hostnames produced a parsed file. 35 of them disallow Google-Extended at the root and 10 disallow GoogleOther, and only 8 name GoogleOther at all. Even among the sites that treat Google's AI training token as a live decision, and that are the most practised readers of crawler documentation on the web, the third Google crawler is addressed by fewer than one in eight. What the 18 keep open is the part worth stating plainly, because it is evidence that these are decisions rather than accidents. All 18 leave Googlebot allowed at the root. Seventeen of the 18 also disallow Google-Extended, so the file draws exactly the line the documentation offers: search in, training out, generic crawler out. The one exception is cnn.com, which closes the root to GoogleOther and leaves Google-Extended allowed, the only file in the corpus to order those two that way. Three of the 18, congress.gov, cnn.com and theglobeandmail.com, also name the Image and Video variants explicitly, which Google's page says is unnecessary because both list GoogleOther as a token and only one token needs to match for a rule to apply. The nine wildcard cases are a different act altogether: those files refuse everything unnamed, which is the pattern behind the finding that [559 of 581 blocked pages were lost to a rule that never named the crawler](https://lantad.co/blog/blocked-by-robots-txt-559-of-581-by-a-wildcard). ## 39 of the 64 names arrived with a platform default, and none of them blocks anything The single largest source of the GoogleOther token in this corpus is not a decision by a site owner. 39 of the 1,015 parsed files open with the comment line marking them as the generated Squarespace robots.txt, all 39 name GoogleOther, and all 39 leave the root open to it. That is 39 of the 64 files naming a GoogleOther token, and 35 of them sit in the wix-squarespace stratum, which contributed 54 parsed files in total. The reason those names do nothing is the group selection rule from the section above, applied to a file where 29 crawler tokens and the wildcard head a single group. The generated file lists AI2Bot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, Meta-ExternalAgent and sixteen others on consecutive user-agent lines, then one more user-agent line carrying the wildcard, and only then the rules. Because a user-agent line that follows another user-agent line extends the group rather than starting one, all thirty tokens receive the same rules, and those rules are ordinary housekeeping: a handful of Disallow paths for configuration, search and account pages, and a long run of query-string patterns. No AI crawler is refused anything. This blog reached the same conclusion about the platform from the other direction when it found that [61 of 66 Squarespace sites named an AI crawler and did not block it](https://lantad.co/blog/squarespace-ai-crawlers-61-of-66-named-not-blocked), and the mechanism itself was counted on 18 October 2026 across a different token. So the distribution of this token has two sources that look identical in a grep and are opposites in effect. Eighteen files were written by somebody who read Google's documentation, decided the generic crawler was not welcome, and wrote an exact Disallow for it while keeping Googlebot. Thirty-nine inherited the name from a platform that put it in the wildcard's group. The practical consequence for a reader is that the presence of GoogleOther in a robots.txt file carries no information on its own, and neither does its absence, because absence and an inert mention produce the same crawl. The only way to know which of the three cases a site is in is to evaluate the file per token against the path in question, which is what the [robots.txt tester](https://lantad.co/tools/robots-txt-tester) does, and the broader pattern that the wildcard rather than the name decides most outcomes was established when this blog counted how often [robots.txt names the wildcard and not the AI crawler](https://lantad.co/blog/robots-txt-names-the-wildcard-not-the-ai-crawler). ## Questions and answers **What is GoogleOther, and what does a GoogleOther robots.txt rule control?** GoogleOther is a Google crawler with its own robots.txt token. Google's common crawlers documentation, carrying Last updated 2026-10-15 UTC and read on 21 October 2026, describes it as the generic crawler that may be used by various product teams for fetching publicly accessible content from sites, and says it may be used for one-off crawls for internal research and development. A rule under User-agent: GoogleOther controls that crawler and nothing else Google runs. The page names no product and does not say what the fetched content is used for. **Does blocking Google-Extended also block GoogleOther?** No. They are separate tokens, and under RFC 9309 a crawler obeys only the group whose user-agent value matches it, falling back to the wildcard group when none does. GoogleOther does not match Google-Extended and is not a prefix of it, so it falls through to the wildcard. On 21 October 2026, 43 of the 1,015 robots.txt files Lantad parsed disallowed Google-Extended at the root and left GoogleOther allowed there. **Does blocking GoogleOther affect Google Search?** Googlebot is the crawler Google documents for Search, and it answers to a different token, so a rule naming GoogleOther selects no group for Googlebot. That is what the files refusing it do in practice: all 18 of the corpus files that disallowed GoogleOther at the root under a group naming the token left Googlebot allowed at the root on 21 October 2026. This post measured robots.txt files and Google's documentation, not ranking outcomes, so it makes no claim about what any block costs in Search. **How many sites block GoogleOther?** In this corpus, 27 of 1,015 parsed robots.txt files disallowed GoogleOther at the root on 21 October 2026, and 63 named the token at all. 18 of the 27 named it in the group that blocked it, every one with an exact Disallow: /, and 9 were caught by a wildcard group naming no Google token. 10 of the 27 are news or media sites, out of 68 news hostnames whose file parsed. These are 1,419 hostnames from two committed sampling frames, not a sample of the web, so the proportions describe this corpus only. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot