Google-CloudVertexBot robots.txt: 26 of 1,023 files named it, and all 26 named Google-Extended as well

What does a Google-CloudVertexBot robots.txt rule control?

Google's common crawlers page states that crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by site owners for building Vertex AI Agents, and that it has no effect on Google Search or other products. The page carries Last updated 2026-10-15 UTC and was read on 23 October 2026. It gives the robots.txt token as Google-CloudVertexBot, lists Googlebot as a second token for the same crawler, and says of the common crawlers that they always obey robots.txt rules when crawling automatically.

The other half of the picture is on the Vertex AI side, and it is the half that says who does the asking. Google's data preparation page for its generative AI app builder, at docs.cloud.google.com/generative-ai-app-builder/docs/prepare-data, carries Last updated 2026-10-21 UTC and tells the customer building a website data store to make sure that Google-CloudVertexBot can access the content, adding that the crawler needs to crawl and index the information including any paywalled content. The same page states that a customer planning to use advanced website indexing must be able to verify the domains for the URL patterns in the data store. Domain verification is the detail that separates this crawler from every other AI crawler this blog counts: the fetch is gated on somebody proving they control the site, so a refusal here does not keep a stranger out, it breaks a build that somebody with access to the domain commissioned.

That distinction matters because the paywall instruction reads alarmingly on its own. A publisher who sees an instruction to let a Google crawler behind the paywall, and who has already read that Google's paywall markup names two crawlers and both are Google's, is being told something narrower than it sounds: the page is addressed to the party building the data store, who has verified the domain, about their own content. It is not a request to the open web. Set it against the controls that do govern ambient fetching. Google-Extended is the token Google points at for training and grounding, which this blog measured when 52 of 54 sites refusing Google-Extended still allowed Googlebot. The retrieval that carries citations has its own guide on getting cited in AI Overviews. And Google runs fetchers that ignore robots.txt rules written for the wildcard entirely, as the count of fourteen Google agents a robots.txt wildcard does not stop established. Google-CloudVertexBot belongs to none of those categories. It is the one Google crawler on the list whose arrival implies a prior request.

What was measured on 23 October 2026, and what was not

The population is the 1,419 hostnames in the two committed corpus seed files in this repository: worker/seeds/corpus-seeds-platform.json with 392 hostnames across ten platform strata, and worker/seeds/corpus-seeds-industry.json with 1,027 across eight industry strata. Each hostname was asked once for its robots.txt over HTTPS, as LantadBot/1.0, from one network location, with redirects followed and a twenty second timeout. That user agent is the one described on the bot page, and no request was sent under a user agent belonging to anybody else.

The funnel sets every proportion below. 1,103 hostnames answered HTTP 200. 286 answered another status, 163 of them 403, 55 of them 503, 51 of them 404, 7 of them 202, with 10 single or low-count instances of 401, 406, 418, 429, 498 and 529. 30 requests produced no status at all: 9 reached the twenty second timeout and 21 failed at connection time, and this scan did not record whether DNS, TLS or a reset caused each of those 21. Of the 1,103 that answered 200, 35 served an HTML document at the robots.txt address rather than a robots file, a defect this blog has counted across the same corpus before, and 45 more parsed into no user-agent group at all, usually a file holding only a Sitemap line or a comment. A file with no group can name no token, so both sets were excluded. That leaves 1,023 files, carrying 7,108 user-agent groups and 68,900 allow and disallow lines between them, and 1,023 is the denominator for every file count in this post.

Parsing and evaluation used parseRobotsTxt and evaluateRobots from core/src/robots.ts, the code that ships in the product rather than a script written for this post. A file counts as naming a token when a user-agent line lowercases to that exact string, which is sound here because Google states that both the user-agent field name and its value are case-insensitive, and because this corpus has already shown that 84 mis-cased crawler tokens bound anyway. A root verdict is the answer the evaluator returns for the path slash, which is the strongest form of the question, since a file can leave the root open and still close most of what matters. The approach is the one set out on the methodology page and used for the crawlability study.

Three limits apply to all of it. No Google crawler was observed doing anything, because there are no server logs here, so every statement about what this crawler is for is a statement about what Google publishes. One URL per hostname was requested, at the apex, so nothing here covers subdomains, where 73 of 277 gave an AI crawler a different answer from their apex. And nothing here measures whether any of these 26 sites runs a Vertex AI data store, which is the only context in which their rule has an effect at all.

26 files named it, and 23 of those closed the root

26 of the 1,023 files named Google-CloudVertexBot. That is the rarest Google token in this corpus except Google-InspectionTool at 3 and Storebot-Google at 6, and it sits far below the names that circulate: 177 files named GPTBot, 166 ClaudeBot, 151 Google-Extended, 146 CCBot, 135 Bytespider, 127 Meta-ExternalAgent, 119 Applebot-Extended, 117 Amazonbot, 103 PerplexityBot, 81 OAI-SearchBot, 74 Googlebot and 66 GoogleOther. For comparison on the same corpus days earlier, this blog counted 63 of 1,015 files naming GoogleOther and 159 of 1,014 naming ClaudeBot.

What separates this token from the rest is the ratio of names to refusals. 23 of the 26 files that named it disallowed it at the root, every one of them with an exact Disallow slash rather than a narrower pattern. Only three named it and left the root open: mediamarkt.de and geico.com each give it a single-token group with Allow slash, and sonarsource.com puts it in a 31-token group headed by a comment introducing an explicit AI crawler allow. So this is not a token that arrives inert in a platform default, which is the usual story in this corpus and the reason a name in a file carries so little information. Every other Google token here is dominated by generated files that mention a crawler and rule on nothing. This one was typed by somebody who meant it, and what they meant 23 times out of 26 was no.

Who they are follows the publisher pattern that runs through every blocking measurement on this corpus. 10 of the 26 are in the news stratum, which contributed only 68 parsed files: bloomberg.com, chronicle.com, ansa.it, derstandard.at, nation.africa, nos.nl, thehill.com, nzherald.co.nz, theregister.com and theglobeandmail.com. Three more are research or cultural institutions, britishmuseum.org, nhm.ac.uk and sciencedirect.com, and congress.gov arrives through a single group heading 128 tokens. Two are healthcare bodies, cma.ca and petermac.org. The rest are amazon.com and tripadvisor.com, two local news sites, two Bubble builds, one static docs site, and the three allows.

One detail inside those files is worth separating out, because it is a rule that cannot work. Six files head a group with the bare token cloudvertexbot: qcitymetro.com, congress.gov, derstandard.at, slate.com, nzherald.co.nz and theglobeandmail.com. Neither of the two tokens Google publishes for this crawler is cloudvertexbot, so a group headed that way addresses no crawler on Google's page. Five of the six also name the documented token, so the extra line is harmless insurance. slate.com is the exception: it writes the bare form only, which is why it appears in none of the counts above. That is the same class of defect as a name sitting in the wrong place, which this blog measured when it found 2,209 crawler tokens across 1,004 files, many of them addressing nothing that exists.

All 26 arrived alongside Google-Extended, and 64 GPTBot refusals left it open

Every one of the 26 files naming Google-CloudVertexBot also named Google-Extended. Not one file in this corpus named the Vertex token without the training token, in either direction of writing: the 151 files naming Google-Extended include all 26, and the other 125 name Google-Extended and say nothing about the Vertex crawler. The token travels as part of a training opt-out block, not as a decision of its own, and the shape of those blocks shows it. 22 of the 26 give the token its own single-token group, which is what a copied list of AI crawler names produces when each name gets its own stanza, and the four exceptions are bundles of 2, 15, 31 and 128 tokens.

That pairing is the finding, because the two tokens do different work. Google-Extended is documented as the control for whether crawled content trains future Gemini models. Google-CloudVertexBot is documented as serving crawls requested by site owners, with no effect on Search or other products. A file that refuses both has made one decision about Google's use of its content for training and a second, separate decision that nobody may build a Vertex AI agent over its pages, including itself. For most of the 23 that is probably the intended outcome, and a news publisher refusing every Google AI fetch it can name is a coherent position. It is worth knowing which of the two lines costs what, and the Vertex line is the one whose only documented effect lands on a project someone with domain access commissions.

Run the comparison the other way and the enumeration habit becomes visible. 89 of the 1,023 files disallowed GPTBot at the root on 23 October 2026. Under Google's two-token reading, 64 of those 89 left Google-CloudVertexBot allowed, and 25 refused both. 69 files disallowed Google-Extended at the root, and 44 of those 69 left the Vertex crawler allowed. So a site that has gone to the trouble of refusing OpenAI's crawler at the front door is, about seven times in ten, still open to this one, because the lists that circulate do not carry it. The mechanism is group selection rather than anything malfunctioning. RFC 9309 defines a robots.txt group as one or more user-agent lines followed by rules and has a crawler obey the group matching it most specifically, falling back to the wildcard when nothing matches. A Disallow under User-agent: GPTBot is therefore selected by one token and no other.

Underneath all of that sits the wildcard, which decides far more of this corpus than any named rule. 1,015 of the 1,023 files carry a wildcard group, and 926 of the 1,023 have neither a Vertex group nor a Googlebot group, so for those 926 the wildcard is the whole answer. Only 2 of the 25 root refusals under Google's reading come from a wildcard rather than a named rule, wa.gov and helsinki.fi, both of which write a blanket Disallow slash that closes the site to everything unnamed including Googlebot. That is the pattern behind the finding that 559 of 581 blocked pages were lost to a rule that never named the crawler, and the reason this blog keeps separating a named refusal from an inherited one, as it did when it counted how often robots.txt names the wildcard and not the AI crawler. Divergence between two tokens from one vendor is routine here, as 81 of 1,008 files allowing one of Googlebot and GPTBot and refusing the other established.

Google lists two robots.txt tokens for this crawler, and our parser reads one

Google's crawler list gives some crawlers more than one robots.txt token, and says in its own words that you need to match only one crawler token for a rule to apply. For Google-CloudVertexBot the published tokens are Google-CloudVertexBot and Googlebot. So a file with no Vertex group but with a Googlebot group hands this crawler the Googlebot rules, and the wildcard never comes into it.

The fallback is published per crawler and cannot be derived from the name, which is the part that catches a parser out. Google's robots.txt specification states that only one group is valid for a particular crawler and that its crawlers find the group with the most specific user agent matching them, and that page, carrying Last updated 2026-08-31 UTC, gives the worked example: with a googlebot-news group, a wildcard group and a googlebot group in one file, Storebot-Google follows the wildcard, because there is no specific Storebot-Google group. Storebot-Google is a Google crawler sitting beside this one in the same table, and it falls past the Googlebot group to the wildcard because its published token list holds one entry. Google-CloudVertexBot does not, so it stops at Googlebot. Nothing in either name predicts which of those two things happens.

Lantad's parser derives it from the name. The group selection in core/src/robots.ts matches a group token when it equals the bot token or is a prefix of it, taking the longest such token and falling back to the wildcard, which is the behaviour the file documents for itself. That rule is right for most of Google's table, because Googlebot-Image, Googlebot-Video and Googlebot-News all begin with Googlebot, and GoogleOther-Image and GoogleOther-Video begin with GoogleOther, so the published fallback and the prefix agree. Two entries on the page break the pattern, and both are of the form Google-Something: Google-InspectionTool and Google-CloudVertexBot list Googlebot as their second token, and neither name begins with Googlebot. For those two the parser falls to the wildcard where Google's table points at the Googlebot group.

This corpus gives the size of that gap rather than leaving it as a worry. Under the prefix reading, 33 of the 1,023 files close the root to this crawler; under Google's two-token reading, 25 do. The 8 files that differ are coralvilleanimalhospital.com, botcity.dev, amsterdam.nl, scielo.org, eluniversal.com.mx, sap.com, instacart.com and gmarket.co.kr, and every one of them differs the same way: a blanket wildcard Disallow slash that the Googlebot group overrides, so the documented reading is the more permissive one and the prefix reading reports a block that Google's own table says would not happen. 71 files have a Googlebot group and no Vertex group, which is the set where the question can arise at all, and 54 of those 71 write Googlebot rules that differ from their wildcard rules. Probing ten representative paths rather than the root alone, 20 of the 71 return a different verdict somewhere. Nothing in this post changes the parser: the scanner evaluates this crawler as it did yesterday, and the figure to carry away is that the reading affects 8 root verdicts in 1,023 files and 20 files across a wider path probe. The habit of publishing the number rather than the reassurance is the same one behind the finding that the AI Overviews opt out does not live on your site.

What to check on your own site

Start with whether the question applies. This crawler's documented effect runs through a Vertex AI data store, and advanced website indexing requires verifying the domains in that store, so if nobody at your organisation has built or commissioned one, a rule for this token changes nothing today and refusing it costs nothing today either. The asymmetry is worth stating in those terms: the cost of a block lands on a future internal project, and the benefit it buys is not a reduction in training exposure, which Google-Extended covers, nor a reduction in Search crawling, which Googlebot covers.

If the question does apply, evaluate the token rather than reading the file. Three of the files in this corpus name it and allow it, 23 name it and refuse it, six write a bare cloudvertexbot line that addresses nothing, and 926 have no group that could apply, so the presence or absence of the string tells you almost nothing about the verdict. Evaluating a path per token is what the robots.txt tester does, and the one extra step here is to check what your Googlebot group says, because Google's published fallback for this crawler is Googlebot and a wildcard Disallow does not have the last word where that group exists. 71 files in this corpus are in exactly that position.

Then check the thing a robots.txt file cannot tell you. A fetch that robots.txt permits can still fail at the paywall, the consent wall or the login, and the Vertex page's instruction about reaching paywalled content is a warning about precisely that path. This blog measured the general case when 2 of 66 password pages tripped the login wall test, and the gap between a permitted fetch and a readable page is most of what AI visibility measurement is for. Robots.txt is the first gate, not the last one.