Fourteen Google agents that a robots.txt wildcard does not stop

Google sorts its own traffic into three classes, and only one always obeys robots.txt

Google's crawling documentation now lives on its own site rather than inside Search Central, and the overview page of Google's web crawling carries a last updated date of 12 June 2026. It opens by splitting everything Google sends at a website into three categories. That split is the whole subject here.

Common crawlers are the ones used for Google's products, with Googlebot named as the example, and the page states that they always respect robots.txt rules for automatic crawls. This is the category almost everyone has in mind when they write a rule. It is also the category our scanner does not model, which we set out in full when we audited the one crawler our scanner does not model.

Special-case crawlers are described as being used by specific products where there is an agreement between the crawled site and the Google product about the crawl process. The example the overview gives is not buried in a footnote. It says AdsBot ignores the global robots.txt user agent with the ad publisher's permission, and it says it in the opening summary.

User-triggered fetchers are described as part of tools and product functions where the end user triggers a fetch. A person asks a Google product for something, and a Google system goes and gets it on their behalf.

The exemptions live entirely in the second and third categories, and both are documented rather than inferred. None of this is a leak or a discovery. It is written on two pages that Google maintains, dates and links from its own navigation. The only reason it lands as a surprise is that most public advice about AI access treats robots.txt as one lever with one set of consequences.

Counting the entries on those two pages on 31 July 2026 gives nine user-triggered fetchers and five active special-case crawlers. Fourteen named agents, every one of them Google's, for which a group headed with an asterisk is not the thing that decides.

The nine user-triggered fetchers, and the one sentence that governs all of them

Google's user-triggered fetchers page carries a last updated date of 16 July 2026 and documents nine of them, each with its own section and its own user agent table. In the order the page lists them: Chrome Web Store, whose token is Google-CWS. Feedfetcher, as FeedFetcher-Google, used for RSS and Atom feeds for Google News and WebSub. Gemini Notebook, as Google-GeminiNotebook. Google-Agent. Google Messages, as GoogleMessages, which generates link previews for URLs sent in chat messages. Google Pinpoint. Google Publisher Center, as GoogleProducer. Google Read Aloud. And Google Site Verifier, which fetches Search Console verification tokens.

One sentence on that page governs all nine. Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. The word generally is doing real work and Google does not enumerate the exceptions to it, so the honest reading is that robots.txt is not the mechanism that decides these requests rather than that it is guaranteed to be ignored in every case.

The page also tells you where the addresses come from, which is the part a site owner can act on. The IP ranges are published in three JSON objects: user-triggered-fetchers.json, user-triggered-fetchers-google.json and user-triggered-agents.json. That is the same shape of artefact we opened and counted when we read OpenAI's four crawler IP files, and the same shape of artefact that exists because a user agent header proves nothing on its own.

Reverse DNS is documented too. The masks are given as a gae.googleusercontent.com pattern for Google owned fetchers and a google-proxy pattern under google.com for user owned ones. Those are the identity checks available to a site owner here, and they run at the edge or in the logs rather than in a file, which is the same conclusion we reached about why a user agent is a claim rather than an identity.

Ignoring the global group and ignoring the file are two different exemptions

Google's special-case crawlers page, last updated 11 February 2026, lists five active entries and draws a distinction that matters more than the count does.

Four of them, APIs-Google, AdsBot-Google-Mobile, AdsBot-Google and Mediapartners-Google, each carry the same line in their robots.txt row: the global user agent is ignored. Each entry then shows an example group headed with that crawler's own token, which is the page telling you the remedy in the same breath as the exemption. These four still read a named group. What they do not read is the group you wrote for everybody.

The fifth is different in kind. Google-Safety's row says the Google-Safety user agent ignores robots.txt rules, with no named group offered and no example to write. The page also states, above the table, that the special-case crawlers may ignore robots.txt rules and so they operate from a different IP range than the common crawlers.

Why the four cannot be caught by a wildcard is not a quirk of those crawlers, it is the group selection rule. Google's robots.txt specification page, last updated 8 July 2026, states that only one group is valid for a particular crawler, that the crawler finds the group with the most specific matching user agent, and that user agent specific groups and global groups are not combined. A crawler exempted from the global group therefore has nothing left to obey unless you name it.

This is ordinary RFC 9309 group semantics rather than anything Google invented, and it is the same specificity rule our own parser implements: pick the most specific matching group token, fall back to the asterisk group, merge the groups that share the winning token. The difference is that our parser applies the fallback for any token it does not find named, because a file reading can only ever tell you what the file says. Reading a status code has the same property, which is why a 404 on robots.txt allows every crawler while a 503 blocks them all even though nothing about the site changed in between.

Google-Agent is the AI relevant one, and it is filed as a fetcher rather than a crawler

Of the fourteen, one belongs to the subject this site exists for. Google's crawling documentation changelog records that on 20 March 2026 Google added the Google-Agent user agent, along with the IP ranges for user triggered agents, to the documentation on verifying requests from Google crawlers and fetchers.

Google's description of it is one sentence: Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request. Its addresses come from user-triggered-agents.json, the third of the three JSON objects named on the fetchers page. Its user agent string is a Chrome string with the token appended in the compatible clause, in both a mobile and a desktop form, so anything matching on a substring will see a browser unless it looks for the token specifically.

The filing decision is the finding here. An agent that navigates a website and performs actions on it is, by any plain reading, doing what a crawler does, at least as far as the site on the receiving end is concerned. Google has classified it with the link preview generators and the feed readers, and that classification is exactly what carries the robots.txt exemption. The reasoning is consistent with the category, because a person did ask for it. The consequence for a site owner is that the growth area in automated traffic sits in the class that the file does not govern.

It is worth being precise about what this is not. Google-Agent is not documented as a training crawler and not documented as a search crawler, so a rule you write about it is not a rule about whether you appear in Google's answers. That question runs through Googlebot and through Google-Extended, which is a robots.txt product token that sends no user agent of its own. Keeping those separate is most of the work in reasoning about AI visibility on Google at all, and conflating them produces confident answers to the wrong question.

These tokens change, and the changelog is the only place the change is announced

A rule written against a user agent token is only as durable as the token, and three entries from the same changelog show how fast that moves.

On 16 July 2026 Google updated the NotebookLM user agent to Google-GeminiNotebook, because NotebookLM is now Gemini Notebook. The entry advises anyone who hardcoded the old value to update the string to avoid potential bugs, and says the old value will continue to be supported to allow a smooth transition. The fetchers page puts a date on that transition: it lists Google-NotebookLM as a former agent supported until August 2026, which from the day this post is published is next month.

On 14 July 2026 Google corrected the user agent string it publishes for Google-InspectionTool. The stated reason is that the documented string mistakenly included a semicolon which did not match the actual user agent string in production. A documented value that did not match the shipped value is the cleanest possible illustration of why a match on a vendor's published string is a fragile identity check, and it is a variation on the failure modes we set out in AI crawler detection is an ordering problem.

On 4 May 2026 Google added documentation on authenticating requests with Web Bot Auth, and gave as the reason that some Google user agents are now starting to use it. That is a first party statement of adoption for a scheme whose drafts we read on 29 July 2026 when we wrote up what Web Bot Auth actually specifies, and it is the direction this whole problem is heading: identity established by a signature on the request rather than by a string anyone can type.

The practical reading is that a token list is a perishable asset. Ours is published on our own bot page for exactly that reason, so that anyone receiving our requests can check what we claim against what we say we claim.

What a Lantad scan covers here, and what it does not

The inconvenient part of this post is the part about our own product, so it goes last and it goes plainly.

Lantad's crawler registry holds 15 tokens. Exactly one of them is Google's, Google-Extended, and it is not any of the fourteen agents described above. So a Lantad scan reports what your robots.txt says to fifteen named AI crawler tokens, and it says nothing at all about Google-Agent, Google-Safety, AdsBot-Google or any of the others, because they are not in the registry and no evaluation is run for them.

Adding them would not fix the deeper problem, and this is the honest limit rather than a roadmap note. Our parser reads a file and applies the group selection rules to a token. That produces a correct statement about what the file says to that token. It cannot produce a statement about what an agent documented as generally ignoring the file will actually do, because that outcome is decided in the fetcher and not in the file. Printing an allowed or blocked verdict for Google-Safety would be a confident wrong answer of exactly the kind we described when we explained why we will not grade a page we could not measure, and our methodology is written to keep those two claims apart.

What the file layer is still good for is everything it was always good for, and our robots.txt tester will show you which group each token lands in and which rule decided it. Just do not read a clean result there as a statement about total automated access, because a second layer sits above it: the argument that two layers decide whether AI can read your site applies here with the roles reversed, since your edge is the only place a fetcher that ignores your file can still be refused.

If you want the answer for your own site rather than in general, it is in your access logs, not in a scan. Search them for the nine tokens listed above, then check the source addresses against the three JSON files Google publishes. That is evidence about requests you actually received, which outranks anybody's summary of a documentation page, this one included.