A renamed crawler token leaves your robots.txt group matching nothing

What Google changed on 16 July 2026

Google's crawling documentation changelog carries an entry dated 16 July 2026 headed Updating the NotebookLM user agent. Its reason is one line: NotebookLM is now Gemini Notebook. The token moved with the product, and the entry records that the old value remains supported during the transition.

The fetcher's own page is more specific about how long that transition lasts. Read on 5 August 2026 and carrying Last updated 2026-07-16 UTC, it gives the current token as Google-GeminiNotebook, publishes two full request user agent strings using it, one mobile and one desktop, and names the former agent as Google-NotebookLM, supported until August 2026. There is no day of the month attached to that deadline, which is worth noticing rather than resolving: the honest reading is that the old name stops being guaranteed at some point in the month this post was written, and Google has not said which point.

Two other dates put the change in proportion. The same changelog records Google-NotebookLM being added to the list of user-triggered fetchers on 9 October 2025, so the retired name was the current one for roughly nine months. And this is not the only rename on that page. Google's list of user-triggered fetchers names nine fetchers, and Google Read Aloud carries the same footnote pattern: its token is Google-Read-Aloud and its former agent, google-speakr, is marked deprecated. Two of nine entries on one page therefore carry a superseded name, which makes renaming a normal maintenance event rather than an exception.

None of this is hidden. It is published, dated and linked, and Google's crawler documentation is among the better vendor documentation in this area. The problem is not disclosure. The problem is that a robots.txt file has no way to consume a disclosure. It is a static text file matched by string, and the string it matches against was updated on a page nobody has an obligation to reread. That gap is the whole subject, and it applies well beyond this one fetcher: the same three-class Google crawler taxonomy covers common crawlers, special-case crawlers and user-triggered fetchers, and the tokens in all three classes are vendor property. If you keep a list of names to block or allow, whether by hand or in a tool that enumerates AI crawlers, that list is a cache of somebody else's documentation and it has no invalidation policy.

Why a group headed by a retired token matches nothing

The robots exclusion standard is precise about how a crawler picks its group, and the precision is what makes the failure quiet. RFC 9309 states that crawlers must use case-insensitive matching to find the group that matches the product token and then obey the rules of that group, and that where more than one group matches, the matching groups' rules must be combined into one group. Matching is on the name. There is no version, no alias table and no redirect.

The standard then says what happens when nothing matches, and it says it in two steps that are easy to run together and should not be. First: if no matching group exists, crawlers must obey the group with a user-agent line with the star value, if present. Second: if no group matches the product token and there is no star group, or no groups are present at all, no rules apply.

Read those in order against a file whose author wrote a group for Google-NotebookLM and nothing else about Google. Once the old name retires, that group matches no product token any crawler presents. If the file has a star group, the fetcher is now governed by whatever the star group says, which for most sites is a general allow. If the file has no star group, the second sentence applies and no rules apply at all. Either way the outcome is decided by a part of the file the author was not thinking about when they wrote the rule, and neither outcome is an error.

This is the same class of behaviour as the status code semantics we have written about before, where a 404 on robots.txt allows every crawler and a 503 blocks them all. In both cases the specification defines a default for an absence, the default is reasonable in isolation, and the site owner never chose it. The difference is that a status code is at least observable from outside: you can request the file and see what came back. A group that no longer matches is invisible to that request, because the file itself is unchanged and still parses.

Our own parser follows the documented behaviour with the refinement crawlers actually implement, and the rule is written down in the header comment of core/src/robots.ts: a crawler picks the most specific matching group, meaning an exact token match or the longest group token that is a prefix of the bot token, case insensitive, falling back to star, and all groups carrying that winning token are merged. That is a design decision we took to match observed crawler behaviour rather than a finding about the web, and it has a consequence worth stating plainly. Under a prefix rule, a group headed Google-NotebookLM does not match a bot calling itself Google-GeminiNotebook, because the first is not a prefix of the second. A group headed simply Google would match both, and would also match a great deal else. If you want to know what your file does for a given name today, the direct answer comes from running the file and the token through a parser rather than from reading it, and we publish how ours evaluates rather than asking anyone to take it on trust.

Nothing on your site reports that a rule stopped applying

Consider what would have to happen for a site owner to find out. The file has not changed, so a content diff shows nothing. The syntax is still valid, so a linter passes. The group is still there in the text, so anyone rereading the file sees a rule that looks exactly as it did when it worked. The crawler does not send an error, because from its side nothing went wrong: it looked for its own name, did not find it, and followed the standard.

Server logs do not close the gap either, and for a reason that is structural rather than a matter of tooling. A log line records what arrived, not what a rule intended. If the fetcher was being blocked and now is not, the evidence is an increase in requests carrying a user agent string you were not watching for, buried in traffic from a name that did not exist in your dashboards last month. Absence of a signal is what a working rule and a silently dead rule both look like, which is the same trap as three crawler tokens that will never appear in your logs, where a zero means the token was never a user agent in the first place.

Timing makes it worse rather than better. Even a deliberate edit does not take effect when you save it: crawlers hold cached copies, and the controlled experiment we wrote up found fewer than 40 percent of AI assistant and AI search bots refetching robots.txt inside a 168 hour window. A rename inverts that latency. The vendor's change lands whenever the crawler updates its own identity, and your file, which is the thing that would have to change to keep up, changes only when a human decides to look.

There is one more layer of separation between a written rule and an outcome, and it is the one most likely to surprise. A robots.txt rule is a request, not an enforcement mechanism, which is why 234 of 592 sites that ban GPTBot served it a 200 anyway when a request carrying that name actually knocked. So a site can have a rule that no longer matches, a rule that matches and is ignored, and a rule that matches and is honoured, and all three look identical in the file. The only way to tell them apart is to send a request and read the response, which is what fetching a page as a named crawler is for.

The narrower lesson is that robots.txt cannot express anything conditional. Its vocabulary is names and paths, and it has no way to say this rule was written on this date against this vendor's then-current token. Extensions that try to add meaning face the reverse problem, as with Cloudflare's Content-signal line, which our parser records as an unknown field and ignores, because an unrecognised field is discarded by design. A file that discards what it does not recognise and defaults quietly when it recognises nothing is a file that will never tell you it has stopped working.

What this particular rename costs, which is less than it looks

The inconvenient detail belongs here rather than in a footnote, because it changes what a reader should do. Gemini Notebook is a user-triggered fetcher, and Google documents that class as generally ignoring robots.txt rules, on the reasoning that the fetch is made because an end user asked for it. That is stated on the Gemini Notebook page itself, and it is the same property that made fourteen Google agents not decided by the group you head with an asterisk worth writing about in the first place.

So a group headed Google-NotebookLM with Disallow lines under it was not enforcing much before 16 July 2026. It expressed a preference to a fetcher documented as generally not consulting the file. After the rename it expresses that preference to nobody at all. The practical loss is close to zero, and saying otherwise would be manufacturing an emergency out of a documentation update.

What the case is genuinely good for is showing the mechanism cheaply, on a fetcher where the stakes are low, before it happens to a token where they are not. Nothing about the rename mechanism is specific to user-triggered fetchers. A common crawler that does respect robots.txt for automatic crawls can be renamed the same way, on the same kind of changelog entry, with the same transition window, and the consequence would be a genuine block quietly turning into an allow. Google-Agent, added on 20 March 2026 for agents running on Google infrastructure, is a reminder that the list grows as well as churns.

There is a second reason the loss is smaller than the headline suggests, and it is one this blog has argued before. A name in a request is a claim rather than an identity: any client can send any user agent string, so a rule keyed to a name was always a request addressed to whoever chooses to answer to that name. A rename is that same weakness surfacing from the other direction, where the party who owns the name stops using it. Both point at the same conclusion, which is that AI visibility questions are settled by what a request actually receives, and a file describing intentions is evidence about the author rather than about the outcome.

Every crawler registry goes stale, including ours

A tool that evaluates robots.txt per crawler has to hold a list of names, and that list is subject to exactly the decay described above. Ours is core/src/bots.ts, it holds 15 robots.txt product tokens, and the file's own header says the design goal is that adding a bot requires no logic changes anywhere else. The 15 are GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI, ClaudeBot, Claude-SearchBot, Claude-User and anthropic-ai for Anthropic, PerplexityBot and Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot.

Neither Google-NotebookLM nor Google-GeminiNotebook is in that list, and that is a scoping decision rather than an oversight: the registry models crawlers whose behaviour a site owner can influence through robots.txt, and we have already published the audit of what that scoping leaves out, including the one crawler our scanner does not model. Being out of scope is not the same as being irrelevant, and it is recorded rather than implied.

The entry that proves the point about decay is anthropic-ai. It sits in our registry with a source comment describing it as a legacy robots token for which Anthropic publishes no distinct request user agent, which is why it belongs in the group of tokens that produce no log lines. Read on 5 August 2026, Anthropic's page on how site owners can block its crawler, dated 7 April 2026, names three agents and only three: ClaudeBot for training data, Claude-User for user-initiated requests, and Claude-SearchBot for search quality. The string anthropic-ai does not appear on it. We keep the token because sites still carry rules naming it and because reporting on a rule a site actually has is more useful than reporting on the rule we wish it had, but the honest description is that our registry carries a name its vendor's current documentation no longer mentions. A registry is a cache, and ours has a stale entry in it by choice.

That is also why the useful vendor-side signal is how many names a vendor publishes at all, which we counted when we found six of the nine AI vendors we track publish exactly one crawler token. A vendor with one name gives you one string to get wrong. A vendor with four gives you four, and four opportunities for one of them to be superseded on a changelog you do not read.

For a site owner, the check that follows from all of this is small and worth doing on a schedule rather than after an incident. Open your robots.txt and list every product token it names. For each one, open the vendor's own crawler page and confirm the string still appears there, because a name that has quietly moved is the failure this post is about. Then confirm the file has a star group, since without one an unmatched crawler falls to no rules at all. Then, for the tokens that matter to you, send a request under the current name and read what comes back, because the file is a statement of intent and the response is the outcome. Our own crawler publishes what it is and where it comes from so that the same check can be run against us.