Blog / Three AI crawler tokens that will never appear in your logs
Three AI crawler tokens that will never appear in your logs
Google-Extended, Applebot-Extended and anthropic-ai are robots.txt product tokens that send no user agent of their own, so a log search for them returns zero forever, and that zero does not mean what it looks like.
In short
- Lantad's bot registry holds 15 AI crawler tokens as of 28 July 2026, and 3 of them publish no request user agent at all: Google-Extended, Applebot-Extended and anthropic-ai.
- A log search for Google-Extended returns zero requests on every site, always, because Google enforces that token through ordinary Googlebot requests rather than through a crawler that identifies itself as Google-Extended.
- A dashboard listing Google-Extended at zero hits beside GPTBot at four hundred states something false about Google unless it also says the token cannot appear.
- The 12 remaining tokens in Lantad's registry do publish a request user agent, so they are the only ones a log file or an edge worker can observe directly.
- Blocking a robots-only token is still a real decision with real effect: it is read at crawl time from robots.txt, it is simply not observable afterwards in traffic.
There is a check that feels obvious and is wrong. You want to know whether Google is taking your content for Gemini training, so you search your access logs for Google-Extended. You find nothing. You conclude nothing is happening.
Nothing was ever going to be there. Google-Extended is a robots.txt product token, not a crawler. It has no user agent, sends no request, and appears in no log on any site in the world. The same is true of Applebot-Extended and of anthropic-ai. This post is about the difference between a token you can instruct and a crawler you can observe, and why conflating the two produces a confident wrong answer.
-
Google-ExtendedCannot appear A robots.txt token controlling Gemini training use. Enforced through ordinary Googlebot requests, which identify themselves as Googlebot. -
Applebot-ExtendedCannot appear A robots.txt token controlling Apple Intelligence training use. Enforced through ordinary Applebot requests. -
anthropic-aiCannot appear A legacy robots.txt token. Anthropic publishes no distinct request user agent for it. -
GPTBotObservable Publishes a request user agent, so it appears in logs and at the edge like any other client.
A robots.txt token and a crawler are different objects
robots.txt is addressed to a product token, not to a program. When a vendor wants to give you a switch for one use of your content without giving you a switch for all of it, they publish a token that stands for the use rather than for the fetcher.
Google-Extended is the clearest case. Google's crawler documentation lists it as a token controlling whether your content may be used for Gemini training and grounding, and states that it is not a crawler and fetches nothing. The fetching is done by Googlebot, which identifies itself as Googlebot because that is what it is. Disallowing Google-Extended changes what Google may do with a page it already fetched. It does not change whether the page is fetched, and it produces no distinct request to observe.
Applebot-Extended works the same way for Apple Intelligence, over ordinary Applebot requests. anthropic-ai is a legacy token Anthropic still honours and for which it publishes no separate request user agent.
Disallow: GPTBot
- GPTBot reads robots.txt
- GPTBot does not fetch the page
- Your log shows GPTBot hitting robots.txt
- Your log shows no GPTBot page requests
- The decision is visible in traffic
Disallow: Google-Extended
- Googlebot reads robots.txt
- Googlebot still fetches the page
- The page is still indexed for Search
- The content is withheld from Gemini training
- Nothing in your log ever says Google-Extended
Why the zero is worse than a missing row
A missing row invites a question. A zero answers one.
If a per-crawler table simply omitted the robots-only tokens, a reader would notice the absence and go looking. Printing them at zero, next to observable crawlers with real counts, tells the reader that the measurement was taken and the answer was none. For Google-Extended the true statement is that the measurement is impossible and the answer is unknown from this source.
This is why Lantad's own AI visibility page lists all 15 registry tokens rather than only the ones seen, and labels the three robots-only tokens as unable to appear rather than showing them at zero beside the others. An absence is a finding, but only if the page says which kind of absence it is.
- Is the token instructed in robots.txt Readable directly from your own robots.txt, for every token including the robots-only ones.
- Did a crawler claiming this token fetch a page Observable for the 12 tokens that publish a request user agent.
- Did the vendor honour a robots-only token Not observable from your side at all. There is no request to count and no response to inspect.
- Was the content used for training Not observable by anyone outside the vendor, for any token. No tool can measure this, including this one.
What to check instead
Three checks replace the one that does not work.
First, read your own robots.txt and confirm the token is instructed the way you intend. This is the only half of the question fully knowable from your side, and it takes one fetch.
Second, for the 12 observable tokens, look at your logs or your edge for requests and, more usefully, for their response codes. A crawler collecting 404s is a crawler that read nothing, and that is a real and fixable finding.
Third, treat any per-token count as a count of requests claiming that token. A user agent is a claim rather than an identity, which is a separate problem worth its own post.
Common questions
Does blocking Google-Extended stop Google crawling my site?
No. Google-Extended controls whether your content may be used for Gemini training and grounding. Googlebot still fetches the page and Search indexing is unaffected. That is the point of a separate token: it separates the use from the fetch.
Why does my analytics tool show Google-Extended at zero hits?
Because it can only ever be zero. Google-Extended sends no requests, so there is nothing to count. A tool showing it at zero without saying so is reporting an impossible measurement as a real one.
How can I tell whether a vendor is honouring a robots-only token?
You cannot, from your own side. There is no request to observe and no response to inspect. What you can verify is that your robots.txt states the instruction correctly, which is what Lantad checks.
How many AI crawler tokens should I be tracking?
Lantad's registry holds 15 as of 28 July 2026, of which 12 publish a request user agent. The registry is data rather than code, so adding a vendor token is a data change and every surface reading it updates together.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.