# Three AI crawler tokens that will never appear in your logs > Google-Extended, Applebot-Extended and anthropic-ai are robots.txt product tokens that send no user agent of their own, so a log search for them returns zero forever, and that zero does not mean what it looks like. - Canonical page: https://lantad.co/blog/the-crawler-tokens-that-never-appear-in-your-logs - This file: https://lantad.co/blog/the-crawler-tokens-that-never-appear-in-your-logs.md - Last substantive update: 2026-07-28 ## Key facts - **Published:** 2026-07-28 - **Category:** Guides - **Author:** Lantad - **Length:** 1010 words - **Takeaway 1:** Lantad's bot registry holds 15 AI crawler tokens as of 28 July 2026, and 3 of them publish no request user agent at all: Google-Extended, Applebot-Extended and anthropic-ai. - **Takeaway 2:** A log search for Google-Extended returns zero requests on every site, always, because Google enforces that token through ordinary Googlebot requests rather than through a crawler that identifies itself as Google-Extended. - **Takeaway 3:** A dashboard listing Google-Extended at zero hits beside GPTBot at four hundred states something false about Google unless it also says the token cannot appear. - **Takeaway 4:** The 12 remaining tokens in Lantad's registry do publish a request user agent, so they are the only ones a log file or an edge worker can observe directly. - **Takeaway 5:** Blocking a robots-only token is still a real decision with real effect: it is read at crawl time from robots.txt, it is simply not observable afterwards in traffic. ## Summary There is a check that feels obvious and is wrong. You want to know whether Google is taking your content for Gemini training, so you search your access logs for Google-Extended. You find nothing. You conclude nothing is happening. Nothing was ever going to be there. Google-Extended is a robots.txt product token, not a crawler. It has no user agent, sends no request, and appears in no log on any site in the world. The same is true of Applebot-Extended and of anthropic-ai. This post is about the difference between a token you can instruct and a crawler you can observe, and why conflating the two produces a confident wrong answer. ## A robots.txt token and a crawler are different objects robots.txt is addressed to a product token, not to a program. When a vendor wants to give you a switch for one use of your content without giving you a switch for all of it, they publish a token that stands for the use rather than for the fetcher. Google-Extended is the clearest case. Google's crawler documentation lists it as a token controlling whether your content may be used for Gemini training and grounding, and states that it is not a crawler and fetches nothing. The fetching is done by Googlebot, which identifies itself as Googlebot because that is what it is. Disallowing Google-Extended changes what Google may do with a page it already fetched. It does not change whether the page is fetched, and it produces no distinct request to observe. Applebot-Extended works the same way for Apple Intelligence, over ordinary Applebot requests. anthropic-ai is a legacy token Anthropic still honours and for which it publishes no separate request user agent. ## Why the zero is worse than a missing row A missing row invites a question. A zero answers one. If a per-crawler table simply omitted the robots-only tokens, a reader would notice the absence and go looking. Printing them at zero, next to observable crawlers with real counts, tells the reader that the measurement was taken and the answer was none. For Google-Extended the true statement is that the measurement is impossible and the answer is unknown from this source. This is why Lantad's own AI visibility page lists all 15 registry tokens rather than only the ones seen, and labels the three robots-only tokens as unable to appear rather than showing them at zero beside the others. An absence is a finding, but only if the page says which kind of absence it is. ## What to check instead Three checks replace the one that does not work. First, read your own robots.txt and confirm the token is instructed the way you intend. This is the only half of the question fully knowable from your side, and it takes one fetch. Second, for the 12 observable tokens, look at your logs or your edge for requests and, more usefully, for their response codes. A crawler collecting 404s is a crawler that read nothing, and that is a real and fixable finding. Third, treat any per-token count as a count of requests claiming that token. A user agent is a claim rather than an identity, which is a separate problem worth its own post. ## Questions and answers **Does blocking Google-Extended stop Google crawling my site?** No. Google-Extended controls whether your content may be used for Gemini training and grounding. Googlebot still fetches the page and Search indexing is unaffected. That is the point of a separate token: it separates the use from the fetch. **Why does my analytics tool show Google-Extended at zero hits?** Because it can only ever be zero. Google-Extended sends no requests, so there is nothing to count. A tool showing it at zero without saying so is reporting an impossible measurement as a real one. **How can I tell whether a vendor is honouring a robots-only token?** You cannot, from your own side. There is no request to observe and no response to inspect. What you can verify is that your robots.txt states the instruction correctly, which is what Lantad checks. **How many AI crawler tokens should I be tracking?** Lantad's registry holds 15 as of 28 July 2026, of which 12 publish a request user agent. The registry is data rather than code, so adding a vendor token is a data change and every surface reading it updates together. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot