# How to get cited by Gemini: one robots.txt token covers training and grounding > Google documents 11 common crawlers and none of them is a Gemini crawler. The only entry that names Gemini is Google-Extended, which sends no request of its own, and Google's page of 14 July 2026 says the same token governs training and grounding together. - Canonical page: https://lantad.co/blog/how-to-get-cited-by-gemini-one-token-two-systems - This file: https://lantad.co/blog/how-to-get-cited-by-gemini-one-token-two-systems.md - Last substantive update: 2026-09-08 ## Key facts - **Published:** 2026-09-08 - **Category:** Findings - **Author:** Lantad - **Length:** 3420 words - **Takeaway 1:** How to get cited by Gemini starts with a fact that removes most of the usual advice: Google's common crawlers page, fetched on 8 September 2026 and carrying a last updated date of 14 July 2026, documents 11 crawlers and fetchers and none of them is a Gemini crawler. - **Takeaway 2:** Google-Extended is the only one of those 11 entries that names Gemini, and Google's page states it does not have a separate HTTP request user agent string, so no site can see it arrive and no log can count it. - **Takeaway 3:** Google's sentence puts two systems inside one token: Google-Extended manages whether crawled content may be used for training future Gemini models and for grounding, which the same sentence defines as providing content from the Google Search index to the model at prompt time. - **Takeaway 4:** Of seven Google documentation pages Lantad requested on 8 September 2026, one contains the word Gemini. The AI features page, last updated 10 December 2025, names AI Mode 11 times and AI Overviews 14 times and Gemini zero times. - **Takeaway 5:** Lantad's own bot registry files Google-Extended under the purpose training, and the registry has no value for grounding, so this scanner describes the token more narrowly than Google's documentation does. ## Summary There is no Gemini crawler. That is the first thing to settle, because advice on how to get cited by Gemini usually opens by telling you which bot to allow, and Google publishes no bot to allow. On 8 September 2026 we requested Google's page of common crawlers with the scanner's own user agent and redirects followed. It answered HTTP 200, carries a last updated date of 14 July 2026, and documents 11 crawlers and fetchers. Nine of them ship a Mozilla/5.0 user agent string you could match in an access log. Two do not, and one of those two is the only entry on the page that mentions Gemini at all. That entry is Google-Extended, and what it governs is wider than the training opt out it is usually called. Everything below about Google's behaviour is read from Google's published documentation, with the date each page carries, and none of it is a measurement of what Gemini does with a page: this site has not measured Gemini's answers and does not model a Gemini crawler, for the plain reason that Google documents none. What we can add sits beside the documentation rather than inside it: which control removes what, what the [AI crawler directory](https://lantad.co/tools/ai-crawlers) here does with the single Google token, and one thing our own registry states more narrowly than the source does. ## How to get cited by Gemini when Google documents no Gemini crawler [The page Google publishes for site owners](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) who want to know which of its clients will arrive lists 11 entries: Googlebot, Googlebot Image, Googlebot Video, Googlebot News, Google StoreBot, Google-InspectionTool, GoogleOther, GoogleOther-Image, GoogleOther-Video, Google-CloudVertexBot and Google-Extended. We requested it on 8 September 2026 and read the prose out of the response. Nine entries carry a user agent string beginning Mozilla/5.0. Two state that they have no separate HTTP request user agent string, and those two are Googlebot News and Google-Extended. The practical shape of that list matters more than its length. A crawler with a user agent is a thing you can observe: it arrives, your server writes a line, and you can count it. A robots.txt product token with no user agent is a thing you can only instruct. You can write a group for it, and you will never see the result of that group in your own logs, because the fetch that the rule governs is made by an ordinary Google client under an ordinary Google user agent. We have written before about [the crawler tokens that never appear in your logs](https://lantad.co/blog/the-crawler-tokens-that-never-appear-in-your-logs), and Google-Extended is the Google member of that set. One detail is worth recording because it will break links elsewhere. Google's crawler documentation has moved out of Search Central. A request to the old address under /search/docs/crawling-indexing/ returned HTTP 301 on 8 September 2026 with a Location header pointing at /crawling/docs/crawlers-fetchers/, for both the overview page and the common crawlers page. The content is the same, the address is not, and anything that stored the old URL now depends on a redirect being kept. That includes this repository, which is the subject of the last section. Against that background, the question that sends people here has a structural answer rather than a tactical one. You do not get cited by Gemini by admitting a Gemini crawler, because [an AI crawler](https://lantad.co/glossary/ai-crawler) named for Gemini does not exist in Google's documentation. You get there through the same index that serves Search, under the same rules, with one extra token that can take you out. The wider set of Google clients, and what a robots.txt wildcard does and does not reach among them, is covered in the post on [Google fetchers and the wildcard group](https://lantad.co/blog/google-fetchers-ignore-the-robots-txt-wildcard). ## What Google-Extended switches off, in Google's own words The four occurrences of the word Gemini on that page all sit inside a single sentence. Google writes that Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding, and then defines grounding in the same breath as providing content from the Google Search index to the model at prompt time to improve factuality and relevancy, in Gemini Apps and Grounding with Google Search on Vertex AI. Read that as a control surface rather than as a paragraph. Two different systems are named. One is training, which happens once, offline, and produces a model. The other is grounding, which happens per prompt, at answer time, and is the mechanism by which a page you published can be quoted in a reply to a question you never saw. The same token governs both, and Google publishes no way to separate them. That is not a gap Google could easily close inside the format. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), which standardised robots.txt in September 2022, gives a group one product token and a set of allow and disallow rules matched against the URI path. There is no field in that grammar for a purpose. A group either applies to the token or it does not, so a file cannot say yes to grounding and no to training when one token carries both. If you want to see how a specific file resolves for a specific token, that is what the [robots.txt tester](https://lantad.co/tools/robots-txt-tester) here does, though it will tell you what the file says rather than what any crawler then did. The same page settles the question people usually ask next. Google states that Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. So the cost of disallowing it is not a Search cost. On the evidence published so far the cost lands on the AI surfaces instead, which is the finding in the post on how [AI Overviews retrieved less from sites blocking Google-Extended](https://lantad.co/blog/ai-overviews-retrieved-less-from-google-extended-blockers). ## The grounding path runs through the Google Search index Follow Google's own definition and the retrieval question answers itself. If grounding means providing content from the Google Search index to the model at prompt time, then the corpus a grounded Gemini answer draws on is the Search index. There is no second index built by a Gemini crawler, because there is no Gemini crawler. Whatever gets a page into Search is what makes it reachable, and whatever removes a page from Search removes it here too. [Google's AI features page](https://developers.google.com/search/docs/appearance/ai-features), which we requested on the same day and which carries a last updated date of 10 December 2025, states the eligibility rule for the Search surfaces in one line: to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet. It adds that there are no additional requirements to appear in AI Overviews or AI Mode, and no other special optimizations necessary. That page is about AI Overviews and AI Mode rather than the Gemini app, but it is the only place Google states a hard technical gate for a generative surface, and the snippet controls it names are documented for the whole of Search. Those controls are the ones that actually cost you something, and they are set on the page rather than in robots.txt. [Google's robots meta tag documentation](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) covers nosnippet, data-nosnippet, max-snippet and noindex. Each of them narrows what Search may display, and by the eligibility rule above, each narrows the generative surfaces along with it. When this site requested 120 home pages on 6 September 2026, none of the 81 that answered HTTP 200 carried a nosnippet directive, which is the measurement behind [zero of 81 home pages blocking their own snippet](https://lantad.co/blog/how-to-get-content-cited-in-google-ai-overviews-zero-nosnippet). Most sites are not opted out of anything. They have simply never made the choice. The distinction that trips people is between a control on your site and a control somewhere else. A snippet directive is yours: you write it, a crawler reads it, and the effect follows from a file you serve. Other controls in this area are not, which is the argument in [where the AI Overviews opt out actually lives](https://lantad.co/blog/ai-overviews-opt-out-does-not-live-on-your-site), and the per-platform version for Google's answer surfaces is on the [Google AI Overviews page](https://lantad.co/how-to-get-cited/google-ai-overviews). ## Seven Google documentation pages, counted on 8 September 2026 We asked a narrow question of Google's own documentation set: if a site owner goes looking for the word Gemini in the pages Google points them at, where do they find it? So we requested seven pages on 8 September 2026, one GET each as LantadBot/1.0 with an Accept-Language header of en and redirects followed, stripped the markup, and counted literal strings in the visible text. One of the seven contains the word Gemini, and it is the crawler page. The AI features page names AI Mode 11 times and AI Overviews 14 times and Gemini zero times. The generative AI optimization guide, last updated 10 July 2026, names Gemini zero times. So does the robots meta tag page, the robots.txt introduction, the structured data introduction and the Search Essentials page. A publisher who wants to understand the Gemini app from Google's site owner documentation is reading pages that do not mention it. Two caveats, because both matter for anyone repeating this. The Accept-Language header is load bearing: an earlier attempt without it was redirected to Russian, German and Thai localisations of the same pages, which is the behaviour described in the post on how [Googlebot sets no Accept-Language](https://lantad.co/blog/googlebot-sets-no-accept-language). And two of the seven pages returned HTTP 500 on one attempt and HTTP 200 on a retry minutes later, so the counts below are from the successful fetches and the 500s are noise rather than a finding. A single GET is a single GET, which is the general limit on this kind of reading and the reason [our methodology](https://lantad.co/methodology) states what a capture is before it states what it means. What the count supports is modest and worth saying plainly. It is not evidence that Google is hiding anything, and the word count of a page is not a measure of a product. It is evidence about where the answer lives: the only Google page for site owners that connects the Gemini name to a control a publisher can write is the page listing crawlers, and that entry describes a token rather than a client. ## What the Gemini API shows a developer, and what a publisher never sees There is one place the Gemini side of this is legible, and it is not a place a site owner can reach. Google's Gemini API documents a Grounding with Google Search tool, and the page for it at ai.google.dev/gemini-api/docs/google-search, which we requested on 8 September 2026 and which carries a last updated date of 2 September 2026, describes a response that names the search queries the model ran and the URLs it cited. The fields it names are google_search_call, which the page says contains the queries the model executed, google_search_result, which carries a search_suggestions HTML snippet for the caller to render, and url_citation annotations, each linking a segment of the answer to a source URL. Notice who that information is for. The developer holding the API key sees which queries ran and which pages were cited. The publisher whose page was one of those citations sees nothing at all: no request they can attribute, no header, no entry in a log, because the retrieval happened against an index rather than against their server. That asymmetry is the whole reason a category of measurement tools exists, and it is also the reason a citation count from any of them is a count of what one client asked, not of what the engine did. We set out that boundary when we shipped our own Gemini reader, in the post on why [a Gemini citation tool is not an AI Overview tool](https://lantad.co/blog/a-gemini-citation-tool-is-not-an-ai-overview-tool), and the same caution applies to every number in this space, including ours. That is the honest floor under [AI visibility](https://lantad.co/glossary/ai-visibility) as a practice, and the reason [generative engine optimisation](https://lantad.co/glossary/geo) is harder to verify than ranking ever was. The comparison with other engines is instructive rather than decorative. Anthropic publishes three named bots and an IP range file, which is the subject of [how to get cited by Claude](https://lantad.co/blog/how-to-get-cited-by-claude-one-ip-list-three-bots). Perplexity publishes two named crawlers, covered on the [Perplexity page](https://lantad.co/how-to-get-cited/perplexity), and OpenAI publishes several, covered on the [ChatGPT page](https://lantad.co/how-to-get-cited/chatgpt). Google publishes none for Gemini, and gives you one token instead. That is not worse by itself. It is different in a way that changes what you can check: with a named crawler you can look in your own logs, and with a token you can only read the file back and trust the operator. ## Our own registry describes this token more narrowly than Google does Two things in this repository are narrower than the documentation they came from, and both are visible to anyone who reads the source. The bot registry holds 15 AI crawler tokens and gives each one a purpose drawn from a list of exactly three values: training, search and user_agent. Google-Extended is filed under training. Google's page says the token governs training and grounding, and there is no grounding value in our list, so the label this scanner prints is a fair description of half of what the token does. That is a real gap, it is ours, and stating it here is cheaper than letting a reader find it in the code. The second is smaller and will bite sooner. The registry entry for Google-Extended stores a documentation URL under the old Search Central path, and that path now answers HTTP 301. The link still resolves because Google is redirecting it, which is exactly the condition under which stale references survive unnoticed until the day they do not. Both of these are recorded rather than fixed in this post, because a post is not the place to change a scanner and a change to the registry belongs with its tests. None of that alters what a reader should do, and the actions are unglamorous because the mechanism is. Decide whether you want to be quoted in Gemini answers at all, since the same token that refuses training refuses grounding. If the answer is yes, leave Google-Extended alone and spend the effort on being indexable and snippet eligible, because Google's stated gate for its generative surfaces is exactly that. If the answer is no, write the group and accept that both systems go together. You can see the crawler side of your own site with [what GPTBot sees](https://lantad.co/tools/what-gptbot-sees), and the identification this scanner sends is published at [our bot page](https://lantad.co/bot). The last point is the one this site keeps returning to. Nothing above measures Gemini. It measures documents about Gemini, on a stated date, with the fetch described. What an engine emits and what an engine did are different claims, and the gap between them is where confident wrong answers get made, which is why the [research page](https://lantad.co/research) publishes sample sizes before it publishes conclusions and why [structured data](https://lantad.co/glossary/structured-data) work here is scored against what a parser can extract rather than against what a model might appreciate. ## Questions and answers **Is there a Gemini crawler I can allow in robots.txt?** No. Google's common crawlers page, read on 8 September 2026 and last updated 14 July 2026, documents 11 crawlers and fetchers and none of them is named for Gemini. The only entry that mentions Gemini is Google-Extended, which the same page says has no separate HTTP request user agent string, so it is a control token rather than a client that visits your site. **Does disallowing Google-Extended remove my site from Google Search?** No. Google's page states that Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. What it does affect, by the same sentence, is training of future Gemini models and grounding, which Google defines as providing content from the Google Search index to the model at prompt time. **Can I refuse training but still allow grounding?** Not through Google-Extended, and Google publishes no second token for it. One robots.txt group carries one product token, and RFC 9309 gives that group no field for a purpose, so a file cannot distinguish the two uses that Google documents under the one name. Anyone who needs that distinction should say so to Google rather than expect a file to express it. **Where does a grounded Gemini answer get the page it cites?** From the Google Search index, per Google's own definition of grounding on the common crawlers page: providing content from the Google Search index to the model at prompt time to improve factuality and relevancy. That means the reachability question is the ordinary indexing question, and the eligibility rule Google publishes for its Search generative surfaces is that a page must be indexed and eligible to be shown with a snippet. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot