Googlebot sets no Accept-Language, so your site picks the locale for it

What a locale-adaptive page does with a request that names no language

Google's term for this class of page is locale-adaptive, and its definition is worth taking literally: pages that return different content based on the perceived country or preferred language of the visitor. The important word is perceived. Nothing in an HTTP request states who the visitor is. A server infers a country from an IP address and a language from a header, and both inferences can be wrong without anything appearing to fail.

The header is defined in RFC 9110, the current HTTP semantics specification. Section 12.5.4 says the Accept-Language field can be used by user agents to indicate the set of natural languages that are preferred in the response. It is a preference, expressed by the client, and it is optional. The same section settles the case that matters here: if no Accept-Language field is received in a request, the server MAY assume that all languages are equally acceptable. A server answering an unadorned request with its default is therefore not doing anything wrong. The specification permits it in as many words. Whatever your site shows a caller who expressed no opinion is the conformant answer, and on most sites that default was chosen for commercial reasons years before anyone was thinking about machine readers.

Two neighbouring definitions in the same document complete the picture. Section 8.5 defines Content-Language as describing the natural languages intended for the audience of the representation, which is the server stating what it has just sent rather than the client asking for anything. Section 12.5.5 defines Vary, which describes what parts of a request message, aside from the method, host and request target, might play a role in selecting the response representation. Vary is the honest way to admit that a URL is not the whole address of a document. A page that genuinely varies by Accept-Language and does not say so in Vary is misinforming every cache between it and its readers, and the symptom is one visitor being served another visitor's language.

The structural point is not really about language. A locale-adaptive page is a page whose bytes are a function of the request as well as of the URL. Nearly everything downstream assumes the opposite, because a citation assumes the opposite: a link is an address, and an address is expected to resolve to a document. When it instead resolves to a document chosen by the caller's apparent nationality, the question changes from what does this page say to what does this page say to whom, and only the first of those has an answer that fits in a report. That is the same failure shape as prose parity, where the text a browser renders and the text a crawler receives diverge for a reason that has nothing to do with what the page is about.

What Google publishes about how Googlebot fetches a locale-adaptive page

Google's page on crawling locale-adaptive pages carries Last updated 2025-12-10 UTC, and it states two things about its own crawler that almost nothing in this field states about any crawler.

The first is about headers. Google's locale-adaptive pages documentation says the crawler sends HTTP requests without setting Accept-Language in the request header. Not usually, not by default with exceptions: the documented behaviour is that the header is absent. Read alongside the specification, that places Googlebot in precisely the position RFC 9110 describes, a client that has stated no preference, and it places your server in the position of choosing on its behalf.

The second is about location. The same page says the default IP addresses of the Googlebot crawler appear to be based in the USA, and adds that Googlebot crawls with IP addresses based outside the USA in addition to the US-based ones. Both halves carry weight. A site inferring a country from the connecting address will usually infer the United States and sometimes will not, which means a locale-adaptive site can be crawled inconsistently over time without changing anything about itself. Google's guidance for that case is behavioural rather than technical: when Googlebot appears to come from a certain country, treat it as you would treat any other user from that country, and block or allow access consistently on that basis. The same page states that robots meta tags and robots.txt must specify the same rules in each locale, which is worth reading twice by anyone serving a different robots.txt to different regions.

Taken together, those statements describe a crawler that has deliberately given up the ability to ask for a language. That is a design decision rather than an oversight, and it follows from what a fetcher is. An AI crawler building an index cannot represent a reader, because at fetch time there is no reader to represent. It also fits the wider pattern that documented crawler limits are narrower than most people assume. The analogous case for size is already on this site: Googlebot reads the first 2MB of a supported file type and stops there, another published constraint that most site owners have never read.

One caution applies to every claim in this section, and to our own work equally. What a request carries is a claim, not a credential, and any client can send any header it likes. That is the argument in a user agent is a claim, not an identity, and it applies to Accept-Language as squarely as it applies to a product token. A documented behaviour is what a vendor says its fleet does. Confirming that a particular request really came from that fleet is a separate exercise, and it runs on published IP ranges rather than on anything in the request headers.

What the AI crawler documentation says about language, which is nothing

The comparison this site can honestly make is a documentary one, and it is short because there is nothing in the documents to report. Three vendor pages, each read on 13 August 2026.

OpenAI's crawler overview names four agents: OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User. It sets out what each is for and how to control it from robots.txt. It says nothing about language, locale, region, Accept-Language, or the geographic origin of its requests, and the page carries no visible update date. Anthropic's crawler article names three: ClaudeBot for training data collection, Claude-User for user-initiated web access, and Claude-SearchBot. It points site owners at a published file of source IP addresses so a request can be attributed to Anthropic, which is a statement about attribution rather than geography: it tells you a request came from Anthropic, not which country it left from. Language is not mentioned. Perplexity's crawler page names two, PerplexityBot and Perplexity-User, and notes that changes may take up to 24 hours to be reflected in its systems. It says nothing about language or locale either.

Nine tokens across three vendors, and not one published sentence about how any of them treats a multilingual site. This is not an accusation of bad faith. Crawler documentation is written to answer the question site owners actually ask, which is how to allow or block the thing, and language has not yet become a question people ask. It is worth naming precisely because an absence is so easy to fill with an assumption. If you have quietly assumed that a fetch triggered by a French speaker's question carries a French Accept-Language, nothing published supports that. Nothing published contradicts it either. The defensible position is that it is undocumented, and an undocumented behaviour is one you should not design around.

The shape recurs across this corpus, which is the reason to treat it as a pattern rather than a gap in one vendor's writing. Six of the nine vendors we track publish exactly one crawler token, which is fewer than the standard advice to allow search and block training requires anyone to write a working rule. Our reference list of the agents and what each is documented to do sits at the AI crawler directory, and it records what vendors publish rather than what we infer from behaviour. Where a vendor is silent, the entry is silent, which is the only defensible way to keep a list of that kind.

Hreflang annotates separate URLs, it does not switch a page's language

The advice Google gives for multilingual sites is not to detect and adapt. It is to publish separate addresses and annotate the relationship between them. The locale-adaptive page says so directly, recommending separate locale URL configurations annotated with rel=alternate hreflang annotations, and a dedicated page explains the annotation.

Google's localized versions documentation, carrying Last updated 2025-12-22 UTC, lists exactly three ways to declare an alternate version: HTML link elements carrying rel=alternate and an hreflang value, Link headers returned on the GET response, and xhtml:link entries inside an XML sitemap. Three delivery mechanisms, one annotation. Notice what they have in common: every one of them names a URL. The annotation does not switch a page's language. It tells a search engine that a different address holds the same content in another language, and it leaves both addresses independently fetchable. The sitemap form inherits the usual caveat about anything a site asserts about itself, which is the argument in a sitemap telling a crawler where pages are rather than that they changed.

The same page carries a sentence that surprises people who have treated hreflang as a language declaration: Google doesn't use hreflang or the HTML lang attribute to detect the language of a page, and instead uses algorithms to determine the language. The annotation is a relationship between documents, not a label on one of them. A page does not become Portuguese because somebody wrote a Portuguese hreflang value on it, and a page mislabelled that way is not corrected by the label either.

Google's multi-regional and multilingual guidance, also carrying Last updated 2025-12-10 UTC, then closes the loop with two instructions that read as warnings. Avoid automatically redirecting users from one language version of a site to a different language version of a site. And do not use IP analysis to adapt your content, because IP location analysis is difficult and generally not reliable. The same page lists the signals Google says it does use for country targeting: country-code top-level domains, hreflang statements in tags, headers or sitemaps, server location through the IP address of the server, and other local signals such as addresses and phone numbers on the pages, local language and currency, and links from other local sites.

Read as one instruction the advice is coherent: give each locale a stable address, state the relationship between the addresses, and stop trying to guess who is calling. The reason to care beyond Google is that this is the only arrangement an undocumented crawler cannot get wrong. A fetcher with no Accept-Language header and a United States IP address still receives French from a French URL, because that URL is an address rather than a negotiation. For generative engine optimisation the point is sharper still, since the unit of citation is a URL and the engine doing the citing has told you nothing about itself.

What to check on a multilingual site this week

None of this needs a crawler log to act on, which is just as well, because most sites do not keep one worth reading. Four checks follow from the documentation above, and each has a failure mode that produces no error anywhere.

Find out whether your site varies at all. Request your home page twice, once with no Accept-Language header and once with the header set to a language you publish, and compare the two responses. If they differ, you have a locale-adaptive page and everything in this post applies to you. If they are byte for byte the same, language is settled by the URL and you can stop at the first check. The request with no header is the one that matters most, because it is the case Google documents its own crawler producing and the case the specification permits a server to answer with anything at all.

Check what a crawler-shaped request receives, separately from what your browser receives. A browser sends a long Accept-Language list, a set of client hints and a cookie jar. A crawler sends a user agent and frequently little else. Lantad's own per-bot probe is built that way as a matter of configuration rather than discovery: it re-requests the final URL with a single user-agent header carrying the bot's published string, and no Accept-Language. That is a setting in our pipeline, not a finding about anybody's crawler, but it does mean the response we grade is the one a header-light client receives. You can run the equivalent against a single page with what GPTBot sees, and the way the resulting response is scored is set out in our methodology.

Look for a redirect before you look for a translation. An IP-based redirect from the root to a country path is invisible to anyone browsing from the country it targets and highly visible to a crawler sitting in a data centre somewhere else. Google's guidance is to avoid the pattern outright, and the reason to take that seriously has little to do with rankings: a redirect is a place where a request can be answered by something other than your application, which is the general case that two layers decide if AI can read your site sets out.

Then decide what your default should be, and write the decision down somewhere. Somebody chose it once, probably in the first week of the site's life, and on a locale-adaptive site that choice is now the answer given to every client that does not ask, which includes every crawler documented not to ask. It is a one line decision with an outsized effect on the AI visibility of a business whose main market is not the one the default happens to serve, and of the four things here it is the cheapest to check and the easiest to forget.