BlogFindings
Hreflang and AI crawlers: 27 of 719 home pages changed language on a header alone
Lantad requested the home page of 1,027 hostnames on 15 September 2026 and read the bytes with no JavaScript executed. 730 answered with a 2xx and an HTML content type, 255 of those declared a hreflang alternate, and asking the same 730 again with a German Accept-Language header changed the declared language on 27 of them. Four of the 27 publish no alternate at all.
What that means in documentation terms has been written up here before, in Googlebot sets no Accept-Language, so your site picks the locale for it, and that post said plainly that it had no figure for how often a real site answers such a client in a language the operator did not intend. This one has the figure. On 15 September 2026 we asked 1,027 home pages twice, once with no Accept-Language header and once asking for German, and compared what came back. We also counted what those pages declare about their own language versions, because the annotation that exists for this problem is hreflang and nobody has counted how often it is actually there. Our methodology page sets out what this scanner will and will not claim from an external fetch, and the limits at the end of this post are the ones that apply here.
In short
- Hreflang and AI crawlers meet at one question: a crawler that states no language preference gets whatever the origin decides is the default, and hreflang is the only thing on the page that names the other versions. 255 of 730 home pages Lantad read on 15 September 2026 declared at least one alternate, carrying 3,923 alternate links between them.
- Asked a second time with Accept-Language set to de-DE, 27 of 719 comparable home pages returned a different base language subtag than they had returned with no such header. Re-running both header states against those hosts the same day reproduced all 27.
- Four of those 27 declared no hreflang alternate anywhere in the raw HTML: asana.com, agoda.com, erstegroup.com and hu-berlin.de. Their other language versions are reachable by sending a header, and by nothing a crawler reads on the page.
- 23 of the 255 pages carrying hreflang did not list themselves among their own alternates. Google's localized versions documentation, carrying Last updated 2025-12-22 UTC, states that each language version must list itself as well as all other language versions, and that if two pages do not both point to each other the tags will be ignored.
- The string accept-language appears in none of the nine AI crawler vendor documentation pages behind the 15 tokens this scanner evaluates, fetched on 15 September 2026. RFC 9110 section 12.5.4 states that a user agent that does not give the user control over linguistic preference MUST NOT send the header at all.
What we requested, and what answered
One GET per hostname for the site root, over HTTPS, as LantadBot/1.0 with redirects followed, a single attempt, a twenty second timeout and no JavaScript executed. The frame is this repository's committed industry corpus of 1,027 hostnames, an editorial sampling frame of eight roughly equal sectors weighted toward large organisations rather than a random draw from the web. The conduct rules our crawler follows are published at our bot page, and the same frame is behind several earlier counts, including 17 of 380 home pages that sent a crawler zero words.
Of the 1,027, 93 failed at the transport layer, 203 answered a status outside the 2xx range, one answered 2xx with a content type that was not HTML, and 730 answered with both a 2xx status and an HTML content type. Those 730 are the only pages counted anywhere below. Roughly three in ten hostnames in the frame are therefore absent from every figure in this post, and the absent ones skew toward sites that refuse an unfamiliar user agent, which is its own subject and one we have counted separately in 79 of 115 sites that refused a crawler their robots.txt allows.
The first pass sent no Accept-Language header. That is the condition a crawler is documented to arrive in, and it is also the condition under which a server has to fall back on something else: an address, a cookie it has not got, or a default written into configuration years ago. The second pass repeated the request to the same 730 hostnames with Accept-Language set to de-DE,de;q=0.9 and changed nothing else. 719 of the 730 answered both passes with a 2xx and an HTML content type, and those 719 are the comparable pairs.
From each response we recorded the final URL after redirects, the lang attribute on the html element, the document title, and every link element carrying both a rel of alternate and a hreflang value. Nothing was executed, so a page that injects its alternates from JavaScript after load is counted here as carrying none.
GET / twice, same client, 15 September 2026
- GET / HTTP/2 Host: <hostname> pass 1
- User-Agent: LantadBot/1.0 (+https://lantad.co/bot) identical in both passes
- Accept: text/html,application/xhtml+xml identical in both passes
- (no Accept-Language header sent) pass 1 only
- Accept-Language: de-DE,de;q=0.9 pass 2 only
- record final URL, html lang, title, link rel=alternate hreflang both passes
- 730 answered 2xx with HTML in pass 1, 719 in both comparable pairs
Hreflang and AI crawlers: what does the annotation decide?
A hreflang annotation is a link element in the head that names another URL and the language or locale it serves. Google's documentation on localized versions of a page, carrying Last updated 2025-12-22 UTC, describes three equivalent ways to publish the same information, being HTML link elements, HTTP headers and a sitemap, and states that there is no benefit in Search to using more than one. This post counts only the HTML form, because that is the one an external fetch of a page can see.
Two of that page's guidelines decide whether the annotation works at all. The first is that each language version must list itself as well as all other language versions. The second is that if two pages do not both point to each other, the tags will be ignored. A set of alternates is therefore not a list of links, it is a mutual declaration, and a page that names four siblings without naming itself has published something the guidance says can be discarded.
The same page carries a sentence that complicates the obvious reading of everything below, and it belongs here rather than in a footnote: Google does not use hreflang or the HTML lang attribute to detect the language of a page, and uses algorithms to determine the language instead. So hreflang is not how a language is identified. It is how variants of the same content are tied together once the language is already known, which is a narrower job than the annotation is usually credited with, and a good reason not to treat a missing hreflang as a language being invisible.
What it does decide is discoverability of the other versions from the page in hand. A client that fetches one URL and reads the bytes has the alternates or it has nothing. That is the same property that makes a self-referential canonical tag worth checking rather than assuming, which we looked at in five of five canonical tags that pointed at themselves, and it is why the count that matters for AI visibility is not how many languages a site publishes but how many of them a single fetch can find. RFC 9110 section 12.5.4 puts a hard edge on the alternative route: a user agent that does not provide the user with control over linguistic preference MUST NOT send an Accept-Language header field. A crawler has no user at the keyboard to express a preference, so the header a locale-adaptive site is waiting for is one the specification tells that client not to send.
Flow: Fetch one URL, no Accept-Language (header absent) to Server negotiates or redirects; Server negotiates or redirects to Server returns its default; Server returns its default to Page declares hreflang alternates; Server returns its default to Page declares none; Page declares hreflang alternates to Other versions discoverable; Page declares none to Other versions invisible to this client.
255 of 730 home pages declared an alternate
255 of the 730 pages carried at least one link element with a rel of alternate and a hreflang value, and 475 carried none. The 255 carried 3,923 alternate links between them, so the median set is small and the mean is not: 21 pages declared exactly one alternate, while aircanada.com declared 336, teamviewer.com 270 and glossier.com 190. A set of one is worth pausing on, because a single alternate that is the page itself is a self-reference with nothing to point at, and a single alternate that is not the page itself is half of a pair whose other half we did not fetch.
139 of the 255 declared the reserved x-default value. That is the fallback Google's guidance recommends for unmatched languages, and it is specifically recommended for auto-redirecting home pages, which is the population this post is about. The remaining 116 publish a set of alternates with no stated answer for a client whose preference matches none of them.
232 of the 255 included an alternate whose URL resolved to the page's own final address. 23 did not, among them berlin.de, bundesbank.de, bundesregierung.de, lemonde.fr, louvre.fr, edx.org, box.com, kth.se, interpol.int, bankofamerica.com, statefarm.com and op.fi. To take one: lemonde.fr answered in French and declared four alternates, all four pointing at the same English page under en-US, en, en-CA and en-GB, and none pointing at the French page doing the declaring. Under the guideline quoted above that set is incomplete, and the guidance does not describe a partial credit.
Separately from hreflang, 38 of the 730 pages carried no lang attribute on the html element at all. That is the cheapest language signal there is, one attribute in the first tag of the document, and on those 38 pages a parser reading the bytes has nothing to read. It is the same class of omission as a page that ships no structured data at all, which is what we found counting 141 of 382 home pages that carried none in the raw HTML: the machinery is standard, cheap and absent. Across the 692 pages that did declare a lang, 505 declared a base subtag of en and there were 26 distinct base subtags in total, which says more about the composition of this frame than about the web.
- html lang attribute 692 of 730 pages. 38 carried none, so no language is declared in the markup at all.
- At least one hreflang alternate 255 of 730 pages, carrying 3,923 alternate link elements between them.
- Self-referencing alternate 232 of the 255. 23 named siblings without naming themselves, which Google's guidance says makes the set ignorable.
- x-default fallback 139 of the 255 declared it. 116 published alternates with no stated answer for an unmatched preference.
- Exactly one alternate declared 21 of the 255, against 336 on aircanada.com, 270 on teamviewer.com and 190 on glossier.com.
27 pages returned a different language for one header
719 hostnames answered both passes with a 2xx status and an HTML content type. On 28 of them the base language subtag declared in the html element differed between the two requests. We then repeated both header states against those hosts on the same day, and 27 of the 28 reproduced exactly. The one that did not is admin.ch, which returned German under both headers on the second run, so it is excluded from every figure here and 27 is the number this post stands behind.
The switches are unambiguous rather than marginal. slack.com went from en-US to de-DE, wise.com from en-GB to de-DE, airbnb.com from en to de, france.fr from fr to de, slovenia.info from sl to de, rug.nl from nl to en and italotreno.com from it to en. Several changed the URL as well as the document: gitlab.com moved from about.gitlab.com to about.gitlab.com/de-de/, slack.com to slack.com/intl/de-de/, wix.com to a different hostname at de.wix.com. Others changed the document while leaving the address alone, which is the harder case for anyone downstream, because two different documents then share one URL. One oddity is worth recording because it is a formatting error rather than a design choice: wyndhamhotels.com declared en_US and then de_DE in its lang attribute, with an underscore, which is not the separator BCP 47 uses.
Google's page on managing multi-regional and multilingual sites, carrying Last updated 2025-12-10 UTC, states that the Googlebot crawler usually originates from the USA and that the crawler sends HTTP requests without setting Accept-Language in the request header, and it advises avoiding automatic redirection of users from one language version of a site to a different one. 27 sites in this sample do the thing that page advises against, and they do it for a client the same page says will never send the header that makes it work correctly.
Only the first pass resembles a crawler. The second is a thought experiment about what those servers are capable of, and its value is that it proves the capability exists rather than being assumed: these are not sites that happen to be German, they are sites that decide. That distinction matters because content negotiation has come up here before with a different answer. When we checked whether crawlers negotiate for markdown rather than HTML, ten AI crawlers fetched markdown by URL and none of them by content negotiation, and when we asked 36 Framer sites for markdown with an Accept header, 35 of the 36 served it. Servers negotiate. The clients this site exists to measure do not appear to ask.
| Hostname | No header | Asking de-DE | Alternates | URL changed |
|---|---|---|---|---|
| slack.com | en-US | de-DE | 16 | yes |
| gitlab.com | en-US | de-DE | 11 | yes |
| wise.com | en-GB | de-DE | 46 | no |
| bundesbank.de | en-GB | de-DE | 2 | yes |
| rwth-aachen.de | en-US | de-DE | 2 | yes |
| epfl.ch | fr-FR | de-DE | 4 | yes |
| wyndhamhotels.com | en_US | de_DE | 12 | yes |
| snyk.io | en-US | de | 7 | yes |
| airbnb.com | en | de | 96 | no |
| pipedrive.com | en | de | 25 | yes |
| dropbox.com | en | de | 24 | no |
| wix.com | en | de | 22 | yes |
| alibaba.com | en | de | 18 | no |
| france.fr | fr | de | 16 | yes |
| monday.com | en | de | 15 | yes |
| newrelic.com | en | de | 8 | yes |
| slovenia.info | sl | de | 7 | yes |
| luxair.lu | en | de | 6 | yes |
| germany.travel | en | de | 4 | yes |
| snb.ch | en | de | 3 | yes |
| rug.nl | nl | en | 3 | no |
| coursera.org | en | de | 2 | no |
| italotreno.com | it | en | 2 | yes |
| asana.com | en | de | 0 | yes |
| agoda.com | en | de | 0 | yes |
| erstegroup.com | en | de | 0 | yes |
| hu-berlin.de | en | de | 0 | no |
Four of the 27 published nothing for a crawler to follow
The four rows at the bottom of that table are the finding. asana.com, agoda.com, erstegroup.com and hu-berlin.de each returned a different language when asked in German, and each carried no hreflang alternate anywhere in the raw HTML of either response. Their other language versions exist, demonstrably, because we received them. They are reachable by sending a header, and by nothing at all that a client reads on the page it fetched.
Three of the four also changed the URL, which means the alternate addresses are real and stable and could be declared: hu-berlin.de is the exception and returned a different document at the same address. That is the worst shape of the four, because a citation is a URL and nothing else. An answer engine that cited hu-berlin.de in German and an answer engine that cited it in English would produce the identical link, and the document a reader then receives depends on the reader's own browser settings rather than on anything the citing engine saw.
It is worth being precise about how large this finding is not. Four is four, out of 719 pages that answered both passes, and the other 23 negotiators do declare alternates, some of them extensively. The claim is not that the web is broken. It is that a site can be locale-adaptive and undeclared at the same time, that the combination is invisible from the outside unless you send two requests and compare, and that a single scan of a single URL will never surface it. That is the honest limit of any one-shot check, including ours, and it is the same reason a per token robots.txt evaluation beats reading the file by eye.
What none of this measures is whether any engine actually cited any of these pages, in any language, ever. No access log was read. No AI crawler was observed fetching any of these hostnames. Nothing here connects a declared alternate to an appearance in an answer, and no operator publishes a rule that would let that connection be made honestly. The finding is about what is legible in the bytes, which is the only thing an external fetch can ever be about, and what a crawler receives from one page is where that question starts rather than ends.
-
asana.comSwitches, undeclared en with no header, de asking German, moving to asana.com/de. No alternate link element in either response. -
agoda.comSwitches, undeclared en with no header, de asking German, moving to agoda.com/de-de/. No alternate link element in either response. -
erstegroup.comSwitches, undeclared en with no header, de asking German, at a different address. No alternate link element in either response. -
hu-berlin.deSwitches at one URL en with no header, de asking German, same address both times, so one URL serves two documents and a citation cannot distinguish them.
What the crawler vendors publish about language
The scanner evaluates 15 AI crawler tokens, and those tokens belong to nine vendors whose documentation pages are recorded in this repository's crawler registry and listed at our crawler reference. We fetched all nine on 15 September 2026 and searched the delivered bytes for the string accept-language. It appears in none of them.
That is a statement about nine specific pages and not about nine companies. OpenAI's crawler documentation names its tokens and their purposes and says nothing about language or locale. Anthropic's crawler article does the same. Four of the nine pages do contain the string hreflang, but in every case it is the vendor's own support site declaring its own translations in its own head, or a JavaScript property name in a bundled script, rather than guidance to a site owner. There is no vendor statement here to plan against in either direction: nothing promises the header is sent and nothing promises it is not.
Google is the exception and the contrast is instructive. Its documentation on how Google crawls locale-adaptive pages, carrying Last updated 2025-12-10 UTC, states that the crawler sends HTTP requests without setting Accept-Language, that the default IP addresses of Googlebot appear to be based in the USA, and that Googlebot additionally crawls with IP addresses based outside the USA under what it calls geo-distributed crawling. It then recommends separate locale URL configurations annotated with rel alternate hreflang. That is a vendor telling site owners exactly what its client does and what to do about it. Eight other vendor pages say nothing at all.
Read alongside RFC 9110, Googlebot's documented silence looks less like a gap and more like conformance: the specification instructs a user agent with no user-controlled preference not to send the field, and Googlebot has no user. The reasonable default assumption for the other eight is the same behaviour, but it is an assumption, and this site tries not to publish those as findings. What can be said without assuming anything is that a site which negotiates language on a header has bet its non-default versions on a behaviour that one vendor documents itself as not having and eight do not discuss.
-
developers.openai.comNo mention Zero occurrences of accept-language, hreflang or language in the fetched bytes. -
support.claude.comNo mention Zero occurrences of accept-language. One hreflang occurrence, the page's own translation links. -
docs.perplexity.aiNo mention Zero occurrences of accept-language or hreflang in the fetched bytes. -
developers.google.comDocumented elsewhere The crawler overview does not mention it, but Google's international documentation states plainly that the crawler sets no Accept-Language. -
support.apple.comNo mention Zero occurrences of accept-language. The hreflang occurrences are the support site's own alternates. -
developers.facebook.comNo mention Zero occurrences of accept-language, hreflang or language in 5,579 bytes. -
zhanzhang.toutiao.comNo mention Zero occurrences of accept-language or hreflang in the fetched bytes. -
commoncrawl.orgNo mention Zero occurrences of accept-language or hreflang in the fetched bytes. -
developer.amazon.comNo mention Zero occurrences of accept-language. Three hreflang occurrences, all in a bundled script.
What to check on your own site
The check is two requests and a diff, and anyone can run it against their own domain in a minute. Fetch your home page with no Accept-Language header at all, then fetch it again asking for a language you publish, and compare the final URL, the lang attribute and the first paragraph of prose. If the two differ, your site negotiates, and the version a preference-free client receives is the one your visibility rests on. If they are identical, the URL is deciding, which is the shape the guidance recommends and the shape a citation can carry.
Then look at what the first response declares. Count the alternate link elements. Check that one of them resolves to the URL you just fetched, because 23 of the 255 pages in this sample failed that and it is the condition on which the guidance says the whole set is ignored. Check whether x-default is present, since 116 of the 255 published none. These are head elements, they cost nothing to serve, and they are the only description of your other language versions that survives a fetch with no headers and no JavaScript.
Two things not to conclude. Adding hreflang does not make an engine cite the right version, because Google says in its own guidance that it does not use hreflang or the lang attribute to detect a page's language in the first place. And an absent annotation is not the same as an absent page: the other versions are still there, still crawlable at their own addresses if those addresses are linked from somewhere, and possibly in a sitemap that this method never looked at. What the annotation buys is that one fetch of one URL carries the map, which matters most for exactly the clients that fetch one URL and read what they get.
The wider pattern is familiar from every other signal counted on this frame, and from the AI Visibility Index, which prints the sample size beside every figure for the same reason this post prints 730 rather than a percentage. The expensive work of translation is done, the servers are configured to deliver it, and the cheap declaration that would make it legible to a machine reading one response is missing on a meaningful minority of sites. That was the finding when we asked whether pages tell a crawler anything had changed, in 250 of 647 home pages that sent no change signal, and it is the finding here. The gap is not capability. It is declaration.
- Fetch once with no Accept-Language This is the condition a crawler is documented to arrive in. Record the final URL and the html lang attribute.
- Fetch again asking for a language you publish If the final URL, the lang attribute or the prose differs, your site negotiates and a preference-free client gets the default.
- Confirm a self-referencing alternate 23 of the 255 pages carrying hreflang in this sample did not list themselves, which the guidance says makes the set ignorable.
- Confirm x-default is declared 116 of the 255 published alternates with no stated fallback for a preference that matches nothing.
- Confirm an html lang attribute exists 38 of the 730 pages carried none. It is one attribute in the first tag of the document.
Lantad
Published .
A multilingual site answers a question on every request that a single language site never has to: which version. The URL can settle it, if each language lives at its own address. The request can settle it instead, if the server reads the caller's Accept-Language header or guesses from the caller's address. Both are in wide use, and the two produce different results for a client that carries no preferences, which is exactly what an AI crawler is.
Common questions
Do AI crawlers send an Accept-Language header?
No vendor in this sample says. The string accept-language appears in none of the nine AI crawler documentation pages behind the 15 tokens this scanner evaluates, fetched on 15 September 2026. Google documents the behaviour for Googlebot in its international search documentation rather than on its crawler page, stating that the crawler sends HTTP requests without setting Accept-Language in the request header. RFC 9110 section 12.5.4 states that a user agent that does not provide the user with control over linguistic preference MUST NOT send the field, and a crawler has no user, so the same behaviour is the reasonable expectation for the other eight. Lantad has not observed the headers any AI crawler sends and does not claim to have.
How many sites change language based on the Accept-Language header?
27 of 719 home pages that answered two requests from Lantad on 15 September 2026 returned a different base language subtag when the second request asked for de-DE. All 27 reproduced when both header states were repeated the same day. A twenty eighth, admin.ch, did not reproduce and is excluded. The frame is this repository's industry corpus of 1,027 hostnames weighted toward large organisations, so the proportion describes that frame and not the web.
Does hreflang help a page get cited by an AI answer engine?
Nothing measured here says so. Google's localized versions documentation, carrying Last updated 2025-12-22 UTC, states that Google does not use hreflang or the HTML lang attribute to detect the language of a page and uses algorithms instead. What hreflang does is name the other language versions of a page inside the page itself, so a client that fetches one URL and reads the bytes can find them. Whether any engine follows them, and whether following them changes a citation, is not something an external fetch of a home page can answer.
What does a missing self-referencing hreflang break?
Google's localized versions documentation states that each language version must list itself as well as all other language versions, and that if two pages do not both point to each other the tags will be ignored. 23 of the 255 home pages carrying hreflang in this run did not list themselves, among them berlin.de, bundesbank.de, lemonde.fr, louvre.fr, edx.org and box.com. lemonde.fr answered in French and declared four alternates, all four pointing at the same English page and none at the French page doing the declaring.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
Prose Parity
Prose Parity is Lantad's metric for the share of a page's rendered, human-visible text that is present in the raw HTML an AI crawler receives, where 100% means a crawler sees everything a human visitor sees.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of structuring and publishing content so that generative AI engines can fetch it, read it in full, and cite it when they answer a question.
Answer Engine Optimization (AEO)
Answer Engine Optimization (AEO) is the practice of making a page readable, allowed, and structured enough that AI answer engines can use it as a source when they compose an answer.
