Duplicate content and AI crawlers: 305 of 313 home pages answered at more than one live URL

Does duplicate content hurt AI search visibility?

Nothing measured here says it does. What it says is that the thing search engines spend engineering effort resolving is present on almost every site in the corpus, and that the crawlers reading those sites for answer engines publish no statement about resolving it. That is a gap in the record rather than a finding about rankings, and the honest version of this post is the gap, not a claim about what it costs.

The measurement starts where every study on this site starts. Lantad requested /robots.txt from all 1,419 hostnames in the two committed corpus seed files, as LantadBot/1.0, the agent documented on the bot page. 1,098 answered HTTP 200 and 1,064 of those parsed into at least one user-agent group. Every file was parsed with this repository's own parseRobotsTxt in core/src/robots.ts and every verdict computed with its evaluateRobots, the same pair behind the robots.txt tester. 1,408 of the 1,419 left LantadBot allowed at the root, and the 11 that did not were never asked for a page.

Each of the 1,408 was then asked for five things: its home page at the bare apex, the same page at the www host, the same page over plain http, the same page at /index.html, and the same page at the apex with utm_source and utm_medium appended. Redirect following was switched off on all five, because a redirect is the answer rather than an obstacle: a host that sends 301 from its apex to its www host has resolved the duplicate, and the point of the measurement is to separate the hosts that resolve from the hosts that serve. The fetch path otherwise matches the one on the methodology page, no JavaScript executed and no browser involved, against the same sampling frame as the crawlability study.

Why the denominator is 313 and not 1,419

931 of the 1,408 served a readable HTML home page at either the apex or the www host. Only 313 of those served it at the bare apex itself with no redirect, and that number is the one every figure in this post is divided by, for a reason worth stating plainly rather than burying in a limits paragraph.

Four of the five URL forms are built on the apex host. If a site redirects its apex to its www host, it redirects the apex tracking URL and the apex /index.html too, so all four variants come back as 3xx and the site can only ever score one live form. Counting those 618 hosts in the denominator would not measure their duplicates, it would measure the fact that the probe was pointed at a host they do not serve from. So they are reported and set aside. The 313 are the hosts where asking four apex-based questions produces four real answers.

That decision also has to survive the pages moving under the measurement, which they do. 311 of the 313 answered a second read of the apex minutes later. 206 of those 311 were byte identical to the first read and 105 were not, so a third of the home pages in this corpus do not serve the same bytes twice in a row. A comparison keyed on a hash alone would call a third of the duplicates different. Every pair below is therefore matched two ways, byte identical or else identical page title with word counts within 2 percent of each other, and the second test was validated against the first read of each page: 307 of the 311 pages matched themselves. Four did not, which is the measured floor on how often this method misses a duplicate it should have found. That is the same volatility that made redirect chains and soft 404 detection two-pass measurements on this corpus, and that put two of the named candidates for this post out of the piece: both changed state between the capture and the re-read, so neither is quoted here.

What Google documents, and what AI crawler documentation leaves out

The reason a site can live with five addresses for one page is that the dominant reader of the web has handled it for twenty years. Google's page on consolidating duplicate URLs, last updated 2026-07-10 UTC, names three methods and ranks them by strength: a redirect is "A strong signal that the target of the redirect should become canonical", a rel=canonical annotation is "A strong signal that the specified URL should become canonical", and sitemap inclusion is "A weak signal that helps the URLs that are included in a sitemap become canonical". The same page says the methods are optional, that "none of them are required; your site will likely do just fine without specifying a canonical preference", and that where a site specifies nothing, Google will identify which version of the URL is objectively the best version to show.

Read the crawler documentation of the three vendors whose bots fetch pages for answer engines and none of that vocabulary appears. OpenAI's crawler page, read on 15 October 2026 and carrying no date of its own, documents OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User, and uses neither the word canonical nor the word duplicate. The nearest it comes is a note about its own crawl efficiency: where a site allows both OAI-SearchBot and GPTBot, "we may use the results from just one crawl for both use cases to avoid duplicative crawling". That is deduplication of requests, not of addresses. Anthropic's crawler article, dated 7 April 2026, documents ClaudeBot, Claude-User and Claude-SearchBot and says nothing on the subject. Perplexity's bots page, also undated and read the same day, documents PerplexityBot and Perplexity-User and says nothing either.

What the specifications give you is the default, and the default is strict. RFC 3986, the URI generic syntax, January 2005, states that determination of equivalence or difference of URIs is based on string comparison, and the scheme and the host are part of that string. Nothing in it makes a www host and its apex the same resource. That equivalence is a convention sites implement with a redirect, and the three vendors above document neither the convention nor any substitute for it, which is why an AI visibility check cannot assume it and this measurement does not.

The tracking parameter is the duplicate nearly every site has

304 of the 313 home pages served the same page again at a URL carrying utm_source and utm_medium. 166 of those were byte identical to the clean apex response and 138 matched on title and word count. Only 5 of the 309 that answered at all served a different page, which is what you would expect: a marketing parameter is not meant to change anything, and it does not.

This is also the one case the corpus mostly handles. 253 of the 304 responses declared a rel=canonical naming the clean home page URL, which is exactly the control Google's page describes and the strongest instruction a page can carry about its own address. 44 declared no canonical at all, so nothing in the response of those 44 pages distinguishes the address a campaign link produced from the address the site considers its home page. On the earlier capture of five pages this blog found that all five canonical tags pointed at themselves, which is the healthy pattern at that small sample, and the 313 here put the overall rate at 263 declaring a canonical, 251 of them naming the apex URL itself.

Two of the 304 are worth naming because they show the control failing in the subtlest way available. canonical.com served the tracking URL with a canonical of https://canonical.com/?utm_source=lantad&utm_medium=audit, the parameter intact and the ampersand HTML escaped, so the page nominates the campaign address as its own canonical form. ama.com.au did the same thing to a different path, naming https://ama.com.au/site/content/default.aspx?utm_source=lantad&utm_medium=audit. Both were re-fetched by hand the same day and both still did it. A canonical that copies the query it was supposed to resolve does nothing at all, and unlike a missing tag it looks correct in any audit that only checks whether the element is present. A site survey that counts declarations rather than reading them, which is most of what passes for a URL inventory or a sitemap check, would score both of these as compliant.

47 sites served the home page at two hosts, and 13 at two schemes

The www duplicate is the oldest one on the web and 47 of the 313 still have it: the apex and the www host both answered HTTP 200 with the same page, neither redirecting to the other. 27 of the 47 were byte identical across the two hosts. On 20 of the 47 neither response declared a canonical, so there is no statement anywhere on either page about which of the two hosts is the site. flux-academy.com is the clean example, verified by hand on 15 October 2026: the apex returned 258,709 bytes and the www host 258,713, four bytes apart, and neither carried a canonical element. 22 of the 47 did declare one on both and agreed on a single URL, which is the arrangement that makes the duplication harmless.

13 served the home page over plain http as well as https, with no redirect to the secure address. 9 of those 13 carried 100 or more words, so most are real pages rather than stubs, but the set includes some thin ones: archive.org answered http with 1,951 bytes and britishairways.com with 8,997, neither declaring a canonical. Two of the 13 did something worse than omitting the tag. americanelitepainters.com served its http home page with a canonical of http://americanelitepainters.com/, and iitb.ac.in with a canonical of http://iitb.ac.in/, both nominating the insecure address as the one true URL of the page. Both were re-fetched by hand and both still do. This is the mirror image of the robots.txt redirects that crossed hosts on 391 sites and of the subdomain files that answered differently from the apex on 73 of 277: the same content, a different authority, and nothing joining them up.

110 of 311 served the home page at /index.html as well, and that case is handled better than either of the above, with 87 of the 110 naming the clean home page URL in the canonical and 21 naming nothing. The one site in the whole set whose /index.html canonical named /index.html itself answered a redirect when it was re-read by hand hours later, so it is counted and not named. Where a duplicate is resolved, the page the crawler keeps is the one the canonical names, and prose parity is then measured against that page rather than against whichever address a link happened to use.

What this measurement cannot see

Four things, and they bound the finding more tightly than the headline does.

No AI crawler was observed and no request was sent as one. Every byte here was fetched as LantadBot, and the vendor documentation reading says only that three vendors publish nothing about canonicalization, not that their crawlers ignore it. A crawler could honour rel=canonical perfectly and never document it. That distinction is the whole of the Lantad house rule against printing a confident number where the measurement did not reach, and it is the same reason the scanner withholds a grade rather than estimating one.

The word counts are not the shipped extractor. extractFromHtml in core/src/extract.ts classifies main content against boilerplate, and that is the right tool for asking what a crawler gets from a page, which is what the what GPTBot sees tool does. It is the wrong tool for asking whether two responses are the same document, where all that is needed is a number that moves when the document moves. These counts come from a tag stripping counter written for this run, applied identically to both sides of every comparison.

Five URL forms is not the space of duplicates. Case variants, trailing slashes on interior paths, session identifiers, print views, pagination parameters and locale prefixes all produce addresses this probe never asked for, so 305 of 313 is a floor on how many hosts serve one page at several addresses and not a census. A locale prefix in particular is the case most likely to be mistaken for a duplicate when it is not, which is why hreflang was measured separately. And a home page is one page: the apex of a site is where canonicalization is most likely to have been set up deliberately, so interior pages are a separate measurement and probably a worse result.

Finally, the 1,419 hostnames are two committed sampling frames rather than a sample of the web, so no proportion here generalises beyond them, and the 313 are a self-selected quarter of those: sites that answer at their bare apex. A site that redirects its apex to www has already demonstrated the habit this post is measuring the absence of, which means the 313 are, if anything, the less careful end of the corpus.