# Canonical tags and AI crawlers: five of five pointed at themselves > Cloudflare turns a non-self-referencing canonical tag into a 301 for verified AI training crawlers. On five pages this scanner captured on 15 July 2026, all five canonical tags pointed at the page's own address, so the feature would have redirected none of them, and two of the five were not byte-identical to the URL the page was served at. - Canonical page: https://lantad.co/blog/canonical-tags-and-ai-crawlers-five-of-five-self-referential - This file: https://lantad.co/blog/canonical-tags-and-ai-crawlers-five-of-five-self-referential.md - Last substantive update: 2026-08-31 ## Key facts - **Published:** 2026-08-31 - **Category:** Findings - **Author:** Lantad - **Length:** 3660 words - **Takeaway 1:** Canonical tags and AI crawlers were wired together on 17 April 2026, when Cloudflare launched Redirects for AI Training: a toggle that reads the rel=canonical tag already present in a page's HTML and issues an HTTP 301 to verified crawlers in its AI Crawler category, which it says covers GPTBot, ClaudeBot and Bytespider, whenever that tag is non-self-referencing. - **Takeaway 2:** Cloudflare reported that bots in its AI Crawler category visited developers.cloudflare.com 4.8 million times over 30 days and consumed deprecated content at the same rate as current content, and that in March 2026 legacy Workers documentation was crawled around 46,000 times by OpenAI, 3,600 times by Anthropic and 1,700 times by Meta. - **Takeaway 3:** Lantad counted 217 link elements in the raw HTML of five pages captured on 15 July 2026 and found exactly five rel=canonical elements, one per page, every one present before any JavaScript ran and every one naming the page's own address. - **Takeaway 4:** Two of those five canonical values were not byte-identical to the URL the page was served at: gymshark.com and webflow.com both omitted a trailing slash the request carried, a one character difference whose treatment depends on a URL normalisation step that Cloudflare's announcement does not describe and that Lantad did not test. - **Takeaway 5:** Google's canonicalization documentation, carrying Last updated 2026-07-10 UTC, tells sites to include a self-referential canonical on the canonical page itself, and that is precisely the shape Cloudflare excludes from redirection to avoid loops, so a site following that advice on every page receives no redirects from the feature. ## Summary The rel=canonical tag has been advice since April 2012. [RFC 6596](https://www.rfc-editor.org/rfc/rfc6596.html) describes it as a way to designate one address as preferred over resources with duplicative content, and the document is filed as Informational rather than Standards Track. Search engines have treated it as a signal ever since, which is to say a page can name a different address and a client is free to decline. On 17 April 2026 Cloudflare published something that changes what that tag does for one class of client: [Redirects for AI Training](https://blog.cloudflare.com/ai-redirects/) reads the canonical tag already sitting in a page's HTML and, for verified [AI crawlers](https://lantad.co/glossary/ai-crawler) in its training category, converts it into an HTTP 301 before the response leaves the edge. That is an external development. This site did not build it, did not test it, and reports Cloudflare's figures below as Cloudflare's. What can be added here is the thing the feature depends on and that nobody appears to have counted: what canonical tags actually look like on real pages. This scanner captured and stored five pages on 15 July 2026, and every claim about those five below was measured with an HTML parser over the stored bytes rather than inferred from a summary. [How a capture is taken and scored](https://lantad.co/methodology) is published in full. The finding is narrow and it has an awkward shape: the feature has a precondition, none of the five pages met it, and the reason they did not meet it is that they were following Google's published advice. ## What Cloudflare shipped, and what it explicitly does not do The announcement is precise about the mechanism, which is worth restating in its own terms rather than paraphrasing. The feature operates on two inputs: Cloudflare's verified bot category field, and the canonical link elements already in the HTML. When a request arrives from a verified AI Crawler, Cloudflare reads the response HTML, and if a non-self-referencing canonical tag is present it issues a 301 Moved Permanently to the canonical URL before returning the response. Nothing is added to the page and nothing is authored by the site owner. The tag that was already there acquires a second job. The scope is narrower than the phrase AI crawler usually implies on this site. Cloudflare states that its AI Crawler category covers bots that crawl for AI model training, naming GPTBot, ClaudeBot and Bytespider, and that this category is distinct from the AI Assistant and AI Search categories that cover AI agents. So a search-time fetcher such as OAI-SearchBot sits outside it. That distinction matters for anyone reading a vendor's category name as a synonym for every automated client, and it is the same distinction that makes [a crawler token a claim about purpose rather than an identity](https://lantad.co/blog/a-user-agent-is-a-claim-not-an-identity). Verification is doing real work here: the redirect is keyed to Cloudflare's verified bot classification rather than to a user agent string, which is the only version of this that cannot be triggered by anything that types the right name. Of the fifteen crawler tokens this site has catalogued, [six publish a User-Agent string a site can match on](https://lantad.co/blog/gptbot-detection-six-of-fifteen-tokens-publish-a-ua), and a string is not proof of anything. The post is equally clear about the limits, and they are the part a reader should hold on to. It does not retroactively correct training data already ingested. It does not cover unverified crawlers outside the AI Crawler category. Humans and AI agents visiting a deprecated page are not redirected. Cross-origin canonicals are excluded by design, because Cloudflare says those are often used for domain consolidation rather than content freshness. And self-referencing canonicals do not trigger a redirect either, to avoid loops. Each of those exclusions is defensible on its own terms. Together they define a fairly small target: a page, on a paid Cloudflare plan, whose canonical tag names a different address on the same origin. ## Do AI crawlers follow canonical tags? On the evidence Cloudflare published, not reliably, and the company measured this on its own documentation before building anything. Its deprecated Workers documentation carries a deprecation banner, a noindex meta tag and canonical tags pointing at the current pages. Every advisory signal on those pages says the same thing. Cloudflare reports that bots in its AI Crawler category visited developers.cloudflare.com 4.8 million times over the last 30 days and consumed deprecated content at the same rate as current content, and states plainly that the advisory signals made no measurable difference. It gives the March 2026 breakdown for legacy Workers documentation as roughly 46,000 crawls by OpenAI, 3,600 by Anthropic and 1,700 by Meta. Those are Cloudflare's figures from Cloudflare's own property, and this site has not reproduced them. The reason is structural rather than a failure of manners by any particular crawler. A canonical tag was never an instruction. [Google's canonicalization documentation](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls), carrying Last updated 2026-07-10 UTC, opens by saying you can indicate your preference using a number of methods, and then ranks those methods by how strongly they can influence canonicalization: a redirect is a strong signal that the target should become canonical, a rel=canonical annotation is a strong signal that the specified URL should become canonical, and sitemap inclusion is a weak signal. Preference and signal are the operative words in a document written by the company with the most reason to want the tag obeyed. This is a pattern that recurs across every file a site uses to talk to automated clients, and it is worth naming because it predicts which controls work. A sitemap entry is [an assertion a site makes about itself](https://lantad.co/blog/sitemap-lastmod-is-an-assertion) and carries no enforcement. A canonical tag is a preference. By contrast the things that actually change what a client receives are transport level: a status code, a redirect, a refusal. That is why [a 404 on robots.txt allows every crawler while a 503 blocks them all](https://lantad.co/blog/robots-txt-404-and-503-are-opposites), with no negotiation and no interpretation involved. What Cloudflare has done is move the canonical tag across that line for one category of client. The tag stays advisory in the HTML, and the edge answers with a 301 anyway, which is not advisory at all. ## What the canonical tag looked like on five real pages The feature reads something that has to already be there, so the useful question for a site owner is what is actually in their head element. This scanner stores complete raw and rendered captures for a small set of pages, and five of them are real sites rather than synthetic fixtures: astro.build, gymshark.com, the MDN HTML documentation, webflow.com, and the English Wikipedia article on web crawlers. All five were captured on 15 July 2026, all five returned HTTP 200, and none of the five went through a redirect on the way. That is the same five page capture earlier posts here have counted [meta elements](https://lantad.co/blog/eighty-five-meta-elements-two-addressed-a-crawler) and [crawl paths](https://lantad.co/blog/javascript-added-no-new-crawl-paths) over, and it is a small sample stated as one. Counted with an HTML parser over the stored raw response, those five pages carry 217 link elements between them. The distribution is lopsided in a way that says something about what the element is mostly used for now: 86 on the Wikipedia article, 78 on the Gymshark homepage, 25 on MDN, 19 on Webflow and 9 on astro.build. Most of that volume is not about canonicalization at all. Wikipedia's 86 include 47 carrying rel="mw-deduplicated-inline-style" and 26 carrying rel="mw:pageprop/category", which are MediaWiki internals. Gymshark's 78 include 40 preload hints and 23 hreflang alternates; Webflow's 19 include four hreflang alternates. Exactly five of the 217 are canonical elements. One per page, no page carrying two, no page carrying none. Every one of the five was present in the raw HTTP response, before any JavaScript ran, and comparing the raw capture against the rendered capture shows no page whose canonical was injected, changed or removed by rendering. That is the good news in this measurement and it deserves saying, because it is not what happens to every signal: a page can lose most of its text to hydration and [score the same as a page with no text at all](https://lantad.co/blog/hydration-json-scored-the-same-as-no-text). On this small sample the canonical element behaves the way [prose parity](https://lantad.co/glossary/prose-parity) asks everything to behave. It arrives in the bytes. A client without a browser can read it, which is the precondition for any edge feature reading it too. ## Two of the five differed from the served URL by one character Self-referencing is the word the feature turns on, and it is a string comparison somewhere. Three of the five canonical values are byte-identical to the URL the page was served at: astro.build, MDN and Wikipedia. Two are not. Gymshark's homepage was served at https://www.gymshark.com/ and its canonical element names https://www.gymshark.com. Webflow's homepage was served at https://webflow.com/ and its canonical element names https://webflow.com. In both cases the difference is a trailing slash, and in both cases the intent is obviously self-reference rather than a pointer somewhere else. Whether that intent survives a machine reading is not something this measurement can settle, and it would be dishonest to pretend otherwise. Under RFC 3986 an empty path and a path of "/" are distinct strings that most implementations normalise to the same resource for an http URL, and every sensible implementation of a feature like this would normalise before comparing. Cloudflare's announcement does not describe its normalisation step, this scanner did not send a request through Cloudflare's edge to find out, and no claim is made here about what Cloudflare would do with either page. What can be said is narrower and still useful: two of five real homepages carry a canonical value that a naive string equality check would call non-self-referencing, so the correctness of this feature on ordinary sites rests on a normalisation detail that is not documented in the announcement. One caveat belongs with these figures. The Gymshark capture was truncated at 3 MiB, which is a cap this scanner applies and which is a real limit on what any client reads: Googlebot's own documentation, carrying Last updated 2026-02-03 UTC, says it crawls only [the first 2MB of a supported file type](https://lantad.co/blog/googlebot-reads-the-first-two-megabytes), and every fetcher draws that line somewhere. Truncation does not affect the canonical finding, because the head element and its canonical arrived well inside the cap, but a count of elements on a truncated document is a count of what was captured rather than of what the page contains, and that page is a large Shopify storefront of the kind [our Shopify guidance](https://lantad.co/fix/shopify) is written for. The four other captures were not truncated. ## Google recommends the exact tag that switches the feature off Here is the part that is inconvenient for the feature, and it is not a defect in it. Google's canonicalization documentation, the same page carrying Last updated 2026-07-10 UTC, lists among its best practices: do include a rel=canonical link on the canonical page itself, also known as a self-referential canonical. That advice is widely followed, and it is why the measurement above came out the way it did. Every one of the five pages carries a self-referential canonical. Cloudflare excludes self-referencing canonicals from redirection to avoid loops, which is correct behaviour and the only sane choice. The consequence is arithmetic: on those five pages, Redirects for AI Training would have issued zero redirects. That is not an argument against the feature. It is an argument about who it is for. The population it acts on is pages whose canonical names a different URL on the same origin, which in practice means deprecated documentation, consolidated product pages, printer and AMP variants, and paginated or faceted URLs that fold into a parent. Cloudflare's own use case is exactly the first of those, and a documentation site with years of superseded versions is close to the ideal customer for it. A five page marketing site whose every canonical is self-referential gets a toggle that changes nothing, and would be misled by any write-up implying otherwise. There is a broader point here about signals a site can send to an automated client, and it is the same one this site keeps arriving at from different directions. A page's machine-readable head is mostly assertions, and the assertions do not enforce themselves. [Structured data](https://lantad.co/glossary/structured-data) is the clearest case: on this same five page capture, [three of the five pages carried JSON-LD and one held 96 of the 102 nodes](https://lantad.co/blog/three-of-five-pages-carried-json-ld), which tells you the markup was present and tells you nothing about what any consumer did with it. A canonical tag has the same property. The interesting thing about Cloudflare's feature is not that it improves the tag, because it does not touch the tag. It is that it moves the enforcement point from the consumer, which no site controls, to the edge, which some sites do. ## What to check on your own site, and what this post did not measure The useful takeaway is not a recommendation to enable anything, because whether this feature helps you is decided entirely by facts about your own URLs. Four checks answer it, and all four are things you can run without buying anything. Count your non-self-referential canonicals first. If that number is zero, the toggle is a no-op on your site and nothing else in this post applies to you. Then confirm your canonical tags are in the HTML rather than written by JavaScript, because an edge reading the response body sees what the response body contains and nothing later. Then check that the pages you care about are reachable at all, since a canonical tag on a page a crawler is forbidden to fetch changes nothing: [testing robots.txt per crawler token](https://lantad.co/tools/robots-txt-tester) is the prior question, and it is prior for every AI client rather than just this one. And keep an eye on the token lists themselves, which move: Cloudflare will now [generate a robots.txt whose AI crawler list changes without you](https://lantad.co/blog/cloudflare-generates-robots-txt-and-the-token-list-moves), and a category defined by a vendor is a category that vendor can redefine. What this post did not measure should be stated as plainly as what it did. Lantad did not test Cloudflare's feature, did not send a request through an edge with it enabled, and cannot confirm any of Cloudflare's reported figures, which are attributed to Cloudflare throughout. Lantad did not determine how the trailing slash cases would be normalised. Five pages is a small sample and it is not a survey of the web: it is five real pages, named, captured on one date, with the numbers reproducible from the stored bytes. This site does not publish a percentage of the web from five captures, and [it withholds a grade when it cannot measure the thing it is grading](https://lantad.co/blog/why-we-withhold-a-grade) for the same reason. What the five do establish is that the canonical element is present, singular, and in the raw HTML on every one of them, which is a better state than several of the other head signals this capture has been counted for, and that its self-referential form is the norm rather than the exception. Anyone measuring [AI visibility](https://lantad.co/glossary/ai-visibility) should treat an edge feature that acts on a tag most pages carry in a shape it ignores as a narrow tool with a real use, not as a switch that changes what crawlers do with a site. ## Questions and answers **Do AI crawlers follow canonical tags?** Not reliably, on the evidence Cloudflare published on 17 April 2026. It reported that bots in its AI Crawler category visited developers.cloudflare.com 4.8 million times over 30 days and consumed deprecated content at the same rate as current content, despite those pages carrying a deprecation banner, a noindex meta tag and canonical tags pointing elsewhere, and said the advisory signals made no measurable difference. The tag was never a directive: Google's canonicalization documentation, Last updated 2026-07-10 UTC, calls a rel=canonical annotation a strong signal and describes the whole mechanism as indicating your preference. **What does Cloudflare's Redirects for AI Training actually do?** It reads the rel=canonical tag already in your HTML and, when a request comes from a verified crawler in Cloudflare's AI Crawler category and that tag names a different URL on the same origin, it returns an HTTP 301 to the canonical URL instead of the page. Cloudflare names GPTBot, ClaudeBot and Bytespider in that category and distinguishes it from its AI Assistant and AI Search categories. It is a toggle in AI Crawl Control on paid plans, it does not change training data already ingested, and it does not redirect humans, AI agents or unverified crawlers. **Will the feature do anything if my canonical tags point at their own pages?** No. Cloudflare excludes self-referencing canonicals from redirection to avoid loops. That matters more than it sounds, because Google's canonicalization documentation recommends including a rel=canonical link on the canonical page itself, so most well-maintained pages carry exactly the shape the feature ignores. On the five pages Lantad captured on 15 July 2026, all five canonicals were self-referential, so the feature would have issued no redirects on any of them. **Does a canonical tag need JavaScript to be read?** It should not, and on this sample it did not. Lantad counted 217 link elements in the raw HTTP responses of five pages captured on 15 July 2026 and found five canonical elements, one per page, every one present in the raw HTML before any JavaScript ran, with no page whose canonical was added, altered or removed by rendering. That is the state any edge feature or non-rendering crawler needs, since both read the response body rather than a hydrated DOM. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot