BlogFindings
Canonical tags and AI crawlers: five of five pointed at themselves
Cloudflare turns a non-self-referencing canonical tag into a 301 for verified AI training crawlers. On five pages this scanner captured on 15 July 2026, all five canonical tags pointed at the page's own address, so the feature would have redirected none of them, and two of the five were not byte-identical to the URL the page was served at.
That is an external development. This site did not build it, did not test it, and reports Cloudflare's figures below as Cloudflare's. What can be added here is the thing the feature depends on and that nobody appears to have counted: what canonical tags actually look like on real pages. This scanner captured and stored five pages on 15 July 2026, and every claim about those five below was measured with an HTML parser over the stored bytes rather than inferred from a summary. How a capture is taken and scored is published in full. The finding is narrow and it has an awkward shape: the feature has a precondition, none of the five pages met it, and the reason they did not meet it is that they were following Google's published advice.
In short
- Canonical tags and AI crawlers were wired together on 17 April 2026, when Cloudflare launched Redirects for AI Training: a toggle that reads the rel=canonical tag already present in a page's HTML and issues an HTTP 301 to verified crawlers in its AI Crawler category, which it says covers GPTBot, ClaudeBot and Bytespider, whenever that tag is non-self-referencing.
- Cloudflare reported that bots in its AI Crawler category visited developers.cloudflare.com 4.8 million times over 30 days and consumed deprecated content at the same rate as current content, and that in March 2026 legacy Workers documentation was crawled around 46,000 times by OpenAI, 3,600 times by Anthropic and 1,700 times by Meta.
- Lantad counted 217 link elements in the raw HTML of five pages captured on 15 July 2026 and found exactly five rel=canonical elements, one per page, every one present before any JavaScript ran and every one naming the page's own address.
- Two of those five canonical values were not byte-identical to the URL the page was served at: gymshark.com and webflow.com both omitted a trailing slash the request carried, a one character difference whose treatment depends on a URL normalisation step that Cloudflare's announcement does not describe and that Lantad did not test.
- Google's canonicalization documentation, carrying Last updated 2026-07-10 UTC, tells sites to include a self-referential canonical on the canonical page itself, and that is precisely the shape Cloudflare excludes from redirection to avoid loops, so a site following that advice on every page receives no redirects from the feature.
| Page, captured 15 July 2026 | rel=canonical value | Byte-identical to the URL served | Present before JavaScript |
|---|---|---|---|
| https://astro.build/ | https://astro.build/ | Yes | Yes |
| https://www.gymshark.com/ | https://www.gymshark.com | No, trailing slash absent | Yes |
| https://developer.mozilla.org/en-US/docs/Web/HTML | https://developer.mozilla.org/en-US/docs/Web/HTML | Yes | Yes |
| https://webflow.com/ | https://webflow.com | No, trailing slash absent | Yes |
| https://en.wikipedia.org/wiki/Web_crawler | https://en.wikipedia.org/wiki/Web_crawler | Yes | Yes |
What Cloudflare shipped, and what it explicitly does not do
The announcement is precise about the mechanism, which is worth restating in its own terms rather than paraphrasing. The feature operates on two inputs: Cloudflare's verified bot category field, and the canonical link elements already in the HTML. When a request arrives from a verified AI Crawler, Cloudflare reads the response HTML, and if a non-self-referencing canonical tag is present it issues a 301 Moved Permanently to the canonical URL before returning the response. Nothing is added to the page and nothing is authored by the site owner. The tag that was already there acquires a second job.
The scope is narrower than the phrase AI crawler usually implies on this site. Cloudflare states that its AI Crawler category covers bots that crawl for AI model training, naming GPTBot, ClaudeBot and Bytespider, and that this category is distinct from the AI Assistant and AI Search categories that cover AI agents. So a search-time fetcher such as OAI-SearchBot sits outside it. That distinction matters for anyone reading a vendor's category name as a synonym for every automated client, and it is the same distinction that makes a crawler token a claim about purpose rather than an identity. Verification is doing real work here: the redirect is keyed to Cloudflare's verified bot classification rather than to a user agent string, which is the only version of this that cannot be triggered by anything that types the right name. Of the fifteen crawler tokens this site has catalogued, six publish a User-Agent string a site can match on, and a string is not proof of anything.
The post is equally clear about the limits, and they are the part a reader should hold on to. It does not retroactively correct training data already ingested. It does not cover unverified crawlers outside the AI Crawler category. Humans and AI agents visiting a deprecated page are not redirected. Cross-origin canonicals are excluded by design, because Cloudflare says those are often used for domain consolidation rather than content freshness. And self-referencing canonicals do not trigger a redirect either, to avoid loops. Each of those exclusions is defensible on its own terms. Together they define a fairly small target: a page, on a paid Cloudflare plan, whose canonical tag names a different address on the same origin.
Flow: Request arrives at the edge to Verified AI Crawler category; Verified AI Crawler category (verified) to Cloudflare reads response HTML; Verified AI Crawler category (human or agent) to Page served unchanged; Cloudflare reads response HTML to Canonical names another URL; Canonical names another URL (yes) to 301 to the canonical URL; Canonical names another URL (self-referencing) to Page served unchanged.
Do AI crawlers follow canonical tags?
On the evidence Cloudflare published, not reliably, and the company measured this on its own documentation before building anything. Its deprecated Workers documentation carries a deprecation banner, a noindex meta tag and canonical tags pointing at the current pages. Every advisory signal on those pages says the same thing. Cloudflare reports that bots in its AI Crawler category visited developers.cloudflare.com 4.8 million times over the last 30 days and consumed deprecated content at the same rate as current content, and states plainly that the advisory signals made no measurable difference. It gives the March 2026 breakdown for legacy Workers documentation as roughly 46,000 crawls by OpenAI, 3,600 by Anthropic and 1,700 by Meta. Those are Cloudflare's figures from Cloudflare's own property, and this site has not reproduced them.
The reason is structural rather than a failure of manners by any particular crawler. A canonical tag was never an instruction. Google's canonicalization documentation, carrying Last updated 2026-07-10 UTC, opens by saying you can indicate your preference using a number of methods, and then ranks those methods by how strongly they can influence canonicalization: a redirect is a strong signal that the target should become canonical, a rel=canonical annotation is a strong signal that the specified URL should become canonical, and sitemap inclusion is a weak signal. Preference and signal are the operative words in a document written by the company with the most reason to want the tag obeyed.
This is a pattern that recurs across every file a site uses to talk to automated clients, and it is worth naming because it predicts which controls work. A sitemap entry is an assertion a site makes about itself and carries no enforcement. A canonical tag is a preference. By contrast the things that actually change what a client receives are transport level: a status code, a redirect, a refusal. That is why a 404 on robots.txt allows every crawler while a 503 blocks them all, with no negotiation and no interpretation involved. What Cloudflare has done is move the canonical tag across that line for one category of client. The tag stays advisory in the HTML, and the edge answers with a 301 anyway, which is not advisory at all.
| Signal | What it is | Force claimed by its publisher |
|---|---|---|
| Redirect | An HTTP status and a Location header | Strong signal |
| rel=canonical annotation | A preference expressed in the HTML | Strong signal, not a directive |
| Sitemap inclusion | A URL listed in an XML file | Weak signal |
| RFC 6596 | The canonical link relation itself | Informational, not Standards Track |
| Redirects for AI Training | A 301 issued at the edge from that tag | Enforced for verified AI training crawlers |
What the canonical tag looked like on five real pages
The feature reads something that has to already be there, so the useful question for a site owner is what is actually in their head element. This scanner stores complete raw and rendered captures for a small set of pages, and five of them are real sites rather than synthetic fixtures: astro.build, gymshark.com, the MDN HTML documentation, webflow.com, and the English Wikipedia article on web crawlers. All five were captured on 15 July 2026, all five returned HTTP 200, and none of the five went through a redirect on the way. That is the same five page capture earlier posts here have counted meta elements and crawl paths over, and it is a small sample stated as one.
Counted with an HTML parser over the stored raw response, those five pages carry 217 link elements between them. The distribution is lopsided in a way that says something about what the element is mostly used for now: 86 on the Wikipedia article, 78 on the Gymshark homepage, 25 on MDN, 19 on Webflow and 9 on astro.build. Most of that volume is not about canonicalization at all. Wikipedia's 86 include 47 carrying rel="mw-deduplicated-inline-style" and 26 carrying rel="mw:pageprop/category", which are MediaWiki internals. Gymshark's 78 include 40 preload hints and 23 hreflang alternates; Webflow's 19 include four hreflang alternates.
Exactly five of the 217 are canonical elements. One per page, no page carrying two, no page carrying none. Every one of the five was present in the raw HTTP response, before any JavaScript ran, and comparing the raw capture against the rendered capture shows no page whose canonical was injected, changed or removed by rendering. That is the good news in this measurement and it deserves saying, because it is not what happens to every signal: a page can lose most of its text to hydration and score the same as a page with no text at all. On this small sample the canonical element behaves the way prose parity asks everything to behave. It arrives in the bytes. A client without a browser can read it, which is the precondition for any edge feature reading it too.
Two of the five differed from the served URL by one character
Self-referencing is the word the feature turns on, and it is a string comparison somewhere. Three of the five canonical values are byte-identical to the URL the page was served at: astro.build, MDN and Wikipedia. Two are not. Gymshark's homepage was served at https://www.gymshark.com/ and its canonical element names https://www.gymshark.com. Webflow's homepage was served at https://webflow.com/ and its canonical element names https://webflow.com. In both cases the difference is a trailing slash, and in both cases the intent is obviously self-reference rather than a pointer somewhere else.
Whether that intent survives a machine reading is not something this measurement can settle, and it would be dishonest to pretend otherwise. Under RFC 3986 an empty path and a path of "/" are distinct strings that most implementations normalise to the same resource for an http URL, and every sensible implementation of a feature like this would normalise before comparing. Cloudflare's announcement does not describe its normalisation step, this scanner did not send a request through Cloudflare's edge to find out, and no claim is made here about what Cloudflare would do with either page. What can be said is narrower and still useful: two of five real homepages carry a canonical value that a naive string equality check would call non-self-referencing, so the correctness of this feature on ordinary sites rests on a normalisation detail that is not documented in the announcement.
One caveat belongs with these figures. The Gymshark capture was truncated at 3 MiB, which is a cap this scanner applies and which is a real limit on what any client reads: Googlebot's own documentation, carrying Last updated 2026-02-03 UTC, says it crawls only the first 2MB of a supported file type, and every fetcher draws that line somewhere. Truncation does not affect the canonical finding, because the head element and its canonical arrived well inside the cap, but a count of elements on a truncated document is a count of what was captured rather than of what the page contains, and that page is a large Shopify storefront of the kind our Shopify guidance is written for. The four other captures were not truncated.
URL the page was served at
- https://www.gymshark.com/
- https://webflow.com/
- Trailing slash present in both
- Three other pages matched exactly
rel=canonical value in the HTML
- https://www.gymshark.com
- https://webflow.com
- Trailing slash absent in both
- Self-reference is the evident intent
Google recommends the exact tag that switches the feature off
Here is the part that is inconvenient for the feature, and it is not a defect in it. Google's canonicalization documentation, the same page carrying Last updated 2026-07-10 UTC, lists among its best practices: do include a rel=canonical link on the canonical page itself, also known as a self-referential canonical. That advice is widely followed, and it is why the measurement above came out the way it did. Every one of the five pages carries a self-referential canonical. Cloudflare excludes self-referencing canonicals from redirection to avoid loops, which is correct behaviour and the only sane choice. The consequence is arithmetic: on those five pages, Redirects for AI Training would have issued zero redirects.
That is not an argument against the feature. It is an argument about who it is for. The population it acts on is pages whose canonical names a different URL on the same origin, which in practice means deprecated documentation, consolidated product pages, printer and AMP variants, and paginated or faceted URLs that fold into a parent. Cloudflare's own use case is exactly the first of those, and a documentation site with years of superseded versions is close to the ideal customer for it. A five page marketing site whose every canonical is self-referential gets a toggle that changes nothing, and would be misled by any write-up implying otherwise.
There is a broader point here about signals a site can send to an automated client, and it is the same one this site keeps arriving at from different directions. A page's machine-readable head is mostly assertions, and the assertions do not enforce themselves. Structured data is the clearest case: on this same five page capture, three of the five pages carried JSON-LD and one held 96 of the 102 nodes, which tells you the markup was present and tells you nothing about what any consumer did with it. A canonical tag has the same property. The interesting thing about Cloudflare's feature is not that it improves the tag, because it does not touch the tag. It is that it moves the enforcement point from the consumer, which no site controls, to the edge, which some sites do.
-
Self-referential canonicalNo redirect Excluded to avoid loops. This is the shape Google's documentation recommends on the canonical page itself, and the shape all five captured pages carried. -
Non-self-referencing, same origin301 issued The case the feature exists for: a superseded page whose canonical names its current replacement on the same site. -
Cross-origin canonicalNo redirect Excluded by design, because Cloudflare says these are often used for domain consolidation rather than content freshness. -
No canonical elementNo redirect There is nothing for the edge to read, so the response is returned unchanged.
What to check on your own site, and what this post did not measure
The useful takeaway is not a recommendation to enable anything, because whether this feature helps you is decided entirely by facts about your own URLs. Four checks answer it, and all four are things you can run without buying anything.
Count your non-self-referential canonicals first. If that number is zero, the toggle is a no-op on your site and nothing else in this post applies to you. Then confirm your canonical tags are in the HTML rather than written by JavaScript, because an edge reading the response body sees what the response body contains and nothing later. Then check that the pages you care about are reachable at all, since a canonical tag on a page a crawler is forbidden to fetch changes nothing: testing robots.txt per crawler token is the prior question, and it is prior for every AI client rather than just this one. And keep an eye on the token lists themselves, which move: Cloudflare will now generate a robots.txt whose AI crawler list changes without you, and a category defined by a vendor is a category that vendor can redefine.
What this post did not measure should be stated as plainly as what it did. Lantad did not test Cloudflare's feature, did not send a request through an edge with it enabled, and cannot confirm any of Cloudflare's reported figures, which are attributed to Cloudflare throughout. Lantad did not determine how the trailing slash cases would be normalised. Five pages is a small sample and it is not a survey of the web: it is five real pages, named, captured on one date, with the numbers reproducible from the stored bytes. This site does not publish a percentage of the web from five captures, and it withholds a grade when it cannot measure the thing it is grading for the same reason. What the five do establish is that the canonical element is present, singular, and in the raw HTML on every one of them, which is a better state than several of the other head signals this capture has been counted for, and that its self-referential form is the norm rather than the exception. Anyone measuring AI visibility should treat an edge feature that acts on a tag most pages carry in a shape it ignores as a narrow tool with a real use, not as a switch that changes what crawlers do with a site.
- Count canonicals that name a different URL The feature acts only on non-self-referencing canonicals on the same origin. Zero of those means the toggle changes nothing. All five pages captured 15 July 2026 were self-referential.
- Confirm the canonical is in the raw HTML An edge reads the response body. On the five captured pages every canonical was already in the raw response and none was altered by rendering.
- Check robots.txt per crawler token first A canonical tag on a page a crawler may not fetch changes nothing, and the AI Crawler category named in the announcement covers GPTBot, ClaudeBot and Bytespider rather than every automated client.
- Expect the vendor category to move Cloudflare defines which bots count as AI training crawlers, so the set the feature applies to can change without any edit on your side.
- Cloudflare's own figures reproduced here The 4.8 million visits and the March 2026 crawl counts are reported from Cloudflare's announcement of 17 April 2026 and were not measured or verified by Lantad.
- Trailing slash normalisation tested Two of five canonicals omitted a trailing slash the URL carried. How any edge implementation normalises that before comparing was not tested and is not documented in the announcement.
Lantad
Published .
The rel=canonical tag has been advice since April 2012. RFC 6596 describes it as a way to designate one address as preferred over resources with duplicative content, and the document is filed as Informational rather than Standards Track. Search engines have treated it as a signal ever since, which is to say a page can name a different address and a client is free to decline. On 17 April 2026 Cloudflare published something that changes what that tag does for one class of client: Redirects for AI Training reads the canonical tag already sitting in a page's HTML and, for verified AI crawlers in its training category, converts it into an HTTP 301 before the response leaves the edge.
Common questions
Do AI crawlers follow canonical tags?
Not reliably, on the evidence Cloudflare published on 17 April 2026. It reported that bots in its AI Crawler category visited developers.cloudflare.com 4.8 million times over 30 days and consumed deprecated content at the same rate as current content, despite those pages carrying a deprecation banner, a noindex meta tag and canonical tags pointing elsewhere, and said the advisory signals made no measurable difference. The tag was never a directive: Google's canonicalization documentation, Last updated 2026-07-10 UTC, calls a rel=canonical annotation a strong signal and describes the whole mechanism as indicating your preference.
What does Cloudflare's Redirects for AI Training actually do?
It reads the rel=canonical tag already in your HTML and, when a request comes from a verified crawler in Cloudflare's AI Crawler category and that tag names a different URL on the same origin, it returns an HTTP 301 to the canonical URL instead of the page. Cloudflare names GPTBot, ClaudeBot and Bytespider in that category and distinguishes it from its AI Assistant and AI Search categories. It is a toggle in AI Crawl Control on paid plans, it does not change training data already ingested, and it does not redirect humans, AI agents or unverified crawlers.
Will the feature do anything if my canonical tags point at their own pages?
No. Cloudflare excludes self-referencing canonicals from redirection to avoid loops. That matters more than it sounds, because Google's canonicalization documentation recommends including a rel=canonical link on the canonical page itself, so most well-maintained pages carry exactly the shape the feature ignores. On the five pages Lantad captured on 15 July 2026, all five canonicals were self-referential, so the feature would have issued no redirects on any of them.
Does a canonical tag need JavaScript to be read?
It should not, and on this sample it did not. Lantad counted 217 link elements in the raw HTTP responses of five pages captured on 15 July 2026 and found five canonical elements, one per page, every one present in the raw HTML before any JavaScript ran, with no page whose canonical was added, altered or removed by rendering. That is the state any edge feature or non-rendering crawler needs, since both read the response body rather than a hydrated DOM.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
Prose Parity
Prose Parity is Lantad's metric for the share of a page's rendered, human-visible text that is present in the raw HTML an AI crawler receives, where 100% means a crawler sees everything a human visitor sees.
Structured data for AI (schema markup)
Structured data is schema.org markup, usually written as JSON-LD, that labels a page's facts in a machine-readable format so AI systems can extract and attribute them without having to interpret the surrounding prose.
85 meta elements in five page heads, and two addressed a crawler
Counted with an HTML parser on 24 August 2026 over five real pages this scanner captured and stored on 15 July 2026, the head of those documents holds 85 meta elements. Two are a robots meta tag, and both of those carry nothing but an image preview size. Five of the five tell a social network how to draw a card.
JavaScript added no new crawl paths, and rendering dropped nine locale links
Counted with an HTML parser over five real pages this scanner captured and stored on 15 July 2026, the raw HTTP response held 991 distinct same-host URLs and the rendered DOM held 982. Three URLs existed only after JavaScript ran, and every one of the three was a rewrite of a link already present. Nine of the twelve that disappeared were one page's translations of itself.
Cloudflare will generate your robots.txt, and its token list changes without you
Cloudflare announced Bot Preference Sync on 21 August 2026: a setting on every plan that writes your AI bot policy for Search, Agent and Training into robots.txt, prepends it above whatever file you already have, and refreshes the crawler names inside it from Cloudflare's own bot database. RFC 9309 says a crawler merges every group matching its token and reads the wildcard group only when none matches, so a prepended group changes which rules a named crawler obeys.
