BlogFindings

Crawl to refer ratio: Cloudflare says its own number may overstate

The crawl to refer ratio divides a platform's crawler requests for HTML by the HTML requests whose Referer header names that platform. Cloudflare, which publishes it, states that traffic from Claude's native app carries no Referer header and that the calculations may therefore overstate the ratios, by an amount it says is unclear.

31 August 2026 16 min read Lantad

This is a report of somebody else's measurement, not one of ours. Lantad computed none of these ratios, holds no network telemetry, and cannot reproduce any figure below, so every number here is attributed to Cloudflare and carries the date Cloudflare attached to it. What this site can add sits next to the number rather than inside it: what a ratio published per platform does to the fact that robots.txt is written per token, and what somebody should go and check on their own site once they have finished reading an aggregate about everybody else's. How a capture is taken and scored here is published in full, and it measures a different thing from what Radar measures.

In short

The crawl to refer ratio is Cloudflare's metric, described in its own post of 1 July 2025, and it divides the requests from a platform's crawler user agents that returned Content-type: text/html by the requests for HTML content whose Referer header named a hostname belonging to that same platform, normalised so the result reads as crawls per single referral.
For 19 to 26 June 2025 that post published Anthropic at 70,900:1 and Mistral at 0.1:1, the two ends of its table, meaning Mistral referred roughly ten visits for every crawl request it made while Anthropic made roughly 71,000 crawl requests for each visit it referred.
Cloudflare states in the same post that traffic referred by Claude's native app does not include a Referer header, that it believes the same holds for other native apps, and that because the referral counts therefore include only web based tools the calculations may overstate the respective ratios, but that it is unclear by how much.
A crawl to refer ratio names a company while robots.txt names a token, and those do not correspond: the crawler registry in this repository, read on 31 August 2026, holds 15 tokens across 9 vendors, with OpenAI carrying 3 of them and Anthropic 4, divided 9, 3 and 3 between training, search and user agent purposes.
The ratio is computed over requests crossing Cloudflare's network rather than over any one site, so it publishes no figure for a single domain and reports nothing about whether a particular page is readable by any crawler that fetches it.

Anthropic, 70,900:1

Roughly 71,000 crawl requests for HTML
for each single referred visit
Cloudflare's figure, 19 to 26 June 2025

Mistral, 0.1:1

Roughly ten referred visits
for each single crawl request
Cloudflare's figure, 19 to 26 June 2025

The two ends of the table in Cloudflare's post of 1 July 2025, covering 19 to 26 June 2025. Cloudflare's figures, reported here rather than reproduced: Lantad holds no network telemetry and computed neither value.

What is the crawl to refer ratio?

The definition is arithmetic and worth stating in full, because most of the confusion about the metric comes from people carrying an intuition about it rather than the formula. Cloudflare divides the total number of requests from the user agents associated with a given search or AI platform where the response carried Content-type: text/html, by the total number of requests for HTML content where the Referer header contained a hostname associated with that same platform. The result is then normalised to a single referral, which is why the figures read as ratios ending in one. A published value of 70,900:1 means roughly 71,000 crawl requests for HTML arrived from that platform's crawlers for each one visit that arrived carrying that platform's hostname in its Referer header.

Both halves are counts of HTTP requests seen at an edge, which is the first thing to hold on to. The numerator is not pages, documents or articles: it is requests that returned HTML, so a crawler fetching the same page four times contributes four. The denominator is not people, sessions or customers: it is requests for HTML carrying a particular Referer value. Neither half knows anything about what the crawler did with the bytes, whether the content was used in an answer, or whether the visit that arrived was worth anything. The ratio is a traffic accounting identity, and it is a good one, but it is answering a narrower question than the phrase "takes and gives back" suggests when the number gets repeated.

Cloudflare's post introducing the metric is dated 1 July 2025 and its worked table covers 19 to 26 June 2025, with a broader discussion across nearly the first four weeks of that month. The two values it publishes at the ends of the range are Anthropic at 70,900:1 and Mistral at 0.1:1. That second one is the more interesting of the pair and it is usually left out of the quoting: a ratio below one means the platform referred more visits than it made crawl requests, so Mistral was sending roughly ten visits for every request it made. The metric is not a scale of badness with everybody on it. It is a spread of about six orders of magnitude, and there is a platform at the bottom of it doing the opposite of the thing the metric is usually cited to prove.

The dates matter more than they usually would. Those figures describe one week in June 2025, and this post is being written on 31 August 2026, so they are a year and two months old. Radar recomputes the ratio on a rolling basis and the current values are not the ones above. Lantad attempted to read the live Radar page on 31 August 2026 and received an HTTP 403 rather than a document, so no current figure is quoted here at all. Numbers circulating in secondary coverage of more recent periods were not confirmed at a primary source for this post and are therefore absent from it, which is the same standard applied to every AI crawler figure this site publishes.

Flow: Requests crossing the edge (numerator) to Platform crawler UA, HTML response; Requests crossing the edge (denominator) to HTML request, Referer names platform; Platform crawler UA, HTML response to Divide, normalise to one referral; HTML request, Referer names platform to Divide, normalise to one referral; Divide, normalise to one referral to Crawl to refer ratio.

The calculation as Cloudflare's 1 July 2025 post describes it. A diagram of a published formula, not a measurement Lantad performed and not a pipeline this site runs.

The limit Cloudflare states about its own number

The denominator is a count of requests whose Referer header named the platform. That header is set by the client, and not every client sets it. Cloudflare says so directly in the post, and the sentence is the most useful one in the whole piece: traffic referred by Claude's native app does not include a Referer header, and Cloudflare believes the same holds true for traffic generated from other native apps as well. The consequence it draws is stated in the same breath, that because the referral counts only include traffic from the web based tools of these providers, the calculations may overstate the respective ratios, though it is unclear by how much.

Read that as arithmetic and it is not a small caveat. Referrals sit in the denominator, so referrals that go uncounted make the denominator smaller and the ratio larger. Every visit arriving from a desktop or mobile app rather than a browser tab is invisible to the measurement while the crawl requests that fed it are fully counted. The direction of the error is known and only the size is not, which is a better position than most published metrics manage, and it is exactly the sort of thing that gets stripped out when a figure travels. A ratio of 70,900:1 quoted with no mention that its denominator is missing an unknown share of real referrals is a different claim from the one Cloudflare made.

There is no way to correct for it from outside, either, and this is worth being precise about rather than hand waving at. The missing referrals are missing because a header was never sent, so there is no record at the edge, in a log file, or in any analytics product to recover them from. The only parties who could size the gap are the platforms themselves, since they know how their app traffic splits against their web traffic, and none of them publishes that split. Nothing in OpenAI's crawler documentation, Anthropic's page on its crawler and how to block it or Perplexity's bots guide addresses referral behaviour at all. Those pages document tokens, user agent strings and blocking, which is a different subject.

This is the same class of problem as the one behind the claim that a user agent is a claim and not an identity: both halves of this ratio rest on strings a client chooses to send. The numerator depends on the crawler naming itself honestly in its user agent, and only 6 of the 15 tokens catalogued here publish a User-Agent string a site can match on. The denominator depends on the browser sending a Referer. Neither is a property of the world that a measurement can simply observe. Both are self reports, and a metric built by dividing one self report by another inherits the honesty of both.

Crawler sends its documented UA Counted The request lands in the numerator for that platform.
Crawler sends an unmatched UA Uncounted Numerator falls, so the published ratio understates for that platform.
Referral from a web tool Counted Referer header names the platform hostname and lands in the denominator.
Referral from a native app Uncounted Cloudflare states no Referer header is sent, so the denominator falls and the ratio overstates by an amount it calls unclear.

What each half of the ratio depends on, and what happens to the result when the dependency is not met. The overstatement direction and Cloudflare's unclear by how much are quoted from its 1 July 2025 post; the rest describes the mechanism.

A platform is not a crawler, and robots.txt is written per token

The second limit is structural rather than stated, and it is the one that matters most to somebody deciding what to do about a ratio they have just read. The metric names a company. A site's controls do not name companies, they name product tokens, and the mapping between the two is not one to one. The crawler registry in this repository, read out of core/src/bots.ts on 31 August 2026 and previously published here on 29 August 2026, holds 15 tokens across 9 vendors. OpenAI accounts for 3 of them and Anthropic for 4. Those counts are a record maintained by hand from vendor documentation rather than a discovery, and they are a design decision about what this product models rather than a finding about how many crawlers exist.

The registry also divides those 15 tokens into three purposes, 9 training, 3 search and 3 user agent, and that split is precisely what a single company level ratio dissolves. A training crawler that reads a page and contributes it to a model refers nothing by design and never will, so it can only ever push a ratio upward. A search time fetcher exists to produce the answer a citation sits in, and a user triggered fetch happens because a person asked for that page in that moment. Collapsing all three into one figure per company puts the crawler that cannot possibly refer traffic in the same average as the one whose entire job is to. The comparison across companies then partly reflects how each vendor happens to divide its work between tokens rather than how generous each one is.

That distinction is not academic, because it is the level at which a site owner actually decides anything. RFC 9309, the standard that governs robots.txt, has a crawler merge the groups matching its own product token and fall back to the wildcard group only when no group matches it. So a rule is written against GPTBot or ClaudeBot or PerplexityBot, one token at a time, and a site can allow the search fetcher while disallowing the training crawler of the same company. A ratio that names the company gives no guidance about which of its tokens to write a rule for. The full token list this scanner works from is published, and our own robots.txt and crawler identity is too.

There is a vocabulary mismatch underneath this as well. Cloudflare's table names Mistral, and Mistral has no entry in the registry described above, because that registry covers the crawlers this product probes rather than every platform Radar can see traffic from. Neither list is wrong and neither is complete. They were built for different jobs, which is a good reason not to read a platform's absence from either one as meaning anything about the platform. The related problem of what a bot declares about itself versus what it can prove was covered separately in what a bot declares and what it verifies.

Vendor	Tokens in the registry	Purposes covered
OpenAI	3	training, search, user agent
Anthropic	4	training, search, user agent
Perplexity	2	search, user agent
Google	1	training
Apple	1	training
Meta	1	training
ByteDance	1	training
Common Crawl	1	training
Amazon	1	training

The crawler registry in core/src/bots.ts, read on 31 August 2026, grouped by vendor. A record maintained by hand from vendor documentation and a design decision about what this product models, not a census of crawlers that exist.

What the ratio cannot tell you about your own site

The third limit is the plainest and the easiest to forget. The ratio is computed over requests crossing Cloudflare's network, which is an enormous and genuinely useful vantage point, and it is not your vantage point. There is no per site crawl to refer ratio in what Cloudflare published, and the aggregate does not decompose into one. Your site's crawler traffic depends on how many pages you have, how often they change, whether anything links to them, and what your own robots.txt says, and none of those resemble the network average. A figure describing the whole of a network is a fact about the network.

It also says nothing at all about the thing this product exists to measure, which is whether a crawler that does arrive can read the page it fetched. A ratio counts requests and responses. It does not look inside the response. A site with an unremarkable ratio can serve an empty shell to a crawler that executes no JavaScript, and the request still counts as an HTML response in the numerator exactly as a full article would. The gap between what a browser shows and what arrives in the initial HTTP response is prose parity, it is not visible in any traffic metric, and it is the difference between being crawled and being readable. What GPTBot actually sees on a page is a separate question from how often it asks.

The traffic consequences of blocking are a real subject and they have been measured elsewhere, which is worth separating from this metric rather than blending into it. Published work has found that blocking AI crawlers cost large publishers about 7 percent of traffic, and that AI Overviews cut English Wikipedia traffic by about 15 percent. Those are findings about outcomes at named sites over stated periods. A crawl to refer ratio is an input to that argument and not a substitute for it, and the crawl budget advice that gets applied to AI crawlers largely predates the question.

None of this makes the metric bad. It makes it a network level diagnostic that belongs in a policy argument, which is what Cloudflare built it for and how its own post frames it. The failure is only in the transfer: reading a published aggregate about an industry and treating it as a measurement of your own domain. That transfer happens constantly with AI visibility numbers generally, and it is the reason this site publishes what its scans can and cannot establish alongside the scans.

How much does platform X crawl the web relative to referrals Answered This is the metric, at network scale, for the period stated.
How much does platform X crawl my site Not answered No per site figure is published and the aggregate does not decompose to one.
Can a crawler read my pages once it arrives Not answered The ratio counts requests and never inspects the response body.
Which robots.txt token should I write a rule for Not answered The figure names a company; rules name tokens, and a company can hold several.

Questions a reader typically brings to the crawl to refer ratio, and whether the metric as Cloudflare defines it can answer them. A reading of the published definition, not a test of any site.

What to check on your own site instead

The useful move after reading a ratio is to stop looking at aggregates and go and establish four things about your own domain, all of which are cheap and none of which requires network telemetry. They are ordered here from the one that most often turns out to be wrong to the one that is most often already fine.

First, which tokens your robots.txt actually matches. Because RFC 9309 merges the groups matching a crawler's product token and only falls back to the wildcard when nothing matches, a file that names a crawler at all can end up granting that crawler more than the wildcard group does, which is the opposite of what the author usually intended. Reading the file is not the same as evaluating it against a specific token, and a robots.txt tester that evaluates per crawler answers a different question from opening the file and looking at it. Second, whether the tokens you meant to name are the tokens the vendors currently publish. Product tokens change, and Google's crawler overview is the kind of page that has to be reread rather than remembered.

Third, whether a page is readable once fetched, which is where a ratio has nothing to offer and where most sites that have a problem have this one. If the text a reader sees is assembled by JavaScript after load, a crawler that does not execute it receives markup with no prose in it, and no amount of crawl traffic converts that into a citation. Fourth, whether the page states plainly what it is about, which is the ordinary generative engine optimization work of headings, structure and machine readable claims, and which matters only after the first three are true. Guides for the individual platforms exist for getting cited in ChatGPT and getting cited by Perplexity.

The honest summary of this post is short. Cloudflare publishes a well defined metric, states a real limitation in its own number, and gives a figure per platform for a stated period. It is worth quoting, with its date and with its caveat attached, and it is worth remembering that it was measured somewhere other than your server. Everything it cannot answer is answerable, and answering it starts on your own domain rather than in somebody else's aggregate. The Cloudflare post remains the place to read the definition rather than any summary of it, this one included.

Evaluate robots.txt against each crawler token, not by reading the file RFC 9309 merges matching groups and uses the wildcard only when no group matches the token.
Confirm the tokens you named are the ones vendors currently publish Tokens are renamed and added; a stale group matches nothing and blocks nothing.
Check the page is readable in the initial HTTP response A crawler that runs no JavaScript reads what the server sent, not what a browser assembles.
Only then work on how clearly the page states what it is about Structure and machine readable claims matter after a crawler can reach and read the text.

Four checks a site owner can run on their own domain after reading a network level ratio. An ordering of ordinary checks, not a scored output and not a measurement of any site.

Written by

Lantad

Published 31 August 2026.

The crawl to refer ratio is the figure that anchors most arguments about AI crawling. It is one number per platform, Cloudflare publishes it on its Radar service, and it gets quoted whenever somebody wants to show that an AI company takes a great deal of content and sends back very little traffic. The attention is deserved, because the spread it exposes is genuinely large. It also carries three limits. One of them Cloudflare states plainly in the post that introduced the metric, and the other two follow from where the measurement is taken. A reader quoting the number without them is quoting something narrower than they think they are.

Common questions

What is a good crawl to refer ratio?

Cloudflare publishes no threshold, and the metric is network wide rather than per site, so there is no published figure a single domain can be measured against. What its 1 July 2025 post does show is the spread: Anthropic at 70,900:1 and Mistral at 0.1:1 for 19 to 26 June 2025, with a ratio below one meaning the platform referred more visits than it made crawl requests.

Can I calculate the crawl to refer ratio for my own site?

You can compute the same arithmetic from your own server logs, by counting HTML responses served to each platform's crawler user agents and dividing by HTML requests whose Referer names that platform. It will not be comparable with Cloudflare's published figures, because those cover a different population of requests, and it inherits the same gap: referrals from native apps carry no Referer header and will be missing from your denominator too.

Why does Cloudflare say the ratios may overstate?

Because referrals sit in the denominator and some are never counted. Cloudflare states that traffic referred by Claude's native app does not include a Referer header and that it believes the same holds for other native apps, so only web based referrals are counted. A smaller denominator produces a larger ratio, and Cloudflare says it is unclear by how much.

Does a high crawl to refer ratio mean I should block that crawler?

The ratio does not answer that, because it describes a network rather than your site and does not distinguish a company's training crawler from its search fetcher. Those are separate robots.txt tokens with different consequences: blocking a training crawler withholds content from model training, while blocking a search time fetcher can remove a site from the answers where citations appear.

See what AI can read on your site

Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.

Scan your site free How we measure

In short

What is the crawl to refer ratio?

The limit Cloudflare states about its own number

A platform is not a crawler, and robots.txt is written per token

What the ratio cannot tell you about your own site

What to check on your own site instead

Common questions

See what AI can read on your site

Related

AI crawler

AI visibility

Generative Engine Optimization (GEO)

A user agent is a claim, not an identity

LLM crawler detection: a bot declares three things and proves one

Blocking AI crawlers cost large publishers about 7 percent of traffic