BlogFindings
Cloudflare pay per use: the crawl is countable and the use is not
Cloudflare announced on 1 July 2026 that it is shaping pay per crawl into pay per use, with experiments running alongside Ceramic.ai and You.com. A crawl arrives as a request in your own access log. A use happens inside an answer engine, where the site that produced the page has nothing to count.
This post reports somebody else's announcement. Lantad did not build any of this, has measured none of its adoption and holds no invoice from any party to it, so nothing below is a measurement of ours. What this site can add is the part a scanner is qualified to add: the difference between a unit you can count in your own logs and a unit that exists only inside somebody else's system, and which of the two a scan of your pages is able to reach.
In short
- Cloudflare announced on its own blog on 1 July 2026 that it is starting to shape pay per crawl into Cloudflare pay per use, with experiments running alongside Ceramic.ai and You.com, and it calls the arrangement an experiment rather than a shipped feature.
- Cloudflare states in that post that more than 50 percent of crawl traffic from good bots goes to re-fetching pages that have not changed, and that more than 20 percent of the web sits behind its network, giving neither a method nor a period.
- The Pew Research Center analysis Cloudflare cites, published 22 July 2025 over the March 2025 browsing data of 900 US adults, recorded a traditional search result clicked on 8 percent of visits where an AI summary appeared against 15 percent where none did, and a link inside the summary on 1 percent.
- A crawl leaves an HTTP request in a site's own log and a use happens inside an answer engine, so a per-use price is settled on a quantity the site owner cannot count and the paying party reports.
- Lantad has measured no pay per crawl or pay per use adoption and could not measure it from outside, because a scanner is never a client with a billing relationship and so is never served the branch a paying crawler is served.
| Unit | Where the event happens | Leaves a trace on your server | Who produces the count |
|---|---|---|---|
| A crawl | An HTTP request to your origin or edge | Yes | Either side, from one connection |
| A referred visit | A person or agent arriving from an answer | Sometimes | You, where a Referer header is sent |
| A use inside an answer | The engine's own retrieval and generation | No | The engine, reported to you |
What is Cloudflare pay per use, and what does it price?
Cloudflare published Making AI search smarter on 1 July 2026, credited to Matthew Conroy, and the page carries a modified date of 15 July 2026. It announces two things that are easy to run together and should not be. The first is a research programme in which Cloudflare would tell answer engines which pages have genuinely changed, so that crawlers stop re-fetching the ones that have not. The second is the pricing change: in its own words, it is starting to shape Pay Per Crawl into Pay Per Use, with experiments running alongside two named AI companies, Ceramic.ai and You.com.
The reason it gives for changing the unit is the clearest sentence in the post. Crawling, it says, is a crude measure of value, because a single page might be crawled once and then cited in thousands of answers, or crawled over and over and never used at all. That is an accurate description of the problem. A crawl counts arrivals, and arrivals are not consumption. Anyone who has watched a crawler take the same unchanged page forty times in a week and produce nothing recognises both halves of it.
What replaces the crawl is not one model but several. Ceramic is described as having built a pay-per-query model, under which publishers who opt in can be paid when their content appears in Ceramic's search results, and its founder and chief executive Anna Patterson is quoted in the post saying that bringing that model to Cloudflare's network ensures "millions of content owners can seamlessly opt in to be compensated every single time their content appears in our search results". You.com is described differently: agents can pay on demand for a specific piece of premium content they need, without any upfront commitment. Cloudflare adds that new payment models are being tested, naming Pay per Query and Pay per Result as examples, and says it has the infrastructure to support them all.
Two qualifications belong with all of that and both are Cloudflare's own. It writes that it wants to be honest that this is an experiment and that there is a lot to learn, including exactly how it holds up at the scale of the Internet. And the older mechanism has not been withdrawn: pay per crawl still answers an unpaid crawler with HTTP 402 and a crawler-price header, and Cloudflare's product documentation still carries the feature under that name. No directive expresses any of this, and nothing in a site's robots.txt changes because the announcement was made.
-
Research programmeAnnounced Signals about what has actually changed, offered to answer engines so they re-fetch less. Cloudflare says it plans to make the capability broadly available later this year. -
Pay Per UseExperiment Cloudflare writes that it is starting to shape Pay Per Crawl into Pay Per Use, and separately that it wants to be honest that this is an experiment. -
Ceramic.aiNamed partner Described as having built a pay-per-query model under which publishers who opt in can be paid when their content appears in Ceramic's search results. -
You.comNamed partner Described as letting agents pay on demand for a specific piece of premium content they need, with no upfront commitment. -
Pay Per CrawlStill documented Cloudflare's product documentation still describes the feature under that name, and the announcement does not say it is being withdrawn.
Why the crawl was a poor unit for either side
Cloudflare gives one figure for how much crawling is wasted and one for the vantage point it claims. It states that its data suggests more than 50 percent of crawl traffic from good bots goes to re-fetching pages that have not changed, and that more than 20 percent of the web sits behind its network. Neither figure arrives with a method, a measurement period or a sample, so both are assertions by one company about its own traffic rather than results a reader can check, and this post treats them that way. What the first would mean if it holds is plain enough: roughly half of what a site spends serving well behaved crawlers buys nobody anything, and a signal saying nothing here has changed would remove that spend on both sides at once.
The demand side of the argument rests on somebody else's work, and that is the half worth opening rather than relaying. Cloudflare cites a 2025 Pew Research Center study for the claim that AI summaries suppress clicks. The Pew analysis, published on 22 July 2025 by Athena Chapekis and Anna Lieb, used the browsing data of 900 US adults on the KnowledgePanel Digital panel, covering every URL they visited during March 2025, and contains 68,879 unique Google searches of which 12,593 produced an AI summary. Users who encountered a summary clicked a traditional search result on 8 percent of visits. Users who did not encounter one clicked a result on 15 percent of visits, which Pew describes as nearly twice as often. A link inside the summary itself was clicked on 1 percent of visits. Sessions ended outright on 26 percent of pages carrying a summary against 16 percent of pages showing only traditional results.
Those are the figures at their source, and one detail is worth keeping because it is exactly the sort of thing that decays as a claim travels between documents. Pew's own comparison is 8 against 15, which it calls nearly twice; a relay that renders this as about half is close, and is not what the study says. The study also covers Google alone, by its own statement, because identifying AI-generated summaries on other engines was beyond what it could do. So it is evidence about one engine in one month from one panel, which is a great deal more than most numbers in this argument carry, and still narrower than the general proposition it is usually recruited to support.
The traffic side of the same economy has been counted from another angle, and the counting is hard there too. Cloudflare's own crawl to refer ratio divides crawler requests for HTML by the HTML requests whose Referer names that platform, and Cloudflare says plainly that its own number may overstate the result by an amount it cannot quantify. That is the same shape of problem arriving one layer down: the numerator is observable at the edge and the denominator depends on what a client chooses to send.
A use happens where you cannot see it
Here is the measurement problem stated plainly, and it is not an objection to the idea. A crawl is an event on your infrastructure. It arrives as an HTTP request, it lands in your access log with a claimed user agent and an address, and you can count it without anybody's cooperation or permission. That is why a price per crawl, whatever else is wrong with it, is auditable in principle: two parties are looking at the same event from opposite ends of one connection, and a dispute between them is a dispute about records that both hold.
A use is not that. When an answer engine grounds a sentence in your page, the event happens inside its retrieval and generation pipeline. Nothing about it touches your server. There is no request to log, no header to inspect, no artefact left behind on your host, which means the count that decides your payment is produced entirely by the party doing the paying. Cloudflare is candid that reporting is part of what is being offered rather than a property of the mechanism: content owners in the Ceramic programme are promised the top queries leading to their content appearing in search results, the specific webpage and snippet, and their average search result ranking position. That is genuinely more than most publishers get today from any engine. It is also a report, and a report is a claim by its author.
This is the same asymmetry that appears wherever a control tries to govern use rather than access, and this blog keeps arriving at it from different directions. A licence directive in robots.txt changes no access by its own admission. The Content Signals directives that one Cloudflare crawl endpoint honours are declarations of intent that nothing verifies against eventual behaviour. Web Bot Auth settles which operator is asking with a signature, and states nothing at all about what the identified party may then do with the page. In each case the layer available to a site owner was built to answer one question, whether to serve a request, and is now being asked a second question for which it holds no evidence.
None of that makes a per-use price wrong. Paying for outcomes rather than arrivals is the more honest of the two arrangements, and it is roughly where the advertising industry landed decades ago after the same argument. It does mean the verification problem is real and currently unsolved, and that a publisher signing up should know which number on the invoice they are able to check independently. First-party reporting is the only answer anybody has proposed, which is why the difference between an engine that publishes the grounding query behind a citation and one that publishes none matters more than it sounds.
Flow: Crawler requests a page (you count this) to Request in your access log; Request in your access log to Engine stores the content; Engine stores the content to Engine retrieves it at answer time; Engine retrieves it at answer time to Answer grounded in your page; Answer grounded in your page (only they see this) to Use counted and paid.
What a scan can and cannot observe here
Lantad has measured nothing about pay per use, holds no data on how many hosts price their content rather than blocking it, and could not collect that data from outside if asked. The reason is structural rather than a gap in the product. A priced crawl is served differently depending on whether the client holds a billing relationship, and a scanner is definitionally not that client, so every request it sends takes the unpaid branch and sees only what an unpaid crawler sees. Any figure anybody publishes about how much of the web is priced rather than refused has to come from inside a network that observes both branches, and that network is one of the parties to the arrangement.
What a scan does establish is the precondition, and it is worth separating from the commercial question because the two are constantly conflated in the same sentence. Before a page can be crawled, stored, retrieved, cited, priced or paid for, it has to be fetchable and readable by the client asking for it. That is the whole of what an external measurement is entitled to claim. What an AI crawler actually receives for a given URL is visible in the crawler view of a single page, and if the words a reader sees are not present in that response, no pricing arrangement downstream will produce a citation there is anything to pay for.
The access layer has moved this quarter as well, and it moves independently of any of this. From 15 September 2026, new domains onboarding to Cloudflare receive defaults under which bots classified as Training or Agent are blocked on pages that carry ads while Search stays allowed. Cloudflare will separately generate a managed robots.txt and refresh the crawler names inside it without you. Both of those change what a crawler is permitted to fetch, which is the layer RFC 9309 actually specifies, and both are observable from outside precisely because they act on the request rather than on what follows it.
The honest summary of the split is short enough to keep. Access is measurable by anyone holding an HTTP client. Use is measurable only by the party doing the using. Everything this site publishes sits on the first side of that line, including the aggregate findings on the research page, and no amount of product work moves it across.
A scan can observe
- Whether a named crawler token is allowed a path
- What the raw response holds before scripts run
- Which structured data types the page publishes
- Whether the page is reachable, and at what status
A scan cannot observe
- Whether an engine stored or retrieved the page
- Whether any answer was grounded in it
- Whether the host prices access rather than refusing it
- Any figure on an invoice between publisher and engine
What to check on your own site while this is an experiment
Nothing in this announcement is a setting anybody turns on today, and that is the first practical point. Cloudflare says the research capability is planned to be broadly available later this year, and describes the payment work as an experiment run with named partners. There is no directive to write, no file to publish, no header to send and no field in a dashboard. Anybody selling a pay per use readiness audit this month is selling a document.
What does repay attention is the property the two initiatives share, because it is the same property under both. The research programme wants to tell answer engines which pages have genuinely changed so they stop re-fetching the ones that have not. The payment work wants to reward the pages an answer actually used. Both favour a page that is fresh, distinguishable and cheap to read, and both penalise a page that is expensive to fetch and yields nothing new when fetched. Those are properties a site owner can check today, at no cost, with no commercial arrangement in place and no vendor's cooperation required.
So the list is the ordinary one, in the order the failures actually happen. Confirm the crawlers you care about are permitted, resolved per product token rather than for a wildcard group, which is what a robots.txt test answers and what our own crawler's conduct policy documents for the one this site runs. Confirm the page's words arrive in the first response rather than being assembled afterwards, since that is the difference between prose parity and a page a JavaScript-blind client reads as empty. Confirm the dates a machine reads agree with the dates a reader sees, because freshness is precisely the signal Cloudflare's programme proposes to sell on your behalf. Then aim at the engine your buyers actually use rather than at an average of all of them, since three of the eight engines in our registry cannot search the web at all.
The commercial question underneath, whether being paid beats being read, is genuinely open and this post takes no position on it. There is some evidence on the neighbouring question: pages from publishers holding an OpenAI licensing deal collected 10.2 ChatGPT citations each on average against 6.9 for unlicensed publishers in one joint study of 129.3 million citations, which is an association and not a mechanism. What is not open is which half of the arrangement a publisher can verify. Until a use leaves some trace on the site that produced it, answer engine optimisation means optimising against a number somebody else counts, and honest AI visibility work should say so rather than imply otherwise.
- Named crawlers permitted, per token Resolved against the site's robots.txt for each product token separately, rather than for a wildcard group.
- Words present before JavaScript runs The share of the rendered visible text that is already in the raw HTTP response.
- Structured data that labels the entity JSON-LD present, parseable, and describing the thing the page is about.
- Machine readable dates that agree with visible ones Freshness is the signal Cloudflare's research programme proposes to pass to answer engines.
- A pay per use setting to configure There is none. Cloudflare describes the payment work as an experiment with named partners and the research capability as planned for later this year.
Lantad
Published .
Every control this scanner measures answers a question about a fetch. A robots.txt rule names which clients may request which paths. A rule at the edge decides whether a request is served at all. A price on a crawl, which Cloudflare shipped in 2025, decides what a request costs. Cloudflare pay per use is the first of these to move the question past the fetch entirely, because it prices what an answer engine did with the page afterwards, and that turns out to be a different kind of quantity for anybody who wants to check it.
Common questions
What is Cloudflare pay per use?
Cloudflare pay per use is the direction Cloudflare said on 1 July 2026 it is taking pay per crawl: compensating content owners when an answer engine uses their content rather than when a crawler fetches it. Cloudflare describes it as an experiment run alongside two named AI companies, Ceramic.ai and You.com, rather than a shipped feature, and there is nothing for a site owner to configure. Pay per crawl itself remains documented under its own name.
How is pay per use different from pay per crawl?
The unit being priced. Pay per crawl prices an HTTP request, which arrives at your server and appears in your access log, so both parties can count it from opposite ends of the same connection. Pay per use prices what an answer engine did with the content afterwards, which happens inside that engine and leaves nothing behind on your host. Cloudflare's stated reason for the change is that a page might be crawled once and cited in thousands of answers, or crawled repeatedly and never used.
Can a scanner tell me whether my content is being paid for?
No, and no external tool can, this one included. A priced crawl is served differently to a client that holds a billing relationship, and a scanner is not such a client, so it always takes the unpaid branch. What an external scan does establish is whether the page can be fetched and read by a named crawler in the first place, which is the precondition for any of the commercial layer applying to it at all.
Do Google AI summaries actually reduce clicks to publishers?
The Pew Research Center analysis published on 22 July 2025 found that among 900 US adults sharing browsing data in March 2025, a traditional search result was clicked on 8 percent of visits to Google pages carrying an AI summary and on 15 percent of visits to pages without one. A link inside the summary was clicked on 1 percent of visits. That study covers Google only, by its own statement, and it measures one month of one panel rather than the whole web.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
Answer Engine Optimization (AEO)
Answer Engine Optimization (AEO) is the practice of making a page readable, allowed, and structured enough that AI answer engines can use it as a source when they compose an answer.
Pay per crawl prices AI crawlers with HTTP 402, a code the spec leaves undefined
Cloudflare's pay per crawl answers an unpaid AI crawler with HTTP 402 Payment Required and a crawler-price header naming the amount. RFC 9110 defines that status code in one sentence: it is reserved for future use.
Block AI crawlers: Cloudflare's new default depends on whether the page carries ads
Cloudflare's bots changelog entry of 1 July 2026 states that from 15 September 2026 new domains onboarding to Cloudflare receive defaults under which bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed. That is a rule about a page, and robots.txt has no way to write one.
