Blog / Pay per crawl prices AI crawlers with HTTP 402, a code the spec leaves undefined
Pay per crawl prices AI crawlers with HTTP 402, a code the spec leaves undefined
Cloudflare's pay per crawl answers an unpaid AI crawler with HTTP 402 Payment Required and a crawler-price header naming the amount. RFC 9110 defines that status code in one sentence: it is reserved for future use.
In short
- Cloudflare's pay per crawl, announced on its blog on 1 July 2025, answers a crawler that sends no payment header with HTTP 402 Payment Required plus a crawler-price response header, and returns HTTP 200 with a crawler-charged header when the crawler agrees to the price.
- RFC 9110, HTTP Semantics, published in June 2022 as STD 97, defines 402 Payment Required in a single sentence stating that the code is reserved for future use, so the status carries no interoperable meaning that a crawler and a server can both rely on.
- RFC 9309 sorts robots.txt fetch results by status class rather than by code, and says a response in the 400 to 499 range means the crawler MAY access any resources on the server, which is the permissive branch that 402 falls into.
- The published crawler documentation of OpenAI, Anthropic and Perplexity, read on 2 August 2026, names user agents and blocking methods and says nothing about payment headers or about what any of those crawlers does with a 402.
- Lantad has measured no part of this and cannot measure it from outside: a scanner sends no payment header, so a priced page and a blocked page hand it the same refusal.
Every access control this site measures answers a question about permission. A robots.txt rule names which clients may fetch which paths. A meta robots tag says what may be done with a page after it is fetched. A bot rule at the edge decides whether a request is served at all. Pay per crawl asks a different question, and it is one that HTTP has never had a defined answer for: not whether a crawler may read the page, but what reading it costs.
Lantad has measured none of what follows. The mechanism below is read from Cloudflare's own announcement and product documentation, and from the two RFCs the mechanism sits on top of, all fetched on 2 August 2026. What this post adds is the part that changes what any outside measurement can tell you. A price and a block are the same response to a client that carries no wallet, and the client carrying no wallet includes every scanner, every archive fetch, and any AI crawler whose operator has not signed an agreement in advance.
What the HTTP specification defines
- 402 Payment Required
- "The 402 (Payment Required) status code is
- reserved for future use."
- That is the whole of section 15.5.3.
What a pay per crawl response carries
- HTTP 402 Payment Required
- crawler-price: USD XX.XX
- Retry carries crawler-exact-price to accept
- Success returns crawler-charged on a 200
What RFC 9110 says about 402, in full
RFC 9110, HTTP Semantics, was published in June 2022 and carries the designation STD 97, which makes it the current definition of what HTTP status codes mean rather than one proposal among several. Section 15.5.3 is the entry for 402, and it is one sentence long. The 402 (Payment Required) status code is reserved for future use. There is no second paragraph, no description of when a server should send it, and no guidance on what a client should do next.
The contrast with its neighbours is the point. Section 15.5.1 defines 400 Bad Request as a response for a request the server cannot or will not process due to something that is perceived to be a client error, and gives malformed request syntax, invalid request message framing and deceptive request routing as its examples. That is a definition a client can act on: retrying the identical request is pointless, and the fault is at the client's end. 402 gives a client nothing comparable to work from.
Mozilla's reference documentation is blunter about the consequence. The MDN entry for 402 describes it as a nonstandard client error response status code reserved for future use, says it was created to enable digital cash or micro payment systems, and states that no standard use convention exists and different systems use it in different contexts. It adds that no browser supports a 402 and that an error will be displayed as a generic 4xx status code.
That is what makes this worth writing about rather than filing as trivia. A status code is a contract between two programs whose authors have never met. A 404 works because both ends already agree what it means, so a crawler that receives one can drop the URL without asking anybody. A 402 has no such agreement behind it. Whatever a given host means by a 402, the client on the other end learns it from documentation that lives outside HTTP, or it does not learn it at all. Every question in this post follows from that one gap, and it is a gap in the specification rather than in anybody's implementation.
| Reference | Date it carries | What it says about 402 |
|---|---|---|
| RFC 9110 section 15.5.3, STD 97 | June 2022 | "The 402 (Payment Required) status code is reserved for future use." One sentence, no further guidance. |
| MDN HTTP status reference | Read 2 August 2026 | Nonstandard, reserved for future use, created for digital cash or micro payment systems; no standard use convention exists. |
| Cloudflare, Introducing pay per crawl | 1 July 2025 | Returned to a crawler that sends no payment header, carrying a crawler-price header naming the amount. |
What a crawler actually receives from a priced page
Cloudflare published Introducing pay per crawl on 1 July 2025. Its product documentation, at the page headed what is pay per crawl, carries a last updated date of 28 July 2026 and states that the feature is currently in closed beta, with the price set per zone. The pay per crawl FAQ adds that a site owner can only set a single price that applies to all crawlers configured with the Charge option, so this is one number for a whole property rather than a rate card per path.
Identity comes before money. The example request in the announcement carries Signature-Agent, Signature-Input and Signature header fields, with a keyid, an alg of ed25519 and a tag of web-bot-auth, which is the Web Bot Auth shape built on RFC 9421, HTTP Message Signatures, published in February 2024. This site has already covered what Web Bot Auth specifies and what it does not, and the relevant part here is that the signature answers who is asking, not whether they will pay. Both questions have to be answered before a charge can happen, and they are answered by different headers.
The announcement describes two paths through the exchange. In the reactive one, a crawler requests a page without declaring anything, receives HTTP 402 Payment Required with a crawler-price header naming an amount in a currency, and can then repeat the request with a crawler-exact-price header agreeing to that price. In the proactive one, the crawler sends crawler-max-price on the first request, declaring the most it is willing to pay, and the server decides whether the page is available at or below that ceiling. A successful charged fetch returns HTTP 200 OK with a crawler-charged header stating what was billed.
The commercial layer behind those headers is a normal payment relationship rather than a protocol. Cloudflare's announcement states that it acts as the Merchant of Record for pay per crawl and provides the underlying technical infrastructure, and that it aggregates all the events, charges the crawler, and distributes the earnings to the publisher. Its setup documentation for an AI crawler owner lists verifying the crawler and connecting Stripe among the steps before any content can be paid for. A crawler that has done none of that cannot be charged, which means it is never priced. It is simply refused by whatever rule would have applied anyway.
Sample Illustrative, not a measurement of any real site.
GET /article, signed crawler, no payment declared
- GET /article HTTP/1.1 Signature-Agent, Signature-Input, Signature present
- HTTP 402 Payment Required crawler-price: USD XX.XX
- GET /article HTTP/1.1 (retry) crawler-exact-price: USD XX.XX
- HTTP 200 OK crawler-charged: USD XX.XX, server: cloudflare
- GET /article HTTP/1.1 (unsigned client) no identity, so no charge is possible
A price in front of robots.txt would read as permission
RFC 9309, the Robots Exclusion Protocol, was published in September 2022, and its handling of failed fetches is written in terms of status classes rather than individual codes. Under the heading Unavailable Status it describes a server responding with status codes indicating that the resource in question is unavailable, gives the 400 to 499 range as the HTTP example, and states that the crawler MAY access any resources on the server. Under Unreachable Status it treats the 500 to 599 range as meaning the file is undefined, and there the crawler MUST assume complete disallow.
402 sits in the first range. By the letter of that rule, a conforming crawler that asks for robots.txt and receives a 402 concludes that it may crawl everything. A price attached to the rules file reads as the absence of rules. This site has argued the same structural point before, that a 404 and a 503 on robots.txt are opposites despite both being failures, and 402 lands on the permissive side of that same split without anybody having decided that it should.
Two honest qualifications belong here, and they matter more than the observation. The first is that this is a reading of the specification, not an observation of a deployment. Nothing in Cloudflare's published material says robots.txt is served under a price, and Lantad has not fetched a priced robots.txt anywhere. What is being described is a consequence of defining crawler behaviour by status class at a time when a new meaning is being attached to a code inside one of those classes. The second is that RFC 9309 says in its own opening section that its rules are not a form of access authorization, which is exactly what a payment gate is. The two mechanisms are answering different questions, and a crawler reading only the first will draw a conclusion the second never intended.
For a site owner the practical version is narrower and easier to check. Whatever your infrastructure does with page requests, the rules file needs to return a 200 with a parseable body to the clients you want following your rules, and you can confirm that for your own domain with a robots.txt tester without waiting for anybody's beta. The failure mode worth guarding against is not a crawler that pays and a crawler that does not. It is a crawler that never receives the file that would have told it what to do.
Flow: Crawler requests /robots.txt (200) to 2xx: parse and follow the rules; Crawler requests /robots.txt (402, 403, 404) to 4xx, called Unavailable; Crawler requests /robots.txt (500 to 599) to 5xx, called Unreachable; 4xx, called Unavailable to MAY access any resource on the server; 5xx, called Unreachable to MUST assume complete disallow.
No crawler vendor documents what it does with a 402
The three vendors whose crawlers drive most AI answer traffic all publish a page telling site owners how to control them, and all three were read on 2 August 2026. OpenAI's crawler documentation names OAI-SearchBot for surfacing sites in ChatGPT search, OAI-AdsBot for checking pages submitted as advertisements, GPTBot for training data, and ChatGPT-User for user initiated fetches. Anthropic's crawler guidance, which carries a last updated date of 7 April 2026, names ClaudeBot for training, Claude-User for fetches made on behalf of somebody asking a question, and Claude-SearchBot for search quality. Perplexity's bot documentation names PerplexityBot for surfacing and linking sites in results and Perplexity-User for user actions.
None of those three pages mentions payment, a 402, or any header resembling the ones in the previous section. Every one of them is written entirely in the vocabulary of permission: here are the names, here is how to disallow them.
That absence is not evidence about behaviour, and it should not be read as any. A crawler could handle a 402 perfectly well without its public page saying so. What the absence does mean is narrower and still consequential: a site owner who prices a page cannot look up what any of these crawlers will do about it. Whether GPTBot retries with a payment header, treats the 402 as a refusal and drops the URL, or backs off and returns later is not answerable from published documentation, and a scanner cannot answer it either.
The granularity problem underneath this is one the blog has measured before. Six of the nine vendors in Lantad's registry publish exactly one crawler token, which already makes allow search and block training inexpressible for most of them. Pricing sits on top of that and asks for a finer distinction still, between a fetch worth charging for and a fetch worth encouraging, using a signal that no vendor documentation currently acknowledges. The crawler reference on this site lists the tokens that a rule can actually be written against, and that list is the honest boundary of what a name based control reaches.
| Vendor | Page date | User agents documented | Mentions payment or 402 |
|---|---|---|---|
| OpenAI | No date shown | OAI-SearchBot, OAI-AdsBot, GPTBot, ChatGPT-User | No |
| Anthropic | Updated 7 April 2026 | ClaudeBot, Claude-User, Claude-SearchBot | No |
| Perplexity | No date shown | PerplexityBot, Perplexity-User | No |
| Cloudflare pay per crawl docs | Updated 28 July 2026 | Not a crawler vendor; sets the price per zone | Yes |
What an outside scan can tell you about a priced page, and what it cannot
A scanner sends no payment header. It holds no account with a payment processor, it has signed no agreement with any content delivery network, and in Lantad's case it announces itself honestly and asks for the page. That puts it permanently on the refusal branch of the exchange described above. If a page is priced, the scanner receives a 402. If the same page were blocked outright, it would receive a 403. Neither response contains the page, and nothing in either one tells the scanner whether a paying client would have been served.
That is a genuine limit on what this product can report, and it is worth stating rather than papering over. Lantad has not measured pay per crawl adoption and could not measure it from outside if it tried, because the quantity to be counted is the set of hosts that would have served the page to a client with a billing relationship, and a scanner is definitionally not one of those clients. Any number anybody publishes about how much of the web is priced rather than blocked has to come from inside a network that sees both branches.
What an outside fetch does still establish is the thing that decides whether a model can quote you, which is what came back. A response that does not carry the page did not carry the page, whatever the commercial reason, and an engine assembling an answer works from bytes it received rather than from bytes it could have bought. That is why the way this scanner grades a page keys on the response rather than on the intent behind it, and it is the same reasoning that makes two layers decide whether AI can read your site: the file states a policy and the edge returns an answer, and only the second one is what a crawler experienced.
The check available to any site owner today needs none of this apparatus. Request your own important pages with a crawler user agent, record the status and the body you get back, and compare that against what a browser receives from the same URL. Where the two differ you have found something worth explaining, whether the cause is a price, a bot rule, a challenge page or a rendering dependency. That comparison is the whole basis of measurable AI visibility, and it is available before any standards body decides what 402 finally means.
What the response establishes
- The exact status the origin returned
- Whether a body was served at all
- Whether the crawler UA got a different answer
- Whether robots.txt was reachable and parseable
What it cannot establish
- Whether a paying client would have been served
- What price, if any, was attached
- Whether a named crawler has an agreement in place
- How any vendor's crawler responds to a 402
Common questions
What status code does pay per crawl return to a crawler that has not paid?
HTTP 402 Payment Required, carrying a crawler-price response header that names the amount, according to Cloudflare's announcement published on 1 July 2025. A crawler that then repeats the request with a crawler-exact-price header receives HTTP 200 with a crawler-charged header.
Is HTTP 402 a standard status code?
It is registered but undefined. RFC 9110, published in June 2022 as STD 97, gives it one sentence: the 402 status code is reserved for future use. MDN describes it as nonstandard and states that no standard use convention exists, so any meaning it carries comes from the host that sent it.
What happens if robots.txt itself returns a 402?
RFC 9309 resolves robots.txt fetches by status class rather than by code. A response in the 400 to 499 range is what the RFC calls Unavailable, and for that case it says the crawler MAY access any resources on the server. 402 falls in that range, so a conforming crawler would treat it as permission rather than as a price.
Can a scan tell me whether a page is priced or simply blocked?
No. A scanner sends no payment header and holds no billing relationship, so it lands on the refusal branch either way and receives a response with no page in it. The status code differs, 402 against 403, but neither tells you what a paying client would have been served.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.