Blog / A user agent is a claim, not an identity
A user agent is a claim, not an identity
Every per-crawler number you have ever seen, including ours, counts requests that said they were a crawler. Verifying the claim needs an IP check, and most tools do not do one.
In short
- A user agent is a request header the client chooses, so any program can send the exact string GPTBot uses and the receiving server cannot tell from the header alone.
- Lantad labels every per-crawler figure as requests CLAIMING to be that crawler, in the column heading rather than in a footnote, because a fabricated crawler count is indistinguishable from a real one.
- Verification is possible and costs a lookup: OpenAI, Anthropic and Perplexity each publish the IP ranges their crawlers fetch from, and a request from outside those ranges is not that crawler whatever its header says.
- Cloudflare exposes a verified-bot signal that does this work, but it is gated to paid plans, so a tool cannot assume its users have it.
- The practical consequence for a site owner is that an unverified spike in one crawler is a question rather than a finding.
The User-Agent header is a string the client picks. There is no signature on it, no registry binding it, and no step in HTTP where anyone checks. Sending one is exactly as hard as typing it.
That means every per-crawler chart in every AI visibility tool, this one included, counts requests that SAID they were a given crawler. Most of the time that is fine, because most traffic has no reason to lie. It stops being fine the moment a number is used for a decision, and the honest fix is cheap enough that there is no excuse for not knowing which kind of number you are looking at.
Requests arriving at a site claiming to be GPTBot
- GET /pricing User-Agent: compatible; GPTBot/1.2 from a published OpenAI range
- GET /pricing User-Agent: compatible; GPTBot/1.2 from an arbitrary datacentre IP
- Header comparison identical
- Distinguishable without an IP check no
When the difference actually costs you something
For a rough sense of whether AI crawlers reach your site at all, an unverified count is adequate. The failure mode is bounded: you might believe you are crawled slightly more than you are.
It matters when the number drives an action. Three cases recur. A sudden spike in one crawler, read as growing interest, when it is a scraper wearing a well-known name to avoid a block. A per-crawler bandwidth bill attributed to a vendor that did not incur it. And a decision to block a crawler based on volume that was never that crawler in the first place, which costs you the real crawler's access while leaving the impostor untouched.
The last is the worst, because the block is enforced by the same header the impostor is free to change.
- Is any AI crawler reaching this site Adequate. A claim is weak evidence but the question is coarse.
- Which pages do crawlers fetch most Adequate and useful, because the path is observed rather than claimed.
- Is GPTBot traffic up 40 percent this week Not supportable. A spike in a claimed name is as likely to be one impostor as a change at the vendor.
- Should I block this crawler for bandwidth Not supportable, and acting on it blocks the honest client while the impostor edits one header.
How verification actually works
The check is an IP one. A vendor that wants its crawler trusted publishes the address ranges the crawler fetches from, and a request claiming that crawler from outside those ranges is not that crawler. OpenAI, Anthropic and Perplexity each publish ranges for their fetchers, and the more careful method adds a reverse DNS lookup confirming the address resolves to the vendor domain and forward-resolves back to the same address.
Cloudflare does this work and exposes the result as a verified-bot signal, which is the least effort available if you are already behind it. The catch is that the signal is gated to paid plans, so a tool cannot assume its users have it, and a tool that quietly depends on it reports differently for different customers without saying why.
None of this is expensive. It is a list of CIDR ranges, refreshed on a schedule, and a containment test per request.
Header only
- GPTBot: 412 requests
- Confidence: none
- Supports: is anything crawling us
- Does not support: any decision
Header plus published IP ranges
- Verified and unverified counted apart
- An impostor is visible as its own number
- Supports blocking, billing and trend
- Cost: one CIDR lookup per request
What Lantad does about it today
Lantad does not verify crawler identity yet, and says so on every surface showing a per-crawler figure. The word claimed is in the column heading rather than in a note underneath, because a caveat a reader has to go looking for is a caveat that does not exist.
The reason it is not built is ordering rather than difficulty. Verification is a scheduled job pulling published vendor ranges into a lookup structure, plus a containment test at the edge, and it is on the list. Until it ships, the number on the page is a count of claims and the page says the word.
The general principle is worth more than the specific feature. When a tool shows a number about something it cannot directly observe, the useful question is not whether the number is large. It is what the tool would have had to do to know, and whether it says whether it did.
-
ClaimedHeader only What Lantad reports today, labelled as such. Supports coarse questions. -
VerifiedHeader plus IP range Supports blocking, billing and trend decisions. Requires a maintained range list. -
UnlabelledUnknown A per-crawler number with no statement of which of the two it is. The most common state in this market.
Common questions
Can anyone really pretend to be GPTBot?
Yes. The User-Agent header is chosen by the client and no part of HTTP verifies it. Sending the exact string OpenAI publishes takes one line of code, which is why the header alone cannot establish identity.
How do I verify an AI crawler myself?
Check the request IP against the ranges the vendor publishes for that crawler, and optionally confirm reverse DNS resolves to the vendor domain and forward-resolves back. If you are behind Cloudflare on a paid plan, the verified-bot signal does this for you.
Does Lantad verify crawler identity?
Not yet. Every per-crawler figure Lantad shows is labelled as requests claiming that crawler, in the heading rather than in a footnote. Verification is a scheduled range pull plus a containment test at the edge, and it is on the roadmap.
Should I block crawlers that fail verification?
That is a reasonable policy, but note what it does and does not achieve. It stops the impostor using that name, and the impostor's next move is to send a different one. Blocking by identity works best combined with rate limits that apply regardless of the claimed name.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.