BlogFindings
GPTBot detection: 6 of 15 crawler tokens publish a User-Agent to match
A robots.txt product token and an HTTP User-Agent header are two different strings, and only the second one is what your access log holds. Read at nine vendors' own documentation on 27 August 2026, 6 of the 15 tokens in Lantad's crawler registry carry a complete User-Agent string that can be matched literally.
Most of the time the gap does not matter, because the token appears inside the header and a substring search finds it. This post is about the cases where it does not, and it counts them rather than describing them. Lantad's crawler registry in core/src/bots.ts, the same data behind the published crawler directory, records 15 robots.txt product tokens across nine vendors. On 27 August 2026 every one of those vendors' documentation pages was opened and asked one question: does this page publish the complete User-Agent string this crawler sends? Six answered yes without qualification. Two answered with a placeholder or a hedge. Three describe a token that never sends a request of its own. Three belong to crawlers that do fetch pages and publish no header string anywhere. One page could not be read at all, and it is counted as unread rather than as anything else.
In short
- GPTBot detection needs a string the vendor actually publishes, and read at nine vendors' own pages on 27 August 2026, 6 of the 15 robots.txt tokens in Lantad's crawler registry carry a complete User-Agent string that can be matched literally: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User and CCBot.
- RFC 9309 section 2.2.1 requires a crawler to have a product token and only states that the token SHOULD be a substring of the User-Agent header it sends, so a vendor publishing a token and no header is following the standard rather than departing from it.
- Anthropic's support article, carrying a publication date of 7 April 2026, names ClaudeBot, Claude-User and Claude-SearchBot and publishes no full User-Agent string for any of the three, which is why the three Anthropic strings in Lantad's own registry are recorded as taken on trust rather than sourced.
- Three of the 15 tokens describe no request at all: Google's crawler documentation, Last updated 2026-07-14 UTC, states that Google-Extended doesn't have a separate HTTP request user agent string, and Apple states that Applebot-Extended does not crawl webpages.
- Amazon publishes Amazonbot's User-Agent with the literal placeholder Chrome/W.X.Y.Z inside it and Meta says the string you will see in your log files will be similar to one of the following, so two further tokens of the 15 have no exact string to match.
Flow: 15 tokens, 9 vendors (always published) to robots.txt product token; 15 tokens, 9 vendors (sometimes published) to User-Agent header sent; User-Agent header sent (6) to 6 published in full; User-Agent header sent (2) to 2 hedged or placeholder; User-Agent header sent (3) to 3 crawl, no header published; robots.txt product token (3) to 3 send no request; User-Agent header sent (1) to 1 page unreadable.
What does GPTBot detection actually need?
There are three questions people mean by this and only one of them is about strings. The first is whether a crawler is allowed to fetch a page, which is a robots.txt question and needs a token. The second is whether a request that already arrived came from the crawler it claims to be, which is a verification question and needs an IP range or a signature. The third is the one this post is about: given a log full of requests, which lines are that crawler? That question is answered by matching text against the User-Agent header, and it cannot be answered at all if nobody has told you what text to expect.
The three questions get conflated because the token usually sits inside the header, so one word appears to serve all three. It does not. This site has written the verification half of the problem up before, in the post arguing that a user agent is a claim rather than an identity: any program can send the exact header GPTBot sends, so a header match tells you what a client said about itself and nothing more. That post is about whether to believe the string. This one is about whether the string exists.
The distinction has a practical edge for anyone writing a log rule or a firewall rule. A rule keyed on the token alone is broad, because plenty of headers contain a short word by accident, and it is also blind, because a crawler whose header omits the token entirely will never match it. A rule keyed on the vendor's published header is precise and needs the vendor to have published one. Lantad's own classifier faces the same choice and resolves it by ordering, which is the subject of the post on AI crawler detection as an ordering problem: registry tokens are tested before the classical search engine list and before a generic substring match, longest token first, so a short name cannot swallow a longer one that contains it.
What none of that ordering can supply is a string nobody wrote down. That is the gap this count measures, and the answer is not uniform across the field. It splits almost exactly along vendor lines, which is what makes it worth a table rather than a sentence.
robots.txt product token
- Letters, underscores and hyphens only, per RFC 9309
- Matched case insensitively against your rules
- Published by every vendor in the registry: 15 of 15
- Answers: may this crawler fetch the page
- Appears in your robots.txt, not in your access log
HTTP User-Agent header
- A free-form string the client sends on each request
- The token SHOULD be a substring of it, per RFC 9309
- Published in full by 6 of the 15 in the registry
- Answers: which log lines belong to this crawler
- Appears in your access log, not in your robots.txt
RFC 9309 requires a token and only recommends the header
The reason the field looks inconsistent is that it is not being inconsistent. The specification asks for one string and merely suggests the other, and the vendors publishing only a token are doing what the standard requires of them.
RFC 9309 is the Robots Exclusion Protocol, and its section 2.2.1 defines the user-agent line. It sets a hard constraint on the token itself: the product token MUST contain only uppercase and lowercase letters, underscores and hyphens. It sets a hard constraint on how the token is used: crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group. Both of those are requirements on the crawler's relationship with your rules file.
Then it addresses the header, and the verb changes. The product token SHOULD be a substring of the identification string that the crawler sends to the service, and, in the case of HTTP, the product token SHOULD be a substring in the User-Agent header. SHOULD, in the vocabulary these documents use, means a recommendation that can be departed from with reason. There is no requirement anywhere in the document that a vendor publish the header it sends, and no requirement that the header contain the token, only that it ought to.
That is a defensible design. The protocol governs a negotiation about access, and access is decided by the token. Everything downstream of the fetch, including whatever a site owner wants to do with a log file afterwards, is outside what the protocol set out to standardise. The consequence for anyone doing detection is that the missing header is a gap in vendor practice rather than a violation, which changes what you can reasonably ask for and rules out treating an absence as bad faith. It also means the gap will not close on its own. The work that could close it is the IETF effort on cryptographic crawler identity, which this site has read in detail in the post on what Web Bot Auth actually specifies, and which is not yet something an ordinary log rule can use.
-
Token character setMUST The product token MUST contain only uppercase and lowercase letters, underscores and hyphens. -
Case-insensitive group matchMUST Crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey its rules. -
Token inside the User-Agent headerSHOULD The product token SHOULD be a substring of the identification string the crawler sends, and in HTTP a substring in the User-Agent header. -
Vendor publishes the header stringNot addressed The document places no obligation on a vendor to publish the User-Agent header its crawler sends.
What the nine vendors publish, token by token
Every documentation page named in the registry was opened on 27 August 2026 and read for one thing: a complete User-Agent header, printed as a string, for the token in question. Six tokens have one. OpenAI's crawler documentation publishes a full header for every crawler it lists, and GPTBot's is given as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. OAI-SearchBot and ChatGPT-User are printed the same way on the same page. Perplexity's bots guide does the same for PerplexityBot and Perplexity-User, and adds a JSON endpoint of IP ranges for each, saying that the IP addresses are updated regularly and should be the source of truth. Common Crawl's CCBot page publishes CCBot/2.0 (https://commoncrawl.org/faq/) and documents both a JSON range file and reverse DNS.
One correction to our own record belongs here rather than in a footnote. OpenAI's page lists four crawlers and Lantad's registry carries three of them: OAI-AdsBot is documented by the vendor with a full header and is absent from the registry. That is a gap in the registry, not in OpenAI's documentation, and it is recorded here because this site has counted OpenAI's four crawler IP range files before and should have noticed. The 15 in this post is a count of what Lantad tracks, which is not the same as a count of what exists, and no figure here should be read as a census of AI crawlers on the web.
The table below is the whole result. Read it by column rather than by row: the interesting fact is not that any one vendor is better documented than another, it is that the answer to a single simple question splits the field into four groups of roughly equal size. Six with a literal string, two with something that looks like a string and is not, three that publish a token for a crawler that fetches pages without saying what it sends, and three that publish a token precisely because nothing fetches under it.
The count of 15 tokens across nine vendors is the same registry this site read when it found that six of the nine vendors publish exactly one crawler token rather than the separate training and search names the standard advice assumes. That earlier count was about how many names a vendor gives you. This one is about whether the names are usable in the one place a site owner can actually observe a crawler.
| Product token | Vendor | Full User-Agent published | What the vendor's page says |
|---|---|---|---|
| GPTBot | OpenAI | Yes | Complete header printed, GPTBot/1.4, plus a JSON IP range file |
| OAI-SearchBot | OpenAI | Yes | Complete header printed, OAI-SearchBot/1.4 |
| ChatGPT-User | OpenAI | Yes | Complete header printed, ChatGPT-User/1.0 |
| PerplexityBot | Perplexity | Yes | Complete header printed, plus a JSON IP range endpoint |
| Perplexity-User | Perplexity | Yes | Complete header printed, plus a JSON IP range endpoint |
| CCBot | Common Crawl | Yes | CCBot/2.0 printed, plus a JSON range file and reverse DNS |
| Amazonbot | Amazon | Placeholder | Header printed with the literal token Chrome/W.X.Y.Z inside it |
| Meta-ExternalAgent | Meta | Hedged | Two forms given, prefaced by will be similar to one of the following |
| ClaudeBot | Anthropic | No | Robots token named, no header string on the page |
| Claude-SearchBot | Anthropic | No | Robots token named, no header string on the page |
| Claude-User | Anthropic | No | Robots token named, no header string on the page |
| anthropic-ai | Anthropic | No request | Legacy robots token, no distinct crawler documented for it |
| Google-Extended | No request | Doesn't have a separate HTTP request user agent string | |
| Applebot-Extended | Apple | No request | Applebot-Extended does not crawl webpages |
| Bytespider | ByteDance | Unread | The documentation page returned only its title on 27 August 2026 |
Three Anthropic crawlers fetch pages and publish no User-Agent
The three rows that matter most are the Anthropic ones, because they are the only case in the set where a crawler that genuinely fetches your pages comes with no published string to find it by. Anthropic's support article on crawling, which carries a publication date of 7 April 2026, names ClaudeBot for training, Claude-User for user-initiated fetches inside Claude, and Claude-SearchBot for search indexing. It explains what each one does and how to block it. It prints no User-Agent header for any of the three, and instead points readers to a JSON file of IP ranges at claude.com/crawling/bots.json.
That is a coherent position, and arguably the better one. An IP range is verifiable and a header is not, so a vendor that gives you ranges instead of a string has given you the stronger primitive for deciding whether a request is real. It is the wrong primitive for the question in this post. A site owner reading last month's log has a text field and no live socket, and cannot check an address that was allocated at the time and reassigned since. This site has already found that even the range files are less stable than they look, counting thirteen crawler IP range files where six carried two dates that disagree.
The uncomfortable half of this belongs to us. Lantad's registry does carry a User-Agent string for each of the three Anthropic tokens, and those strings are not sourced. The file says so in its own header comment, recording them as widely reported strings we cannot source and flagging them as the one part of the registry still taken on trust after every other entry was confirmed against vendor documentation on 4 August 2026. They are not decoration: the scanner sends the value in that field as a request header when it runs a per-crawler probe, which is how the access sub-score described in the scoring methodology is produced. So a probe labelled ClaudeBot is a probe sent with a string Anthropic has not published, and any result it returns is a result about that string.
Naming that is cheaper than the alternative. The robots.txt probe run across the golden fixtures found all 60 user agent probes permitted and 11 of them answered differently anyway, which is exactly the kind of result whose interpretation depends on the header being the one the vendor actually sends. Where it is not, the honest report is that a string was sent and a server answered, not that Anthropic's crawler was tested.
- Robots.txt token for each crawler ClaudeBot, Claude-User and Claude-SearchBot are all named on the page, with the purpose of each.
- Instructions for blocking each one The page sets out how a site owner disallows each token in robots.txt.
- IP ranges for verification The page points to a JSON file of ranges at claude.com/crawling/bots.json rather than to a header string.
- Full User-Agent header string No complete header is printed for any of the three, so there is no vendor-published string to match a log line against.
- Lantad's stored string is sourced The three entries in core/src/bots.ts are marked as widely reported and unsourceable, and the scanner sends them as request headers regardless.
Two more strings carry a placeholder or a hedge
Between the six clean answers and the three absent ones sit two that look like answers and do not survive being used as one. Amazon's Amazonbot page prints Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36. The W.X.Y.Z is Amazon's own placeholder for whichever Chrome version is live, so the published string is a template rather than a string, and no request ever carries those four letters. A log rule can match the stable part of it. An exact comparison against the documented value matches nothing, ever.
Meta's crawler page carries the same problem in words instead of letters. It gives two forms for meta-externalagent, one bare and one with a parenthesised link, and prefaces them with the sentence that the specific UA string that you will see in your log files will be similar to one of the following. Similar is not a specification. One of the two forms also renders its contact link as a relative path, which is the documentation page printing its own link rather than anything a client could transmit. This site counted the tokens on that page when it noted that Meta documents five crawler tokens, and the header question is the one the page answers least firmly. Neither developers.facebook.com nor support.apple.com is on this site's registered outbound host list, so both are named here and left as plain text: developers.facebook.com/documentation/sharing/webmasters/web-crawlers and support.apple.com/en-us/119829.
The three no-request rows are a different thing entirely and should not be read as a failure of documentation, because in each case the vendor has answered the question clearly and the answer is that there is nothing to match. Google's common crawlers documentation, Last updated 2026-07-14 UTC, states that Google-Extended doesn't have a separate HTTP request user agent string, that crawling is done with existing Google user agent strings, and that the robots.txt user-agent token is used in a control capacity. Apple's page states that Applebot-Extended does not crawl webpages and is only used to determine how to use the data crawled by the Applebot user agent. Searching a log for either name will find nothing, correctly, and a tool that reports zero Google-Extended requests as a finding is reporting the design of the token.
Bytespider is the one row this run could not settle. The documentation page recorded in the registry, zhanzhang.toutiao.com/docs/intro/26899, returned only its page title when requested on 27 August 2026, so no header could be read from it and the row is marked unread rather than counted either way. The registry holds a string for it that was confirmed on 4 August 2026, and a token whose vendor page cannot be reached is exactly the situation that produces the stale rules described in the post on renamed crawler tokens.
What this does not show, and what to match on instead
This is a count of documentation on one day, and the limits are worth more than the number. Fifteen tokens is what this scanner tracks and not what exists on the web: OAI-AdsBot is documented by its vendor and missing from the registry, which is one demonstrated omission and therefore evidence that there are others. No crawler was observed for this post and no log was analysed. Nothing here shows that an undocumented header is rare in real traffic, that a documented one is sent reliably, or that any vendor's crawler behaves as its page describes. A vendor page is a statement of intent, and the only thing measured here is whether the statement contains a string.
The strings themselves also rot. Five of the twelve headers in this registry were found to be wrong when they were last checked against vendor documentation on 4 August 2026, including a GPTBot version that had moved from 1.2 to 1.4 and an Amazonbot entry that was a Safari-on-Macintosh string appearing nowhere in Amazon's documentation. Three weeks later the OpenAI values still match. That is a small reassurance about a file that has already been wrong once, and it is the reason a date is attached to every figure in this post rather than to the post as a whole.
For anyone writing an actual rule, the useful shape follows from the table rather than from any single row. Match the token as a case-insensitive substring, because that is the one thing RFC 9309 obliges the crawler to make findable and it works for every row in the set that sends a request at all. Do not anchor on a full published header, since two of the eight that have one cannot be matched exactly and version numbers move inside the rest. Do not expect the three control tokens in a log, since their vendors have said plainly that nothing fetches under them. Then treat every match as a claim and verify the ones that matter against the range files, which is the same order of operations the crawler directory and the GPTBot view of a page are built on.
The larger point is that the two halves of this problem are drifting apart. Access control is well specified and universally documented, because the standard requires it. Observation is neither, because the standard is silent, and the vendors filling that silence are choosing different things: OpenAI and Perplexity publish both the header and the ranges, Anthropic publishes the ranges and directs you to them, Google and Apple explain that the question does not apply. A site owner comparing what an AI crawler is permitted to do against what one demonstrably did is working from two unequal records, and that inequality is the honest starting point for any figure about AI visibility built on log data. The post on how a GPTBot ban still served a 200 is the same lesson from the other end, and if you are specifically trying to see Claude's crawlers, the practical guidance is on the page about getting cited by Claude rather than in any log rule.
Sample Illustrative, not a measurement of any real site.
Matching a log line, token first
- grep -i 'gptbot' access.log token substring, works for every crawler that sends one
- match on 'GPTBot/1.4' exactly breaks on the next version bump
- match on 'Chrome/W.X.Y.Z' exactly never matches, the vendor string is a template
- grep -i 'google-extended' access.log zero by design, the token sends no request
- reverse lookup or range check the match the only step that answers whether it was real
Lantad
Published .
Somebody who wants to know whether ChatGPT has been reading their site goes looking in an access log, and the thing they search that log for is a name. GPTBot is the name they have. It is printed in every guide, it is the word they put in their robots.txt, and it feels like an identifier. It is an identifier, but not of the thing they are searching. GPTBot is a robots.txt product token, and a product token is a label a crawler answers to when it reads your rules. The line in your log is an HTTP User-Agent header, and those are two different strings published in two different places, when they are published at all.
Common questions
How do I detect GPTBot in my server logs?
Match the string GPTBot case insensitively against the User-Agent header. OpenAI's crawler documentation, read on 27 August 2026, publishes the complete header as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, so an exact match works today and breaks at the next version bump. A header match tells you only what the client claimed. OpenAI also publishes JSON files of the IP ranges each of its crawlers fetches from, and checking the request address against those is the step that decides whether the claim was true.
Is a robots.txt product token the same as a User-Agent string?
No. RFC 9309 section 2.2.1 defines a product token as a name made only of letters, underscores and hyphens, matched case insensitively against the groups in your robots.txt. The User-Agent is a free-form header sent on each request. The specification says the token SHOULD be a substring of that header, which is a recommendation rather than a requirement, and it places no obligation on a vendor to publish the header at all.
Why can I not find a User-Agent string for ClaudeBot?
Anthropic does not publish one. Its support article on web crawling, carrying a publication date of 7 April 2026, names ClaudeBot, Claude-User and Claude-SearchBot, explains what each does and how to block it, and prints no complete User-Agent header for any of them. It points instead to a JSON file of IP ranges at claude.com/crawling/bots.json. Any full ClaudeBot string you find elsewhere, including the one in Lantad's own crawler registry, is reported rather than sourced from Anthropic.
Why do Google-Extended and Applebot-Extended never appear in my logs?
Because neither one fetches anything. Google's common crawlers documentation, Last updated 2026-07-14 UTC, states that Google-Extended doesn't have a separate HTTP request user agent string, that crawling is done with existing Google user agent strings, and that the token is used in a control capacity. Apple states that Applebot-Extended does not crawl webpages and is only used to determine how the data crawled by Applebot may be used. Both are robots.txt controls over how already-crawled content is used, so zero requests is the expected result rather than a finding.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
A user agent is a claim, not an identity
Every per-crawler number you have ever seen, including ours, counts requests that said they were a crawler. Verifying the claim needs an IP check, and most tools do not do one.
AI crawler detection is an ordering problem
Turning a user agent string into an AI crawler label is a matching order with named failure modes, and every one of them produces a confident wrong answer rather than an error.
Six of the nine AI vendors we track publish one crawler token
The standard advice is to disallow training crawlers and allow search crawlers. That instruction needs two published names to write, and Lantad's registry records six of nine vendors publishing exactly one, while Cloudflare reported on 1 July 2026 that mixed-use crawlers are over 36 percent of crawler activity on its network.
Crawler IP range files carry two dates, and six of thirteen disagree
Thirteen published crawler IP range files from Google, OpenAI, Apple, Perplexity and Microsoft, read on 19 August 2026, each answer the question of when they last changed twice, and on six the two answers differ by between 13 and 942 days.
