# LLM crawler detection: a bot declares three things and proves one > Cloudflare's rebuilt bot submission form, launched on 28 August 2026, asks an operator to declare what its bot does, how it uses what it reads, and who runs it. The automated review that follows fetches an IP list, confirms reverse DNS or validates a Web Bot Auth signature, so the one claim a machine settles is identity, and the three that describe intent are taken on the operator's word. - Canonical page: https://lantad.co/blog/llm-crawler-detection-declared-three-verified-one - This file: https://lantad.co/blog/llm-crawler-detection-declared-three-verified-one.md - Last substantive update: 2026-08-29 ## Key facts - **Published:** 2026-08-29 - **Category:** Findings - **Author:** Lantad - **Length:** 3335 words - **Takeaway 1:** LLM crawler detection has four signals to work with and only three of them can be checked: a User-Agent header is a string any client can send, while a published IP list, a forward confirmed reverse DNS lookup and an HTTP Message Signature each tie a request to something an operator controls. - **Takeaway 2:** Cloudflare's BotBase for Operators post of 28 August 2026 states that the new intake form asks an operator for three things, the behaviors its bot performs, the level of content use it needs expressed in the Content Signals model, and whether the operator is Direct or Intermediary. - **Takeaway 3:** The same post states that submissions are now reviewed automatically against a duplicate check, a User-Agent pattern specificity check and a verification check that fetches the operator's IP list, confirms reverse DNS or validates a Web Bot Auth signature, and that the number of new bots submitted each year has grown about 7 times in volume since 2023. - **Takeaway 4:** Cloudflare's verified bots documentation, read on 29 August 2026, names 11 behavior classes a tracked bot can carry and two operator labels, Direct and Intermediary, against the three purpose classes in Lantad's own crawler registry, which are training, search and user_agent. - **Takeaway 5:** Lantad has measured none of this and cannot: a submission status, a signature header and a verification verdict all live at a network edge or in a site's own logs, and none of them appears in an external scan of a hostname. ## Summary An access log is a list of claims. Each line carries a User-Agent string the client chose to send, an IP address that is harder to invent but says nothing about purpose, and a request for a page. Deciding which of those lines belongs to a model training crawler, which belongs to an assistant fetching a page for somebody who just asked a question, and which belongs to neither is the work people mean when they search for LLM crawler detection. It is harder than the guides suggest, and the reason is not technical difficulty. It is that almost everything a bot says about itself is unverifiable in principle. On 28 August 2026 Cloudflare published [BotBase for Operators](https://blog.cloudflare.com/botbase-for-operators/), which rebuilds the other end of that problem: the form a bot operator fills in to get its crawler listed in Cloudflare's directory. The post is worth reading closely for a reason its own framing does not emphasise. It sets out, in one place, exactly which claims about an [AI crawler](https://lantad.co/glossary/ai-crawler) a large network now asks for, and which of those claims it validates by machine. The two lists are different sizes, and the gap between them is the honest boundary of what detection can deliver. Lantad has run nothing against BotBase, holds no directory entry and has seen no submission. What follows reports the announcement and the documentation behind it, then says what a site owner can and cannot conclude from either. ## What does LLM crawler detection actually verify? Four signals are available to anyone trying to name the client behind a request, and they are not equal. Three of them can be checked. One cannot be checked at all, and it is the one almost every guide leads with. The User-Agent header is a string the client composes. Nothing about sending it proves anything, which is why a token appearing in your log is evidence of a claim rather than of a crawler. The practical position is worse than that, because vendors often do not publish a string to compare against in the first place: read at nine vendors' own documentation on 27 August 2026, [6 of the 15 robots.txt tokens in Lantad's crawler registry carry a complete User-Agent string that can be matched literally](https://lantad.co/blog/gptbot-detection-six-of-fifteen-tokens-publish-a-ua). For the rest there is no exact text to grep for, so even the unverifiable signal is incomplete. The second signal is a published list of IP ranges. It is checkable in the sense that a request either came from a listed address or it did not, and it is the mechanism most operators offer. Its weaknesses are administrative rather than cryptographic. Lantad read 13 published crawler IP range files from Google, OpenAI, Apple, Perplexity and Microsoft on 19 August 2026 and found that [six of the 13 returned a creationTime and an HTTP Last-Modified header that disagreed](https://lantad.co/blog/crawler-ip-files-carry-two-dates-that-disagree), with none of the files carrying a version number, an expiry or a signature. A file you cannot date is a file you cannot tell has gone stale. The third is reverse DNS, forward confirmed. [Google's page on verifying requests from its crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests), carrying Last updated 2026-03-20 UTC and read on 29 August 2026, names exactly two methods: command line tools for one-off lookups, where you run a reverse DNS lookup on the accessing address, verify the domain name is googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that name and check it returns the original address; and matching against the published IP lists for large scale work. The same page splits Google's own clients into three categories and says plainly that the special-case crawlers and fetchers may or may not respect robots.txt rules, which is a useful reminder that identity and obedience are separate questions. The fourth is a cryptographic signature over the request itself, which is what [Web Bot Auth specifies](https://lantad.co/blog/what-web-bot-auth-actually-specifies) on top of [RFC 9421 HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421.html), a Standards Track document published in February 2024. This is the only one of the four where the proof travels with the request rather than being reconstructed afterwards from a list somebody else maintains. ## What a bot operator now declares about itself The BotBase announcement describes an intake form rebuilt around a taxonomy rather than a single label, and it asks for three separate things. The first is behavior, and the operator may select every one that applies rather than the closest match. The announcement gives indexing pages for search, acting as an agent on a user's behalf, collecting data, training models and supporting SEO tools as examples. [Cloudflare's verified bots documentation](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), read on 29 August 2026, sets out the full set: 11 behavior classes named Search, Agent, Training, Transact, Data Collection, Security Testing, SEO, Ads Verification, Social / Link Preview, Feed Fetching, and Monitoring and Operations, of which Search, Agent and Training are also available to site owners as managed presets across all plans. The second is content use, declared in the Content Signals model that site owners already use to write their own rules. The announcement's worked example is a robots.txt line reading Content-Signal: search=yes, ai-train=no, use=reference, telling every crawler that indexing the page for search and keeping a reference is acceptable while training on it is not. The operator's content use declaration is what gets compared against exactly that kind of preference, which is a genuinely useful piece of plumbing: it turns two unrelated free text conventions into two halves of one comparison. It is also the point at which the same file is being written from both ends, since Cloudflare separately [generates and periodically refreshes robots.txt from the same bot database](https://lantad.co/blog/cloudflare-generates-robots-txt-and-the-token-list-moves) for customers who turn that on. The third is who is running the bot, and the documentation gives two labels. Direct means the bot is operated by a single, narrow operator, usually on its own infrastructure, and only that operator can send requests presenting as it. Intermediary means an agentic service a wide range of end users can operate, where the operator runs the software but each action is initiated by a different end user. That distinction matters more than it looks. A general purpose assistant fetching a page because somebody typed a question into a third party app built on its API is running on the assistant operator's infrastructure while carrying a request nobody at that operator decided to send, and the two parties are not the same when it comes to conduct. Cloudflare's own policy already treats conduct as revocable, and one of its listed breaches, [an AI crawler that does not respect the crawl-delay directive in robots.txt](https://lantad.co/blog/crawl-delay-can-unverify-a-crawler), turns on a directive the robots exclusion standard never defined. ## Which of those claims does the automated review check? The reason for automating the review is given plainly, and it is a volume problem. The announcement states that the number of new bots submitted each year has grown sharply, increasing about 7 times in volume since 2023, and that every submission previously followed a fully manual path where somebody checked it against an internal rubric and made a judgement call. That does not scale, so the checks were rebuilt to run on their own. Three checks are named. The first asks whether the bot is a duplicate of one already tracked. The second asks whether the User-Agent pattern is specific enough to identify this bot without overlapping one already registered, which is a collision check on the namespace rather than a test of whether the string is true. The third is the one the announcement calls most important: does the claimed verification method actually hold up. There the system fetches the operator's IP list, confirms its reverse DNS, or validates its Web Bot Auth signature, and anything that needs a closer look is routed to a person with the reason already flagged rather than landing in a queue as a blank entry. Read against the previous section, the shape is clear. Three of the four claims a submission carries describe what the bot intends to do with what it reads, and none of them has a stated automated test, because none of them is testable from a form. The fourth claim is about identity, and it is the one that gets settled by machine. That is not a criticism of the design. It is the same boundary that every identity system runs into, and Cloudflare's own definition of a verified bot puts both halves on the record: the documentation sets two bars, honest self-identification, which it says means declaring who you are deterministically through a cryptographic signature, a published IP list with a stable user-agent, or reverse DNS, and non-abusive behaviour, which is assessed over time rather than at submission. The identity half is also the half with a live standards effort behind it. When Lantad read the [Web Bot Auth working group document page](https://datatracker.ietf.org/wg/webbotauth/documents/) on 29 July 2026 it listed ten active Internet-Drafts, two of them filed on 19 July 2026. The behaviour half has no equivalent, and the research is not encouraging about shortcuts: a paper read here on 6 August 2026 found that [the honeysite defended only by robots.txt blocked one of the twelve automation tools it tested](https://lantad.co/blog/web-agents-identifiable-robots-txt-stopped-one), the one whose framework ships with robots.txt obedience switched on by default. ## What none of this puts in your own access log A directory entry is held by the network that maintains it. Nothing about a bot being accepted into BotBase appears on your origin unless your traffic passes through that network, and even then what you see is a classification rather than the declarations behind it. So it is worth being exact about what a site owner can check without any of this. You can send a name and see what comes back. That is what Lantad does, and the result is the reason this post is careful about the word detection. Probing 12 AI crawler user agent strings against five real pages captured on 15 July 2026, [robots.txt permitted the crawler in all 60 of those probes while 11 came back unlike the baseline fetch](https://lantad.co/blog/robots-txt-allowed-sixty-probes-eleven-differed): four received HTTP 403 and seven received HTTP 200 carrying visible text that matched the baseline only 87 percent. A probe sends a user agent string and nothing else, so a 403 answered to a claimed GPTBot is evidence that a server treats that string differently, not evidence that OpenAI's crawler is blocked. The distinction is the whole reason that observation is recorded at low confidence. [The crawler view of a page](https://lantad.co/tools/what-gptbot-sees) is a fetch made under a name, and a name is all it is. You can also read the rules you are publishing, which is a separate exercise from reading the file. Resolving a real path against a whole robots.txt is not something anyone does reliably by eye once more than one group is in play, which is what [the robots.txt tester here](https://lantad.co/tools/robots-txt-tester) exists for. And you can read the vendor documentation for the tokens you care about: [OpenAI's crawler page](https://developers.openai.com/api/docs/bots) is the primary source for its own three tokens, and a primary source is the only kind worth quoting, because a token list copied from a roundup is a token list as it stood on an unknown date. What you cannot do from outside is verify a signature you never received, and this is the limit that applies to Lantad as much as to any reader. A Web Bot Auth signature is a header on an inbound request. It arrives at your edge or in your log, and it is invisible to any external scan of a hostname, which is why this site publishes [its own crawler's conduct policy and opt-out](https://lantad.co/bot) rather than claiming to audit anyone else's compliance. ## Three purpose classes against 11, and what the difference costs The comparison worth drawing is between the taxonomy a large network now collects and the one this scanner ships, because the second is much smaller and the gap is not flattering. BOT_PURPOSES in core/src/bots.ts, read on 29 August 2026, holds three values: training, search and user_agent. The 15 tokens in BOT_REGISTRY divide 9, 3 and 3 between them. Those three classes are a design decision recorded in a source file, not a finding about crawlers, and they were chosen to answer one question, which is whether a given token is fetching to train a model, fetching to build an index, or fetching because a person just asked something. Against Cloudflare's 11 behavior classes, that vocabulary has no way to say Transact, no way to say Feed Fetching, and no way to record that a single bot performs two of them at once, which the BotBase form explicitly allows. For what Lantad measures, the narrower vocabulary is mostly adequate, and saying why is more useful than defending it. This scanner grades whether a page is readable by a crawler that arrives, and [the methodology page](https://lantad.co/methodology) sets out what a scan claims and what it refuses to claim from one. Whether the arriving client also does ad verification or uptime monitoring changes nothing about whether the prose is in the HTML. Where the narrower vocabulary does cost something is at the edge of [what AI visibility means](https://lantad.co/glossary/ai-visibility) as a question: a reader who wants to know whether an agent will be able to complete a checkout on their site is asking about the Transact behavior, and nothing in this repository models it. There is a second cost, and it is the one this site has already published against itself. Three of the 15 registry entries carry no published User-Agent string at all, and the three Anthropic entries are recorded in the file's own header comment as widely reported strings taken on trust rather than sourced from a vendor page. A registry with an unverified row in it is exactly the artefact this post has been describing from the outside, and the honest position is that Lantad's list has the same defect as everybody else's: it records claims, and it labels the ones it could not confirm. The [research index](https://lantad.co/research) collects the measurements that do have sources behind them, and the [crawlability study](https://lantad.co/research/crawlability-study) is where the sampled figures live. ## Questions and answers **Can you detect an LLM crawler from the User-Agent string alone?** No, and often not even partially. Any client can send any string, so a match proves a claim was made rather than that a particular operator made the request. It is also incomplete as a claim: read at nine vendors' own documentation on 27 August 2026, only 6 of the 15 robots.txt tokens in Lantad's crawler registry carried a complete User-Agent string that could be matched literally, with three describing no request at all and two published with placeholder text inside them. Verification needs a second signal, which in practice means an IP list, a forward confirmed reverse DNS lookup, or a signature. **Does a bot being listed in Cloudflare's directory mean its stated purpose is true?** The directory records what the operator declared and what the automated review could confirm, and those are different sets. The BotBase for Operators post of 28 August 2026 states that the review checks for duplicates, checks that the User-Agent pattern is specific enough, and validates the claimed verification method by fetching the IP list, confirming reverse DNS or validating the Web Bot Auth signature. Behavior, content use and operator type are declarations. Cloudflare assesses conduct separately and over time, and its policy page lists breaches that can remove a service from the allowlist, including a disclosed purpose that does not reflect the traffic. **What is the difference between a Direct and an Intermediary operator?** Cloudflare's verified bots documentation, read on 29 August 2026, defines Direct as a bot operated by a single, narrow operator, usually on its own infrastructure, where only that operator can send requests presenting as that bot. Intermediary describes an agentic service that a wide range of end users can operate, where the operator runs the software but each action is initiated by a different end user. The distinction matters because a site owner may trust the intermediary operator without trusting every end user driving it, so the party responsible for a request is not always the party that built the client. **Can Lantad tell me which AI crawlers actually reached my site?** No. Lantad scans a hostname from the outside, which means it can send a named user agent and record what the server answers, but it never sees your inbound traffic. A signature header, a verification verdict and a directory classification all exist either in your own logs or at a network edge, and none of them appears in an external scan. What a scan can establish is the other half of the question: whether robots.txt permits a given token, and whether a page served under that name carries the same text as the baseline fetch. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot