AI crawler detection is an ordering problem

What AI crawler detection actually is, and what it cannot be

The input to a classifier like this is small enough to write on a napkin: a User-Agent string, a Referer string, and the hostname the request was served by. Everything else about the request, the path, the status, the country, is recorded but takes no part in deciding who the client was. The output is one label from a fixed set of six, plus the registry entry the request claimed to be, plus the referring host.

That is a deliberately narrow contract, and the narrowness is what makes the module testable. It performs no I/O, reads no clock, and calls no platform API, so every decision it makes is reproducible from two strings in a unit test. When a classification is wrong, the argument about why is an argument about a string, not about infrastructure. That property is worth more than it sounds: most detection code in the wild is welded to a request object, a CDN and a database, and cannot be argued about at all.

Here is the ceiling, stated before anything else, because a post about detection that hides it is selling something. A classifier of this shape cannot verify anything, and the note in our own classifier records why: Cloudflare publishes a bot management product whose verified bot signal does the work of checking a claim against the network it came from, and that signal is plan gated. Every field the module produces is therefore named as a claim, and every count derived from it downstream is honestly "requests claiming to be X".

The second ceiling is what the label does not tell you. A request labelled ai_training means a client sent a user agent containing a token that a vendor documents as its training crawler. It does not mean your content was used for training. Nobody outside the vendor can measure that, which the checklist in our post on the tokens that never appear in your logs states in as many words: not observable by anyone outside the vendor, for any token, and no tool can measure it including ours. Obedience has a similar shape and its own written limit: the AI crawler reference page says plainly that it cannot confirm a crawler obeyed anything. You can see that a crawler fetched your robots.txt and you can see what it fetched afterwards, but you cannot see the decision in between, which is why the glossary entry for AI crawler is careful about what the word covers.

What is left after those two subtractions is still worth having. You can see which tokens are showing up at all, what status codes they collect, and whether the files written for them are being read. A crawler collecting 404s is a real, fixable finding, and it is a number our internal AI page computes directly: claimed crawler hits minus the hits that got an answer under 400. If you want the outside view of the same question, whether the page a crawler reaches contains readable text at all without JavaScript, that is what the what GPTBot sees check does. Either way it is a smaller claim than "we detect AI crawlers", and the smaller claim is the true one.

The list is finite, and every list is somebody's judgement

A detector can only detect what is on its list. Lantad's registry holds 15 robots.txt product tokens from 9 vendors as of 28 July 2026: OpenAI, Anthropic, Perplexity, Google, Apple, Meta, ByteDance, Common Crawl and Amazon. It is declared as data rather than logic, so adding a vendor is a data change and every surface that reads it updates together, including the public crawler directory, which is rendered from the registry on every request and therefore cannot list a crawler the scanner does not check.

The classifier is forbidden from keeping a second list. It imports the registry, which is the only mechanism that stops the scanner and the traffic dashboard disagreeing about what an AI crawler is. Two copies of a list is how one of them goes stale silently, and a stale list in a detector does not throw an error, it just reports absence. That is the whole reason the constraint is written into the module header rather than left as a convention: a convention survives until the first person in a hurry.

Of the 15 tokens, 12 publish a request user agent and 3 do not. That split is derived by filtering the registry in a single function, and the pages that print it, the crawler directory, the robots.txt tester and the scanner identification page, all call that function rather than carrying a typed count of their own. The three that publish nothing, Google-Extended, Applebot-Extended and anthropic-ai, are the subject of an earlier post about tokens that never appear in your logs and are not re-argued here. The relevant point for detection is narrower: a detector matching on those three will report zero forever, and zero is not the same statement as absence.

The user agent strings themselves came from vendor documentation, transcribed offline, and are flagged in our own deviation log for human confirmation rather than presented as independently verified live traffic. The vendor pages are the primary sources and are worth reading directly: OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic documents ClaudeBot and its siblings, Perplexity documents PerplexityBot and Perplexity-User including published IP ranges, Google documents its crawler fleet including Google-Extended, Common Crawl documents CCBot, and Amazon documents Amazonbot. Meta's crawler documentation lives at developers.facebook.com/docs/sharing/webmasters/crawler, Apple's Applebot-Extended article at support.apple.com/en-us/119829, and ByteDance's at zhanzhang.toutiao.com, none of which we link because they are not on this site's registered outbound host list.

Two entries on that list are judgement calls and are logged as such. Amazonbot is documented by Amazon as improving Alexa answers, which is a mixed search and training role. Classing it as search would make a robots.txt block of it alone cost points in the Access component of the score, so it is classed as training, and the reasoning is recorded where the entry lives rather than left implicit. Not every token in an AI crawler list is even run by a model vendor: CCBot belongs to a non-profit archive whose corpus is then used by many of them, which is a different relationship from GPTBot fetching for OpenAI. Neither of those calls is forced by the evidence, which is exactly why they are written next to the data instead of defended in a support thread later.

Sitting beside the registry is a second, unrelated list: classical search crawlers, present only so that ordinary SEO crawling does not pollute the human count. It is described in the code as deliberately modest, and that word is doing real work. An unlisted engine falls into the generic other_bot class, which is coarse but never wrong, whereas a list claiming to be complete would be wrong silently. That trade, coarse and honest over precise and stale, is the same one behind withholding a letter grade on reports, and the same instinct that keeps the glossary definition of AI visibility narrower than the marketing use of the term.

Ordering decides the answer before matching gets a vote

Here is the part that surprises people who have not written one of these. If your detector checks in the wrong order, it will produce a confident, plausible, wrong label, and no test of the individual patterns will catch it. Every pattern can be correct while the pipeline is broken.

The order in Lantad's classifier is fixed and each phase exists because of the phase after it. An empty user agent is settled first. Then registry tokens, sorted longest first and matched case insensitively. Then classical search crawlers. Then the generic self identification markers. Then human, as the fall-through. Those are two separate properties and they are worth separating carefully, because the interesting case is protected by the phase order and not by the sort.

Take Applebot-Extended, which is the example everyone reaches for because the shorter string applebot sits inside it. What keeps the two apart here is that Applebot-Extended is a registry token and applebot is an entry on the classical search list, so the registry phase runs first and a user agent carrying the extended token never reaches the search list at all. A test pins it: a user agent carrying Applebot-Extended must classify as the registry's training token. A second test pins the mirror case, and it is the one that matters more in practice, because plain Applebot is the real user agent Apple sends. That one must classify as a classical search crawler and must not be attributed to the registry's extended token. A detector that collapsed the two would report Apple training on your content every time Apple indexed a page.

The longest-first sort is a different guarantee with a different status, and the honest version is less dramatic. It orders registry tokens against each other so that the most specific claim wins, and as of 28 July 2026 no registry token is a substring of another, so the sort currently protects against a case that does not exist yet. The comment in the code says as much, describing the token it must beat as hypothetical. It is defensive, it costs one sort at module load, and it means the day a vendor ships a token containing an existing one, nothing has to be remembered.

The generic markers create the shadowing case that is real today. The substring bot appears inside Amazonbot and inside Googlebot. A detector that runs its generic bot check first drops a registry AI training crawler and a classical search crawler into the same undifferentiated bucket, and the bucket is plausible enough that nobody investigates. Two tests pin the outcomes separately: Amazonbot must stay in the AI training class, and Googlebot must land as a search crawler despite containing bot.

There is a fourth ordering property that is structural rather than a matter of sequence. The map from a registry purpose to an audience class is total over the purpose union, which means adding a new purpose class to the registry is a compile error in the classifier until somebody decides what audience it belongs to. Without that, a new purpose would land silently in other_bot, and the registry and the dashboard would disagree about a crawler that had just been added, which is the exact failure the shared list exists to prevent. A runtime test iterates the purposes as well, so the guarantee survives anyone reaching for a type assertion.

If you are building your own detection, this is the section to steal. Write the phase order down, write down why each phase precedes the next, and then write one test per shadowing pair rather than one test per pattern. The patterns will pass either way. Only the pairs tell you the pipeline is sound. The same reasoning applies when you check what a crawler can reach at all, which is the subject of the two layers that decide if AI can read your site.

The bot, crawl, spider check fails in both directions

The folk heuristic for crawler detection is to lowercase the user agent and look for bot, crawl, spider or scrape. It is a reasonable starting point and it is wrong in both directions, which is a harder failure than being wrong in one.

In the direction of false negatives, two common link preview fetchers carry none of those markers and no HTTP client marker either. The string facebookexternalhit/1.1 contains no bot, no crawl and no spider. WhatsApp previews send WhatsApp/2.x, which contains nothing at all that suggests a machine. In our classifier both of them were falling through to human until they were added by name, and the comment recording that says so plainly along with how it was found, which was adversarial review rather than a bug report. Both are pinned by a test now, and the test's title is the finding: link preview fetchers are other_bot, not human.

Think about what that class of error does to a number. Every unfurled link in a group chat becomes a human visit. If you are reading your own traffic to decide whether writing for assistants is working, that is a population of machine requests sitting inside your human count, moving in the same direction as your marketing, and there is no signal in the data that separates them afterwards. The five signals that tell AI who you are are all about what a machine can read; this is about not mistaking the machine for a reader in the first place.

In the direction of false positives, substring matching on bot catches CUBOT, an Android phone brand whose user agent contains the string. That is a real person on a real phone being labelled a crawler. The available fixes are to weaken the match to word boundaries, or to name the exception. Word boundaries lose Slackbot, Twitterbot and every other fetcher whose name ends in bot with a letter in front of it, which is most of them, so the exception is named explicitly and carries its own test with a full CUBOT_NOTE_PLUS user agent in it.

Then the case that reverses most people's intuition: an empty or whitespace only user agent is classified as a bot, not a human. Every mainstream browser sends a user agent, so an absent one is a non browser client that has chosen to identify nothing. Treating a missing header as a missing person and defaulting it to human puts scanners, uptime checks and scripted probes into your visitor count. This is one of the few places where the safe default and the intuitive default point in opposite directions.

Which brings us to the class that should worry you most in any detector, including this one. Human is the fall-through. It is what a request gets when nothing matched, which makes it structurally the class most likely to be wrong, and the ordering comment at the top of the classify function records exactly that. Every classifier of this shape has a residual class, and the residual class is where every new fetcher lands on the day it is invented. If your tooling reports a human number with the same confidence it reports a GPTBot number, the tooling is overstating one of them. That is the same instinct behind publishing what a crawler meets on a real storefront rather than a tidy summary.

A single AI traffic number answers none of the three questions

Once requests are labelled, there is a strong pull toward printing one number: AI traffic, this week, up or down. It is the number every dashboard wants because it fits in a tile. It is also the number that answers none of the questions a site owner actually has.

Our taxonomy is six audience classes, and three of them are AI. The training class means content taken for model training, which affects future models and is a slow, structural fact about your site. The AI search class means indexed for retrieval, which is what makes a page citable in an answer right now, and it is the one most directly tied to whether you show up in what AI says about you. The AI user class means a human asked an assistant about you and the assistant went and looked, which is the highest intent signal in the whole set: somebody was already interested enough to ask. Those are three different facts with three different responses, and summing them produces a number whose movement you cannot act on.

This was not a design that arrived correct. An earlier migration comment sketched four classes with all AI collapsed into one ai_crawler bucket. The shipped taxonomy replaced it with six, and the reason is written into the file that superseded it, along with the verification that the string ai_crawler appeared nowhere in the codebase so there was nothing to migrate. Corrections that are recorded where the mistake was made are the only ones that survive, which is also why the methodology page carries its judgement calls in the open instead of in a changelog nobody reads.

There is a fourth number that people want to fold into the AI total, and it is the most tempting one. A human who arrives on your site from chatgpt.com or perplexity.ai or claude.ai is an AI related visit in every marketing sense. It is not a crawler hit. One is a machine reading your site and the other is a person who asked about you and clicked through, and adding them produces a quantity with no meaning. The classification keeps the assistant referral in a separate field from the audience, and a test asserts that an assistant referrer never changes the audience label, so a downstream sum has to be written deliberately rather than falling out of a shared column. If you want to know whether assistants are sending you people, the how to get cited in ChatGPT and Perplexity citation guides are about that question, and the Claude guide covers the third, and it is a different question from crawling.

Two smaller decisions live in the same function and both were failures before they were features. Only the host is kept from the Referer header, never its path or its query string, because the full referring URL arrives in that header and query strings are where personal data enters analytics. A test asserts that the resulting object contains neither the path nor an email address that was in the query, which is a stronger check than a policy sentence on a privacy page. And a same origin Referer is dropped entirely, because every subresource a browser fetches carries one: without the drop, our own hostname becomes the largest referring site on the dashboard, ranked above every real one, and the referrer dimension mostly measures our own pages loading their own assets. That one was also found by adversarial review, not in production.

Counting is the second place crawler detection goes wrong

Classification produces labels. Everything a person actually reads is arithmetic on those labels, and the arithmetic has its own set of traps. Three of ours are counting traps that apply to any log pipeline, and two more are platform traps that come with the analytics store underneath. Each of the five changes a number a reader would otherwise trust.

The first is what counts as success. The obvious rule is 2xx, and the obvious rule is wrong for crawlers specifically. A 304 Not Modified is a successful crawl of a page that has not changed, and crawlers revalidate constantly, so counting only 2xx would classify every polite revalidation as a failure. Our rule is status under 400, and the consequence of the naive rule is precise rather than vague: a metric built as hits minus successful hits would overstate crawler failures by exactly the amount of the crawler's politeness. The better behaved the crawler, the worse it would look.

The second is assets against pages. Everything is logged with no filter at all, because a crawler fetching your CSS is a real signal and throwing it away at write time is unrecoverable. But only page requests are rolled into the table whose column is called views. Without that filter the highest volume paths on this site are stylesheets, scripts and the favicon, and a stylesheet would sit at the top of a page traffic table. Worse, the same omission had inflated a second table: the dimension rollup was counting every subresource while the traffic rollup counted pages, and a page view drags several subresources behind it, so every country and device figure was inflated against the page number printed next to it. The fix was one filter, hoisted out of the traffic query so both rollups share a single definition of what a page request is. The card over that table calls its number page requests rather than page views, which is the smaller and truer claim.

The third is the exception to the second, and it is the one most likely to bite anyone building this. A blunt extension filter that excludes .txt and .xml removes /robots.txt, /llms.txt and /sitemap.xml, which are the three most interesting requests a crawler can make of your site. Those paths are re-admitted by exact full path, so a page whose slug merely ends in .txt is not swept back in with them, and without the exception the flagship metrics would have been permanently empty for a reason living three files away. If you want to see how those files are supposed to read in the first place, the llms.txt checker is the relevant tool, and the evidence on llms.txt is worth reading before you invest in one.

There are two platform level traps underneath all of that, both specific to Cloudflare's Analytics Engine but with equivalents rather than duplicates elsewhere. Crawler hits are summed by the sample interval rather than counted with COUNT(), because the engine samples adaptively under load and COUNT() counts stored rows rather than requests. And the engine silently drops a data point that violates its size limits, which means an oversized row does not raise an error, it looks exactly like missing crawler traffic. Every field is length capped by the writer rather than trusted, with the claimed bot token capped at 32 characters, because a dashboard reading no crawler visits today when the real cause was one long blob is the confident wrong number this whole codebase exists to avoid.

None of these are findings about crawler behaviour. They are engineering decisions with stated reasons, and the reason they belong in public is that the first three will show up in any log pipeline you build, in roughly the order listed, while the last two will have local equivalents wherever your numbers are actually stored.

Our own scanner is the proof that a header is not a crawler

The cleanest demonstration that a user agent is not an identity is that we send other people's. A Lantad scan issues one request per probeable crawler using that crawler's exact published user agent string, to measure whether your server treats GPTBot differently from an ordinary visitor. Twelve of the fifteen registry tokens publish a user agent, so twelve is the ceiling on those probes, and the three robots only tokens are evaluated against robots.txt instead because there is nothing to send.

That means our scanner generates traffic that any AI crawler detector, ours included, would label as GPTBot or ClaudeBot or PerplexityBot. It arrives from Cloudflare's network, shortly after a request identifying as LantadBot/1.0, and there is at most one per crawler per scan. The whole scan is bounded at 20 requests per scan, one connection at a time per host, with a minimum gap of 500ms, all enforced in code rather than promised in a policy.

We answer the resulting support question in public on the scanner identification page, where the first FAQ entry is why your logs showed GPTBot or ClaudeBot arriving from an IP that does not belong to OpenAI or Anthropic. The answer is that it was probably us. Any tool that runs user agent probes creates the same artefact, and if a detector on your side turns that into a crawler count, the count is measuring a scan rather than a crawler.

The inconvenient half is the next paragraph on that same page. Lantad publishes no fixed IP list for its own scanner, because it runs on Cloudflare Workers and its requests leave from Cloudflare's shared network, so an IP check cannot distinguish our traffic from anyone else's on that network. We ask site owners to verify us by the user agent string and by the shape of the traffic instead: one short burst, sequential, spaced, and capped. That is a weaker verification story than the one we would like to have, and writing it on the page that tells you how to identify us is the only version of it that is honest. Every enforcement finding in a report carries the matching caveat, that a server which verifies crawler IPs may answer the real crawler differently from the way it answered our probe.

So here is the plain summary of what Lantad does and does not do with AI crawler detection, which is the answer to the question that probably brought you here. It ships a classifier, it runs on every request the site Worker serves (lantad.co/api/* is routed to a different Worker and never reaches it, so the dataset is a record of the site rather than of the whole zone), it feeds an internal dashboard nobody outside the company sees, and it is not part of any pricing tier or any API. What Lantad sells is an external scan: it fetches your pages the way a crawler would, checks your robots.txt against each registry token, probes with the published user agents, and tells you what a crawler would meet. That is a different measurement from reading your logs, and the research page publishes an aggregate only once there is a real sample rather than showing a made-up chart.

If you want log based AI crawler detection today, the honest recommendation is to build the classifier yourself against your own access logs, with the ordering and counting rules above, or to buy it from someone who ships it, which the comparison pages cover with dated research. If you want to know what a crawler gets when it arrives, that is what a scan measures, and the free tools do the individual checks one at a time. Both are useful. Only one of them is ours.