Free tools / AI crawler directory
Every AI crawler, in one table.
The 15 robots.txt tokens the Lantad scanner checks, from 9 vendors, with what each one is for, whether it publishes a user agent we can probe live, and the robots.txt lines that control it.
- 15 robots.txt tokens
- 9 vendors
- 12 probed live
One policy.
Two different answers.
The policy most sites want: keep the crawlers that can put you in a live answer, turn away the ones that collect training data. Five of the 15 tokens under that policy, with the purpose class that decided each one.
OAI-SearchBot, AI search, allowed
ChatGPT-User, on-demand fetch, allowed
Claude-SearchBot, AI search, allowed
GPTBot, training, blocked
ClaudeBot, training, blocked
The full policy covers all 15 tokens, and the copy-paste version is in the fix list below.
An example policy, not a measurement of any site. This page fetches nothing. To evaluate your own file, use the robots.txt tester.
What this page cannot tell you
- Whether your own robots.txt allows a given token. That needs your file, and the robots.txt tester reads it.
- Whether your server treats a crawler's user agent differently from an ordinary visitor. That needs a request sent under the crawler's own agent, which is what What does GPTBot see on your page? does.
- Whether the page a crawler reaches contains any readable text. That is the measurement a full scan makes.
- Whether any crawler obeyed anything. robots.txt is a request, not a firewall, and this page can give you the correct product token without promising anyone honours it.
15 tokens, 9 vendors
| Crawler | Vendor | What it controls | How we check it | User agent sent | Docs |
|---|---|---|---|---|---|
GPTBot |
OpenAI | Training data | Rule plus live probe | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot | Docs |
OAI-SearchBot |
OpenAI | Answering search | Rule plus live probe | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot | Docs |
ChatGPT-User |
OpenAI | A user's request | Rule plus live probe | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot | Docs |
ClaudeBot |
Anthropic | Training data | Rule plus live probe | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | Docs |
Claude-SearchBot |
Anthropic | Answering search | Rule plus live probe | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com) | Docs |
Claude-User |
Anthropic | A user's request | Rule plus live probe | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) | Docs |
anthropic-ai |
Anthropic | Training data | Rule only, no probe | No request user agent. Cannot appear in a log. | Docs |
PerplexityBot |
Perplexity | Answering search | Rule plus live probe | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) | Docs |
Perplexity-User |
Perplexity | A user's request | Rule plus live probe | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) | Docs |
Google-Extended |
Training data | Rule only, no probe | No request user agent. Cannot appear in a log. | Docs | |
Applebot-Extended |
Apple | Training data | Rule only, no probe | No request user agent. Cannot appear in a log. | Docs |
Meta-ExternalAgent |
Meta | Training data | Rule plus live probe | meta-externalagent/1.1 | Docs |
Bytespider |
ByteDance | Training data | Rule plus live probe | Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/) | Docs |
CCBot |
Common Crawl | Training data | Rule plus live probe | CCBot/2.0 (https://commoncrawl.org/faq/) | Docs |
Amazonbot |
Amazon | Training data | Rule plus live probe | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/131.0.0.0 Safari/537.36 | Docs |
Rule-only tokens are switches a vendor reads inside robots.txt. They send no request of their own, so there is nothing to probe and the report says so rather than inventing a result.
Rendered from the scanner's own bot registry, so this table and the crawlers a scan actually evaluates are the same list. Every row carries an id, so a single crawler can be linked and cited on its own. What each token controls is explained in the next section.
Three kinds of crawler, three different decisions
Every row above carries one of these classes. They are not interchangeable, and blocking them does not cost the same thing.
Training data
Collects pages to train models. Blocking it keeps you out of training data but not out of live AI answers, and it costs nothing in a Lantad scan: a training block is a deliberate owner choice, not a defect.
Answering search
Fetches pages to answer live queries and cite sources. This is the class that puts you in the answer, so a scan treats a block here as a real access loss.
A user's request
Fetches one page on demand because a person asked an assistant about that exact URL. Someone was already interested enough to ask, and the assistant went and looked. A block here is an access loss too.
A scan deducts Access score only when the AI search and on-demand fetch classes are disallowed. Blocking a training crawler is a content decision and costs nothing in the grade. The deduction is set out in the methodology.
Set a policy, then check it landed
Reputable AI crawlers read robots.txt and honour it. It is a request, not a firewall: enforcement belongs at your edge.
- Decide by purpose, not by vendor. The policy most sites want keeps the 6 crawlers that can put you in a live answer and turns away the 9 that collect training data.
- Copy the correct tokens, do not guess them. The recipes below that name tokens are rendered from the same registry as the table, so they cannot name a token that does not exist.
- Test the file you actually published. Which AI crawlers does your robots.txt allow? evaluates every token against your file and quotes the deciding line.
- Check your edge is not overriding it. What does GPTBot see on your page? reports the status your server returned to a request that identified as the crawler.
- Then make sure the page has text to read. Access is the doorway; readability is the room. Start at Fix AI visibility by stack, with Why AI crawlers can't read your Next.js site, Why AI crawlers can't read your React site and Why AI crawlers can't read your Shopify site covering the stacks this happens on most.
4 positions, written out
Most sites want one of these. Pick the paragraph that describes your position and the file underneath it is the whole change.
Let everything in
The default position for most marketing sites: you want to be described, quoted and cited, and there is nothing on the site you would withhold.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlAllow AI search and on-demand fetches, block training
The publisher's position. Search and user-request crawlers may fetch a page to answer a question about it; training crawlers may not take it wholesale.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: CCBot
User-agent: Amazonbot
Disallow: /Keep one section private
You are happy to be read, except for the parts that are not for public summarising: an internal knowledge base, a customer portal, staging content.
User-agent: *
Allow: /
Disallow: /account/
Disallow: /internal/
Disallow: /staging/Block every AI crawler
The full refusal: every one of the 15 tokens named and turned away. The list renders from the registry, so it cannot omit a token the scanner knows.
User-agent: *
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: CCBot
User-agent: Amazonbot
Disallow: /anthropic-ai, Google-Extended and Applebot-Extended are in the token-naming recipes above and will never appear in your logs, because they publish no request user agent. The instruction still lands: it is read at crawl time and enforced through the vendor's ordinary crawler. We wrote up what each of the three actually controls in the crawler tokens that never appear in your logs.
A token is not a crawler, and this is where most robots.txt files go wrong. Some tokens name a bot that will knock on your server with a published user agent. Others exist only inside robots.txt, as a switch a vendor reads before deciding whether to use your content, with no request to observe and nothing to probe.
Lantad reads all 15 in your robots.txt and probes the 12 that publish a user agent. The remaining 3 are reported as rule-only, with the reason, rather than implying a test that never happened.
Common questions
Where does this list come from, and how current is it?
From the bot registry the scanner itself runs: 15 tokens from 9 vendors. Every row on this page, including each user agent string and each robots.txt snippet, is rendered from that registry at build time, so the directory cannot name a crawler the scanner does not check or leave out one it does. The user agent strings were recorded from vendor documentation, and each row links the vendor page it came from so you can check it against the source.
Why do 3 rows show no user agent?
Because anthropic-ai, Google-Extended and Applebot-Extended are robots.txt product tokens, not crawlers. They publish no request user agent of their own, so no request ever arrives carrying those names and a log search for them returns zero on every site, forever. That zero is not evidence of anything. The other 12 tokens do publish a user agent, so they are the only ones a log file or an edge rule can observe directly. What each of the three controls is written up in the crawler tokens that never appear in your logs.
Why are there three tokens for one vendor?
Because vendors separate the jobs. OpenAI and Anthropic each publish separate tokens for training, AI search and user-request fetches, and a site can hold a different position on each. Blocking training while allowing search is a coherent policy, and it needs separate lines to express.
What does the "What it controls" column mean for my score?
Purpose is the class of work the crawler does: training, AI search, or on-demand fetch. A scan deducts Access score only when the AI search and on-demand fetch classes are disallowed, because those are the ones that decide whether you can appear in a live answer. Blocking a training crawler costs nothing in the score: it is a choice about your content, not a defect in your site. The deduction is set out in the methodology.
Does blocking a training token remove me from a model?
No. It signals that your content should not be taken for future training. It does nothing about content already collected, and it does not remove you from a model that was trained before the rule existed.
Does adding these robots.txt lines actually stop a crawler?
It instructs one. robots.txt is a voluntary standard: reputable AI crawlers read it and honour it, and a crawler that ignores it is not stopped by a text file. If you need enforcement rather than a request, that belongs at your edge or in your server config. This page can give you the correct product token; it cannot promise anyone obeys it.
If my robots.txt allows a crawler, can it definitely reach me?
No, and this is the most common surprise on the whole site. A CDN rule or bot-management setting can block a crawler your robots.txt allows. That is why a full Lantad scan probes the 12 tokens that publish a user agent live and reports how your server actually answered, rather than trusting the file.
Does this page check my own site?
No. It fetches nothing and knows nothing about your domain: it is a static reference rendered from the registry. To see which of these tokens your own robots.txt allows, and the exact rule that decides each one, use the robots.txt tester. To see what a crawler actually receives from a page, use What GPTBot sees.
Free to reuse and republish with attribution and a link to this page. This directory is published under CC BY 4.0.
Now check your own file.
The directory tells you what exists. The tester tells you which of these 15 your robots.txt currently allows, and the full scan tells you whether your server agrees with it.