# AI crawler reference: every token and what it is for

> The 15 AI crawler tokens Lantad tracks, grouped by purpose: training, search indexing, and fetches triggered by a person asking an assistant.

- Canonical page: https://lantad.co/tools/ai-crawlers
- This file: https://lantad.co/tools/ai-crawlers.md

## Key facts

- **Tokens:** 15
- **Purpose classes:** training, search, user agent
- **Tokens that can never appear in your logs:** anthropic-ai, Google-Extended, Applebot-Extended. These are robots.txt tokens with no published user agent, so a log search for them returns zero on every site.

## Every tracked AI crawler token

| Token | Vendor | Purpose | Publishes a user agent | Can be observed in logs |
| --- | --- | --- | --- | --- |
| GPTBot | OpenAI | training | Yes | Yes |
| OAI-SearchBot | OpenAI | search | Yes | Yes |
| ChatGPT-User | OpenAI | user_agent | Yes | Yes |
| ClaudeBot | Anthropic | training | Yes | Yes |
| Claude-SearchBot | Anthropic | search | Yes | Yes |
| Claude-User | Anthropic | user_agent | Yes | Yes |
| anthropic-ai | Anthropic | training | No | No, robots.txt only |
| PerplexityBot | Perplexity | search | Yes | Yes |
| Perplexity-User | Perplexity | user_agent | Yes | Yes |
| Google-Extended | Google | training | No | No, robots.txt only |
| Applebot-Extended | Apple | training | No | No, robots.txt only |
| Meta-ExternalAgent | Meta | training | Yes | Yes |
| Bytespider | ByteDance | training | Yes | Yes |
| CCBot | Common Crawl | training | Yes | Yes |
| Amazonbot | Amazon | training | Yes | Yes |

## What each purpose class means

| Purpose | What the fetch is for | What blocking it costs you |
| --- | --- | --- |
| training | Content taken to train future models | Affects future models, not the answers being given today |
| search | Indexing for retrieval at answer time | This is what makes a page citable NOW. Blocking it removes you from answers |
| user_agent | A person asked an assistant about you and it went and looked | The highest-intent signal of the three, and the one most costly to block |

## Why the purpose split changes the decision

Blocking a training crawler and blocking a search crawler are different choices with different consequences. A training fetch affects future models. A search fetch is what makes a page citable in an answer today. A user-agent fetch means a person asked an assistant about you and it went and looked, which is the highest-intent signal of the three.

## Related

- https://lantad.co/tools/robots-txt-tester (markdown: https://lantad.co/tools/robots-txt-tester.md)
- https://lantad.co/glossary/ai-crawler (markdown: https://lantad.co/glossary/ai-crawler.md)
- https://lantad.co/bot (markdown: https://lantad.co/bot.md)

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot