Free tools / AI crawler directory
AI crawler user-agent directory
Every major AI crawler in one place: who runs it, what it is for, the user-agent string it sends, and the exact robots.txt line to allow or block it. 15 crawlers, kept current.
Collects pages to train models. Blocking it keeps you out of training data but not out of live AI answers.
Fetches pages to answer live queries and cite sources. This is the one you usually want to keep.
Fetches a page on demand when a user asks the assistant about that specific URL.
| Crawler | Vendor | Purpose | User-agent sent | Docs |
|---|---|---|---|---|
GPTBot | OpenAI | Training | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot | Docs |
OAI-SearchBot | OpenAI | AI search | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot | Docs |
ChatGPT-User | OpenAI | On-demand fetch | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot | Docs |
ClaudeBot | Anthropic | Training | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected]) | Docs |
Claude-SearchBot | Anthropic | AI search | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; [email protected]) | Docs |
Claude-User | Anthropic | On-demand fetch | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; [email protected]) | Docs |
anthropic-ai | Anthropic | Training | robots token only, no fetch UA | Docs |
PerplexityBot | Perplexity | AI search | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) | Docs |
Perplexity-User | Perplexity | On-demand fetch | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) | Docs |
Google-Extended | Training | robots token only, no fetch UA | Docs | |
Applebot-Extended | Apple | Training | robots token only, no fetch UA | Docs |
Meta-ExternalAgent | Meta | Training | meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) | Docs |
Bytespider | ByteDance | Training | Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected]) | Docs |
CCBot | Common Crawl | Training | CCBot/2.0 (https://commoncrawl.org/faq/) | Docs |
Amazonbot | Amazon | Training | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) | Docs |
How to allow or block them
Add these to your robots.txt. Reputable AI crawlers honor it;
it is a request, not a firewall. Test the result with the
robots.txt tester.
Block every AI crawler
User-agent: *
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: CCBot
User-agent: Amazonbot
Disallow: /Allow AI search, block training
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: CCBot
User-agent: Amazonbot
Disallow: /Most sites should keep the search crawlers: blocking them removes you from the AI answers your customers are already asking for.
Allowing them is only half the job
A crawler you welcome still gets nothing if your page needs JavaScript to render. See what GPTBot actually reads, then scan the whole page.