Every AI crawler we track, and what your site does when it asks for
this page. Turning a model training crawler away is a choice, not a
fault, so those are marked apart from the blocks that cost you score.
Every AI crawler in this list can reach this page.
-
15 / 15
named crawler tokens allowed in robots.txt
-
12 / 12
live probes got the page (HTTP 2xx)
-
3
rule-only or not probed, so no server response to read
Reaching a page is not the same as reading it. The crawlers marked
allowed here are let in and receive the same HTML a browser gets,
but on this page that HTML is a near-empty shell until JavaScript
runs. What they can actually read is scored separately: see
Part of the rendered content is missing from the initial HTML
below.
✓
All 15 AI crawlers we track reach this page normally.
Show them
The AI crawlers that reach this page normally.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
GPTBot
OpenAI
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
OAI-SearchBot
OpenAI
|
Purpose
AI search
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
ChatGPT-User
OpenAI
|
Purpose
Assistant
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
ClaudeBot
Anthropic
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Claude-SearchBot
Anthropic
|
Purpose
AI search
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Claude-User
Anthropic
|
Purpose
Assistant
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
anthropic-ai
Anthropic
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
|
Server responseno request user agent exists to test
|
PerplexityBot
Perplexity
|
Purpose
AI search
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Perplexity-User
Perplexity
|
Purpose
Assistant
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Google-Extended
Google
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
|
Server responseno request user agent exists to test
|
Applebot-Extended
Apple
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
|
Server responseno request user agent exists to test
|
Meta-ExternalAgent
Meta
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Bytespider
ByteDance
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
CCBot
Common Crawl
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Amazonbot
Amazon
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
-
✓
Reaches your page
Your robots.txt allows this crawler and your server returned it the same page it returns to a browser. This measures access, not content: whether that HTML is readable without JavaScript is a separate check, scored as retrievability when the page can be graded.
-
✓
Allowed
Your robots.txt allows this token. It is a robots.txt token only: it sends no user agent of its own, so there is no server response to test and none is implied.