Every AI crawler we track, and what your site does when it asks for
this page. Turning a model training crawler away is a choice, not a
fault, so those are marked apart from the blocks that cost you score.
-
•
1 your choice, or not tested
-
✓
14 can reach this page
Nothing here is broken. 1 crawler is turned away by your own rules or was not tested, and neither costs you score.
-
15 / 15
named crawler tokens allowed in robots.txt
-
11 / 11
live probes got the page (HTTP 2xx)
-
4
rule-only or not probed, so no server response to read
Reaching a page is not the same as reading it. The crawlers marked
allowed here are let in and receive the same HTML a browser gets,
but on this page that HTML is a near-empty shell until JavaScript
runs. What they can actually read is scored separately: see
Part of the rendered content is missing from the initial HTML
below.
Bot access matrix: the AI crawlers your robots.txt or your server treats differently.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
Amazonbot
Amazon
|
Purpose
Training
|
•
Allowed, not tested
|
robots.txtallowed
|
Server responseskipped: per-scan request budget reached
|
✓
14 other crawlers reach this page normally.
Show them
The AI crawlers that reach this page normally.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
GPTBot
OpenAI
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
OAI-SearchBot
OpenAI
|
Purpose
AI search
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
ChatGPT-User
OpenAI
|
Purpose
Assistant
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
ClaudeBot
Anthropic
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Claude-SearchBot
Anthropic
|
Purpose
AI search
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Claude-User
Anthropic
|
Purpose
Assistant
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
anthropic-ai
Anthropic
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
|
Server responseno request user agent exists to test
|
PerplexityBot
Perplexity
|
Purpose
AI search
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Perplexity-User
Perplexity
|
Purpose
Assistant
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Google-Extended
Google
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
|
Server responseno request user agent exists to test
|
Applebot-Extended
Apple
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
|
Server responseno request user agent exists to test
|
Meta-ExternalAgent
Meta
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Bytespider
ByteDance
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
CCBot
Common Crawl
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
-
•
Allowed, not tested
Your robots.txt allows this crawler. The live request test did not run for it on this scan, so the server side is unmeasured rather than clean.
-
✓
Reaches your page
Your robots.txt allows this crawler and your server returned it the same page it returns to a browser. This measures access, not content: whether that HTML is readable without JavaScript is a separate check, scored as retrievability when the page can be graded.
-
✓
Allowed
Your robots.txt allows this token. It is a robots.txt token only: it sends no user agent of its own, so there is no server response to test and none is implied.