Every AI crawler we track, and what your site does when it asks for
this page. Turning a model training crawler away is a choice, not a
fault, so those are marked apart from the blocks that cost you score.
-
!
10 worth a look
-
✓
5 can reach this page
Nothing is fully blocked. 10 rows need a second look before you call this clean.
-
15 / 15
named crawler tokens allowed in robots.txt
-
2 / 12
live probes got the page (HTTP 2xx)
-
3
rule-only or not probed, so no server response to read
Reaching a page is not the same as reading it. The crawlers marked
allowed here are let in and receive the same HTML a browser gets,
but on this page that HTML is a near-empty shell until JavaScript
runs. What they can actually read is scored separately: see
Main content is invisible to AI crawlers
below.
Bot access matrix: the AI crawlers your robots.txt or your server treats differently.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
GPTBot
OpenAI
|
Purpose
Training
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
OAI-SearchBot
OpenAI
|
Purpose
AI search
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
ChatGPT-User
OpenAI
|
Purpose
Assistant
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
ClaudeBot
Anthropic
|
Purpose
Training
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
Claude-SearchBot
Anthropic
|
Purpose
AI search
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
Claude-User
Anthropic
|
Purpose
Assistant
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
PerplexityBot
Perplexity
|
Purpose
AI search
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
Perplexity-User
Perplexity
|
Purpose
Assistant
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
Meta-ExternalAgent
Meta
|
Purpose
Training
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
Bytespider
ByteDance
|
Purpose
Training
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
✓
5 other crawlers reach this page normally.
Show them
The AI crawlers that reach this page normally.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
anthropic-ai
Anthropic
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
Allow: /
|
Server responseno request user agent exists to test
|
Google-Extended
Google
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
Allow: /
|
Server responseno request user agent exists to test
|
Applebot-Extended
Apple
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
Allow: /
|
Server responseno request user agent exists to test
|
CCBot
Common Crawl
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
Allow: /
|
Server responsesame page served (HTTP 202)
|
Amazonbot
Amazon
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
Allow: /
|
Server responsesame page served (HTTP 202)
|
-
!
Server may be blocking this
Your robots.txt welcomes this crawler, but your server answered its user agent with a block. Nothing in your rules asked for that, so this is more likely blanket bot defense reacting to our test request than a rule aimed at this crawler, and the scan records it at low confidence. Servers that verify crawler IPs will block a spoofed user agent from any scanner; the real crawler may pass.
-
✓
Allowed
Your robots.txt allows this token. It is a robots.txt token only: it sends no user agent of its own, so there is no server response to test and none is implied.
-
✓
Reaches your page
Your robots.txt allows this crawler and your server returned it the same page it returns to a browser. This measures access, not content: whether that HTML is readable without JavaScript is a separate check, scored as retrievability when the page can be graded.