Every AI crawler we track, and what your site does when it asks for
this page. Turning a model training crawler away is a choice, not a
fault, so those are marked apart from the blocks that cost you score.
-
!
12 worth a look
-
✓
3 can reach this page
Nothing is fully blocked. 12 rows need a second look before you call this clean.
-
15 / 15
named crawler tokens allowed in robots.txt
-
0 / 12
live probes got the page (HTTP 2xx)
-
3
rule-only or not probed, so no server response to read
Reaching a page is not the same as reading it. The crawlers marked
allowed here are let in and receive the same HTML a browser gets,
but on this page that HTML is a near-empty shell until JavaScript
runs. What they can actually read is scored separately: see
Part of the rendered content is missing from the initial HTML
below.
Bot access matrix: the AI crawlers your robots.txt or your server treats differently.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
GPTBot
OpenAI
|
Purpose
Training
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
OAI-SearchBot
OpenAI
|
Purpose
AI search
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
ChatGPT-User
OpenAI
|
Purpose
Assistant
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
ClaudeBot
Anthropic
|
Purpose
Training
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
Claude-SearchBot
Anthropic
|
Purpose
AI search
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
Claude-User
Anthropic
|
Purpose
Assistant
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
PerplexityBot
Perplexity
|
Purpose
AI search
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
Perplexity-User
Perplexity
|
Purpose
Assistant
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
Meta-ExternalAgent
Meta
|
Purpose
Training
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
Bytespider
ByteDance
|
Purpose
Training
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
CCBot
Common Crawl
|
Purpose
Training
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
Amazonbot
Amazon
|
Purpose
Training
|
!
Server may be blocking this
|
robots.txtallowed
Allow: /
|
Server responseblocked at the server (HTTP 403)
|
✓
3 other crawlers reach this page normally.
Show them
The AI crawlers that reach this page normally.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
anthropic-ai
Anthropic
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
Allow: /
|
Server responseno request user agent exists to test
|
Google-Extended
Google
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
Allow: /
|
Server responseno request user agent exists to test
|
Applebot-Extended
Apple
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
Allow: /
|
Server responseno request user agent exists to test
|
-
!
Server may be blocking this
Your robots.txt welcomes this crawler, but your server answered its user agent with a block. Nothing in your rules asked for that, so this is more likely blanket bot defense reacting to our test request than a rule aimed at this crawler, and the scan records it at low confidence. Servers that verify crawler IPs will block a spoofed user agent from any scanner; the real crawler may pass.
-
✓
Allowed
Your robots.txt allows this token. It is a robots.txt token only: it sends no user agent of its own, so there is no server response to test and none is implied.