Every AI crawler we track, and what your site does when it asks for
this page. Turning a model training crawler away is a choice, not a
fault, so those are marked apart from the blocks that cost you score.
-
✕
3 blocked in a way that costs you
-
✓
12 can reach this page
3 of 15 AI crawlers were turned away in a way that costs you: ClaudeBot, Bytespider, Amazonbot. All were refused only in our test, and the real crawler may pass. Start there.
-
15 / 15
named crawler tokens allowed in robots.txt
-
9 / 12
live probes got the page (HTTP 2xx)
-
3
rule-only or not probed, so no server response to read
Reaching a page is not the same as reading it. The crawlers marked
allowed here are let in and receive the same HTML a browser gets.
On this page that HTML is almost empty: the words only appear once code runs in a visitor's browser.
What they can actually read is scored separately: see
Main content is invisible to AI crawlers
below.
Bot access matrix: the AI crawlers your robots.txt or your server treats differently.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
ClaudeBot
Anthropic
|
Purpose
Training
|
✕
Refused in our test
|
robots.txtallowed
|
Server responseblocked at the server (HTTP 520)
|
Bytespider
ByteDance
|
Purpose
Training
|
✕
Refused in our test
|
robots.txtallowed
|
Server responseblocked at the server (HTTP 520)
|
Amazonbot
Amazon
|
Purpose
Training
|
✕
Refused in our test
|
robots.txtallowed
|
Server responseblocked at the server (HTTP 520)
|
✓
12 other crawlers reach this page normally.
Show them
The AI crawlers that reach this page normally.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
GPTBot
OpenAI
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
OAI-SearchBot
OpenAI
|
Purpose
AI search
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
ChatGPT-User
OpenAI
|
Purpose
Assistant
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Claude-SearchBot
Anthropic
|
Purpose
AI search
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Claude-User
Anthropic
|
Purpose
Assistant
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
anthropic-ai
Anthropic
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
|
Server responseno request user agent exists to test
|
PerplexityBot
Perplexity
|
Purpose
AI search
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Perplexity-User
Perplexity
|
Purpose
Assistant
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
Google-Extended
Google
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
|
Server responseno request user agent exists to test
|
Applebot-Extended
Apple
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
|
Server responseno request user agent exists to test
|
Meta-ExternalAgent
Meta
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
CCBot
Common Crawl
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 200)
|
-
✕
Refused in our test
Your robots.txt welcomes this crawler, but your server answered its user agent with a block. Nothing in your rules asked for that, so this may be blanket bot defense reacting to our test request rather than a rule aimed at this crawler. The scan records it at low confidence and counts it at reduced weight, but it still costs Access score: if the real crawler is refused too, it cannot read your page. Servers that verify crawler IPs will block a spoofed user agent from any scanner; the real crawler may pass.
-
✓
Reaches your page
Your robots.txt allows this crawler and your server returned it the same page it returns to a browser. This measures access, not content: whether that HTML is readable without JavaScript is a separate check, scored as retrievability when the page can be graded.
-
✓
Allowed
Your robots.txt allows this token. It is a robots.txt token only: it sends no user agent of its own, so there is no server response to test and none is implied.