Every AI crawler we track, and what your site does when it asks for
this page. Turning a model training crawler away is a choice, not a
fault, so those are marked apart from the blocks that cost you score.
-
✕
6 blocked in a way that costs you
-
!
2 worth a look
-
•
4 your choice, or not tested
-
✓
3 can reach this page
6 of 15 AI crawlers cannot get your content: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and 2 more. Start there.
-
3 / 15
named crawler tokens allowed in robots.txt
-
12 / 12
live probes got the page (HTTP 2xx)
-
3
rule-only or not probed, so no server response to read
Reaching a page is not the same as reading it. The crawlers marked
allowed here are let in and receive the same HTML a browser gets,
but on this page that HTML is a near-empty shell until JavaScript
runs. What they can actually read is scored separately: see
Part of the rendered content is missing from the initial HTML
below.
Bot access matrix: the AI crawlers your robots.txt or your server treats differently.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
OAI-SearchBot
OpenAI
|
Purpose
AI search
|
✕
Blocked, costs you
|
robots.txtdisallowed by
Disallow: /
|
Server responsesame page served (HTTP 202)
|
ChatGPT-User
OpenAI
|
Purpose
Assistant
|
✕
Blocked, costs you
|
robots.txtdisallowed by
Disallow: /
|
Server responsesame page served (HTTP 202)
|
Claude-SearchBot
Anthropic
|
Purpose
AI search
|
✕
Blocked, costs you
|
robots.txtdisallowed by
Disallow: /
|
Server responsesame page served (HTTP 202)
|
Claude-User
Anthropic
|
Purpose
Assistant
|
✕
Blocked, costs you
|
robots.txtdisallowed by
Disallow: /
|
Server responsesame page served (HTTP 202)
|
PerplexityBot
Perplexity
|
Purpose
AI search
|
✕
Blocked, costs you
|
robots.txtdisallowed by
Disallow: /
|
Server responsesame page served (HTTP 202)
|
Perplexity-User
Perplexity
|
Purpose
Assistant
|
✕
Blocked, costs you
|
robots.txtdisallowed by
Disallow: /
|
Server responsesame page served (HTTP 202)
|
Meta-ExternalAgent
Meta
|
Purpose
Training
|
!
Different page served
|
robots.txtdisallowed by
Disallow: /
|
Server responsedifferent page served (HTTP 200)
|
CCBot
Common Crawl
|
Purpose
Training
|
!
Different page served
|
robots.txtdisallowed by
Disallow: /
|
Server responsedifferent page served (HTTP 200)
|
GPTBot
OpenAI
|
Purpose
Training
|
•
Blocked by choice
|
robots.txtdisallowed by
Disallow: /
|
Server responsesame page served (HTTP 202)
|
ClaudeBot
Anthropic
|
Purpose
Training
|
•
Blocked by choice
|
robots.txtdisallowed by
Disallow: /
|
Server responsesame page served (HTTP 202)
|
Google-Extended
Google
|
Purpose
Training
|
•
Blocked by choice
|
robots.txtdisallowed by
Disallow: /
|
Server responseno request user agent exists to test
|
Bytespider
ByteDance
|
Purpose
Training
|
•
Blocked by choice
|
robots.txtdisallowed by
Disallow: /
|
Server responsesame page served (HTTP 202)
|
✓
3 other crawlers reach this page normally.
Show them
The AI crawlers that reach this page normally.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
anthropic-ai
Anthropic
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
|
Server responseno request user agent exists to test
|
Applebot-Extended
Apple
|
Purpose
Training
|
✓
Allowed
|
robots.txtallowed
|
Server responseno request user agent exists to test
|
Amazonbot
Amazon
|
Purpose
Training
|
✓
Reaches your page
|
robots.txtallowed
|
Server responsesame page served (HTTP 202)
|
-
✕
Blocked, costs you
Your robots.txt tells this crawler not to read the page, and this crawler feeds live AI answers rather than model training. That block costs Access score and appears in the fix list below.
-
!
Different page served
This crawler's user agent received a page whose text does not match the one an ordinary visitor gets. The scan flags it rather than pricing it, so confirm the difference is intentional: AI and search engines treat unexplained mismatches as cloaking.
-
•
Blocked by choice
Your robots.txt tells this crawler not to read the page. This one only collects pages for model training, so the block costs you nothing here: it keeps your content out of training sets and does not affect whether AI can answer questions about you today.
-
✓
Allowed
Your robots.txt allows this token. It is a robots.txt token only: it sends no user agent of its own, so there is no server response to test and none is implied.
-
✓
Reaches your page
Your robots.txt allows this crawler and your server returned it the same page it returns to a browser. This measures access, not content: whether that HTML is readable without JavaScript is a separate check, scored as retrievability when the page can be graded.