Every AI crawler we track, and what your site does when it asks for
this page. Turning a model training crawler away is a choice, not a
fault, so those are marked apart from the blocks that cost you score.
-
✕
5 blocked in a way that costs you
-
•
4 your choice, or not tested
-
✓
6 can reach this page
5 of 15 AI crawlers cannot get your content: GPTBot, ClaudeBot, Bytespider, CCBot and 1 more. Start there.
-
6 / 15
named crawler tokens allowed in robots.txt
-
7 / 12
live probes got the page (HTTP 2xx)
-
3
rule-only or not probed, so no server response to read
Reaching a page is not the same as reading it. The crawlers marked
allowed here are let in and receive the same HTML a browser gets,
but on this page that HTML is a near-empty shell until JavaScript
runs. What they can actually read is scored separately: see
Part of the rendered content is missing from the initial HTML
below.
Bot access matrix: the AI crawlers your robots.txt or your server treats differently.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
GPTBot
OpenAI
|
Purpose
Training
|
✕
Blocked at your server
|
robots.txtdisallowed by
Disallow: /
|
Server responseblocked at the server (HTTP 403)
|
ClaudeBot
Anthropic
|
Purpose
Training
|
✕
Blocked at your server
|
robots.txtdisallowed by
Disallow: /
|
Server responseblocked at the server (HTTP 403)
|
Bytespider
ByteDance
|
Purpose
Training
|
✕
Blocked at your server
|
robots.txtdisallowed by
Disallow: /
|
Server responseblocked at the server (HTTP 403)
|
CCBot
Common Crawl
|
Purpose
Training
|
✕
Blocked at your server
|
robots.txtdisallowed by
Disallow: /
|
Server responseblocked at the server (HTTP 403)
|
Amazonbot
Amazon
|
Purpose
Training
|
✕
Blocked at your server
|
robots.txtdisallowed by
Disallow: /
|
Server responseblocked at the server (HTTP 403)
|
anthropic-ai
Anthropic
|
Purpose
Training
|
•
Blocked by choice
|
robots.txtdisallowed by
Disallow: /
|
Server responseno request user agent exists to test
|
Google-Extended
Google
|
Purpose
Training
|
•
Blocked by choice
|
robots.txtdisallowed by
Disallow: /
|
Server responseno request user agent exists to test
|
Applebot-Extended
Apple
|
Purpose
Training
|
•
Blocked by choice
|
robots.txtdisallowed by
Disallow: /
|
Server responseno request user agent exists to test
|
Meta-ExternalAgent
Meta
|
Purpose
Training
|
•
Blocked by choice
|
robots.txtdisallowed by
Disallow: /
|
Server responsesame page served (HTTP 200)
|
✓
6 other crawlers reach this page normally.
Show them
The AI crawlers that reach this page normally.
| Crawler |
Purpose |
Status |
robots.txt |
Server response |
OAI-SearchBot
OpenAI
|
Purpose
AI search
|
✓
Reaches your page
|
robots.txtallowed
Allow: /
|
Server responsesame page served (HTTP 200)
|
ChatGPT-User
OpenAI
|
Purpose
Assistant
|
✓
Reaches your page
|
robots.txtallowed
Allow: /
|
Server responsesame page served (HTTP 200)
|
Claude-SearchBot
Anthropic
|
Purpose
AI search
|
✓
Reaches your page
|
robots.txtallowed
Allow: /
|
Server responsesame page served (HTTP 200)
|
Claude-User
Anthropic
|
Purpose
Assistant
|
✓
Reaches your page
|
robots.txtallowed
Allow: /
|
Server responsesame page served (HTTP 200)
|
PerplexityBot
Perplexity
|
Purpose
AI search
|
✓
Reaches your page
|
robots.txtallowed
Allow: /
|
Server responsesame page served (HTTP 200)
|
Perplexity-User
Perplexity
|
Purpose
Assistant
|
✓
Reaches your page
|
robots.txtallowed
Allow: /
|
Server responsesame page served (HTTP 200)
|
-
✕
Blocked at your server
Your server answered this crawler's user agent with a block instead of the page, and your robots.txt disallows it as well. The two are consistent, but reputable crawlers stop at the server either way and the scan deducts Access score for it. Servers that verify crawler IPs will block a spoofed user agent from any scanner; the real crawler may pass.
-
•
Blocked by choice
Your robots.txt tells this crawler not to read the page. This one only collects pages for model training, so the block costs you nothing here: it keeps your content out of training sets and does not affect whether AI can answer questions about you today.
-
✓
Reaches your page
Your robots.txt allows this crawler and your server returned it the same page it returns to a browser. This measures access, not content: whether that HTML is readable without JavaScript is a separate check, scored as retrievability when the page can be graded.