Free tools / robots.txt tester
Which AI crawlers does your robots.txt allow?
Paste your robots.txt or fetch one from a domain and see which of the 15 named AI crawler tokens each rule allows or blocks at the site root, plus the lines the parser had to skip.
15 tokens.
One resolved rule each.
The shape of a result, not a measurement of anyone. Every token gets the specific rule that governs it, and the tester shows which group in your file produced that answer. Four of the 15 verdicts are shown here.
A wildcard group that welcomes everything, and two named groups that turn two vendors away. The bar is the allowed share: 13 of 15 tokens fall through to the wildcard.
GPTBot, allowed by Allow: / in the wildcard group
ClaudeBot, allowed by Allow: / in the wildcard group
PerplexityBot, blocked by Disallow: / in its own group
Google-Extended, blocked by Disallow: / in its own group
The wildcard group applies to any token without a group of its own, so GPTBot and ClaudeBot are allowed by a rule that never names them.
An example file, written for this explanation rather than fetched from a real site. Paste your own above to get all 15 verdicts.
What it cannot tell you
- Whether a crawler obeyed. robots.txt publishes a request and enforces nothing; a crawler that ignores it is not slowed down by a byte of your file.
- Whether your edge blocks anyway. A server, CDN or WAF that answers GPTBot with a 403 while robots.txt says Allow still shows as allowed here, because the file says allowed. Measuring that gap takes a request sent under the crawler's own user agent, which is what What does GPTBot see on your page? does.
- Anything about a sub-path. Every verdict is for the site root, which is where a whole-site block shows up. A rule that disallows only /admin will read as allowed here, correctly.
- Whether the page behind the door has any readable text. Access is the doorway; readability is the room, and a full scan measures both.
Fetch or paste.
Resolve. Report.
Three steps, and the middle one is the whole point: resolution, not a text search for your token's name.
-
Fetch or paste the file
Give it a domain and it fetches the robots.txt, or paste the file directly if you are testing a change before you ship it.
-
Resolve per token
Each of the 15 named AI crawler tokens is matched against the groups in your file the way a crawler would match it, most specific group first.
-
Report, including the skips
Allowed or blocked at the site root per token, the group that produced the answer, and every line the parser had to skip.
What the test returns
A per-token verdict rather than a green tick, because the answer genuinely differs between tokens in most real files.
Every token in the registry
All 15 tokens, evaluated at the site root. The directory lists the same set with the robots.txt line that allows or blocks each one.
The line that decided it
Quoted from your own file, or the note that no rule matched and the default applied. A verdict you cannot trace back to a line is a verdict you cannot act on.
The lines the parser skipped
A line with no colon, an empty user-agent value, an unknown field, and any rule that appears before the first user-agent line. Those bytes do nothing, and a file can look correct while containing them.
How a token gets matched
Crawlers do not read robots.txt the way it looks on the page. These are the rules the evaluator applies, in order: RFC 9309 semantics with the refinements the major crawlers actually implement.
| Step | The rule | What it means for your file |
|---|---|---|
| Grouping | Consecutive user-agent lines head one group | A user-agent line that appears after a rule starts a new group. Rules written before the first user-agent line belong to nothing and are reported as skipped. |
| Group choice | Most specific token, case-insensitive | An exact token match wins; otherwise the longest group token that the crawler token starts with; otherwise the wildcard group. A group headed User-agent: Claude therefore covers ClaudeBot, Claude-SearchBot and Claude-User. |
| Merging | Every group with the winning token merges | Rules for one crawler split across several places in the file are combined before anything is compared, so a duplicate group is not silently ignored. |
| Pattern | Prefix match, with * and a trailing $ | A pattern matches the start of the path. An asterisk matches any run of characters, and a dollar sign at the very end anchors to the end of the path. A dollar sign anywhere else is a literal character. |
| Precedence | Longest pattern wins, Allow breaks ties | When two rules match, the longer pattern decides. When they are the same length, Allow beats Disallow. When nothing matches at all, the path is allowed. |
| Empty values | Disallow with no value restricts nothing | The classic allow-everything idiom is honoured as written, and it can never match a path. |
| Crawl-delay | Read, then ignored | It is not a syntax error and it is not flagged as one, but it changes no allow or block verdict, so it never appears in the result table. |
The tokens that can never appear in a log
3 of the 15 tokens in the registry publish no request user agent at all. They are product tokens: real controls, read from robots.txt at crawl time, enforced through the vendor's ordinary crawler. Searching your access log for one returns zero on every site in the world, and that zero says nothing about whether the vendor is using your content.
-
anthropic-aiAnthropic No published request user agent, so no log line, ever. The tester still evaluates it, because your robots.txt still instructs it. Vendor docs -
Google-ExtendedGoogle No published request user agent, so no log line, ever. The tester still evaluates it, because your robots.txt still instructs it. Vendor docs -
Applebot-ExtendedApple No published request user agent, so no log line, ever. The tester still evaluates it, because your robots.txt still instructs it. Vendor docs
The remaining 12 tokens do publish a user agent, so those are the ones a log file or an edge rule can see. Why the distinction produces confident wrong answers is written up in three AI crawler tokens that will never appear in your logs.
If a crawler you wanted came back blocked
Work down this list. The first three edits are in the file itself; the last two are the reason a correct file can still leave you invisible.
- Clear the skipped lines first. A rule before the first user-agent line, a missing colon or an empty user-agent value does nothing at all, and a file can look right while carrying them.
- Use the correct product token. Guessed tokens are the most common cause of a block nobody meant to write. Every AI crawler, in one table lists every one with copy-paste robots.txt lines.
- Decide by purpose, not by vendor. Blocking a training crawler is a content choice and costs nothing in a scan. Blocking an AI search or on-demand crawler is what removes you from live answers.
- Check your edge is not blocking them anyway. What does GPTBot see on your page? sends a request under the crawler's own user agent and reports the status your server actually returned.
- Then check the page has text to read. Letting a crawler in does nothing if the content needs JavaScript. Fix AI visibility by stack is the index, with Why AI crawlers can't read your Next.js site and Why AI crawlers can't read your React site covering the two stacks this happens on most.
- Re-test. The check is free and takes one paste, so confirming the edit costs nothing.
The same functions decide the Access sub-score in a full report. That makes the evaluator identical, not the answer: this page always asks about the site root, a scan asks about the page it scanned, and a robots.txt that answers 5xx is treated here as absent and in a scan as a block. The scoring side is written up in the methodology.
Three people this is built for
robots.txt is the one file where a single character decides whether an assistant can read your site at all.
SEO and GEO teams
You inherited a robots.txt and need to know what it currently says about AI, token by token, before you touch it.
Engineers
You are about to ship a change and want to test the new file before it goes live rather than after.
Publishers
You want training crawlers blocked and search crawlers allowed, which is a distinction most files do not currently make.
A robots.txt written for search engines rarely says what its author thinks about AI. Wildcards, ordering, and the difference between a training token and a request user agent all change the outcome, and most files were written before half these tokens existed. The tester resolves the rule that actually applies, per token.
Any line the parser could not read is listed rather than silently dropped, because a rule you think is working is worse than a rule you know is broken.
Common questions
What does this tool actually check?
It parses the robots.txt you paste, or the one it fetches from the domain you enter, then evaluates the site root for every AI crawler token in our registry, 15 of them. For each token it reports allowed or blocked and quotes the line that decided it, or says the default applied when no rule matched. It also lists the lines the parser had to skip. The parser and the evaluator are the same ones a full Lantad scan runs.
How is a User-agent line matched to a crawler?
Case-insensitively, and by prefix. A group applies to a crawler when its token equals the crawler token, or is the longest group token the crawler token starts with; when neither exists, the wildcard group applies. So a group headed User-agent: Claude covers ClaudeBot, Claude-SearchBot and Claude-User, and User-agent: GPT covers GPTBot. Every group headed by the winning token is merged into one rule set first, so rules split across the file still count.
Which rule wins when two rules match the same path?
The longest matching pattern wins, and on an equal-length tie Allow beats Disallow. A pattern matches a prefix of the path, an asterisk matches any run of characters, and a dollar sign at the end anchors the pattern to the end of the path. A Disallow with an empty value imposes no restriction, and a path no rule matches is allowed.
Does a Disallow line actually stop a crawler?
No. robots.txt publishes an instruction; it enforces nothing. A crawler that honours the file reads it and stays away, and a crawler that ignores it is not slowed down by a single byte of the file. This tool tells you what your file instructs. It cannot tell you whether any given crawler obeyed, and it cannot see a block applied by your server, CDN or WAF on user agent or IP, which is a separate measurement.
Why does Google-Extended never appear in my access logs?
Because it is a robots.txt product token, not a crawler. 3 of the 15 tokens in the registry publish no request user agent of their own (anthropic-ai, Google-Extended, Applebot-Extended), so a log search for them returns zero on every site, always. They are still real controls: they are read from robots.txt at crawl time and enforced through the vendor's ordinary crawler. The other 12 tokens do publish a user agent, so those are the ones a log can show you.
Which path are the verdicts for?
The site root only. Every verdict on the result page is the answer for the root path, which is the one place a whole-site block shows up. A rule that disallows only a sub-path will show as allowed here, because the root itself is still allowed. Read the deciding-rule column before concluding a crawler has free run of the site.
Which tokens does it test?
All 15 named AI crawler tokens in the Lantad registry, across OpenAI, Anthropic, Perplexity, Google, Apple, Meta, ByteDance, Common Crawl and Amazon. The crawler directory lists every one with its vendor, purpose and documentation link.
Does allowing a token in robots.txt mean the crawler can reach me?
Not necessarily, and this is the most common surprise. A CDN rule or bot-management setting can block a crawler your robots.txt allows. The full scan probes the 12 tokens that publish a request user agent live and reports how your server actually answered.
Why does it show skipped lines?
Because a directive the parser cannot read is a rule that never applied, and nothing in a normal robots.txt workflow tells you that happened. Typos, unsupported syntax and stray characters are the usual causes.
Do you store my file?
Not the file itself. The result is kept: the per-token verdicts, the deciding lines and disallowed paths quoted from your file, and the notes about skipped lines are stored so the result page can render them back to you. That result sits at an unguessable address, carries a noindex header, and is deleted after 18 months.
Find out what your file actually says.
Most robots.txt files were written before these tokens existed. Ten seconds tells you whether yours is doing what you assume.