# robots.txt tester for AI crawlers

> Parses a robots.txt and reports, for each of the 15 named AI crawler tokens, whether it is allowed at the site root and which rule decided.

- Canonical page: https://lantad.co/tools/robots-txt-tester
- This file: https://lantad.co/tools/robots-txt-tester.md

## Key facts

- **Tokens checked:** 15
- **Path evaluated:** The site root. A full scan evaluates the page it scanned, which can differ.
- **robots.txt is not access control:** It is an instruction well-behaved crawlers follow. It does not stop anything.

## Related

- https://lantad.co/tools/ai-crawlers (markdown: https://lantad.co/tools/ai-crawlers.md)
- https://lantad.co/bot (markdown: https://lantad.co/bot.md)
- https://lantad.co/glossary/ai-crawler (markdown: https://lantad.co/glossary/ai-crawler.md)

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot