How LantadBot behaves on your site.
We run a crawler, so we owe you the same transparency this product asks of everyone else: what it requests, how often, how it identifies itself, and how to turn it off. Every ceiling and token below is printed from the configuration it runs, so the page changes when the code does, on the same deploy.
LantadBot is the crawler behind
Lantad, the AI visibility checker.
When someone scans a URL with Lantad, our fetcher requests that page
identifying itself honestly as
LantadBot/1.0 (+https://lantad.co/bot). It never follows links, so it reads
the page it was given and nothing it finds inside. One background sample
fetches a page nobody submitted, and only where robots.txt allows it.
One short burst, one host at a time, at least 500ms apart, capped at 20 requests. No links followed. Times and status codes above are illustrative; the user agent is the real string.
What LantadBot asks for
A scan is a small, bounded set of requests. There is no recursive crawl, no link following, and no attempt to enumerate your site.
GET /the-page
Once per scan, twice when content changes between requests
The baseline fetch, with no JavaScript execution. This is the document the parity metric measures.
GET /robots.txt
Once per scan
Resolved for all 15 named AI crawler tokens from a single fetch of the file.
GET /llms.txt
Once per scan
Presence and structure check. A 404 is a normal result and is reported as one.
Headless render
Once per scan
The same URL in a real browser, which loads your assets the way any visitor's browser does.
Per-bot probe
Once per bot
One request per published user agent, for the 12 tokens that have one, to detect blocking at the edge.
A single-page scan is a handful of requests. There is no crawl budget to worry about and no repeated polling.
What we will not do
Constraints we hold ourselves to, stated plainly so you can hold us to them.
-
No recursive crawling
Lantad scans the URLs it is given. It does not follow links to discover more pages, and a free scan never turns into a site-wide crawl you did not ask for.
-
No content republishing
Extracted text is used to compute the score and shown on the report it produced. It is not republished, indexed, or sold.
-
No credential access
LantadBot requests public URLs as an anonymous visitor. It does not log in, submit forms, or attempt to reach anything behind authentication.
-
No sustained load
Requests are rate-limited per host. A busy day of scans on Lantad is a negligible fraction of the traffic any real crawler sends you.
What is kept, and for how long
A scan produces a report at a shareable link. Here is what sits behind that link.
-
The report
Scores, sub-scores, the extracted text used to compute parity, and the findings. It lives at its own URL so anyone with the link can read it without an account.
-
No account required
Free scans are not tied to an identity. There is no profile assembled from the URLs people paste in.
-
Monitoring history
For paid accounts, the score history per monitored page, which is the thing a trend line is made of. It ages out under the same retention rule as everything else: 18 months at most.
-
Nothing sold
Scan data is not sold, licensed, or handed to third parties. Aggregate research is published openly under CC BY 4.0 and names no site without consent.
The enforced limits
Six ceilings, each printed from the same constant the fetcher reads at runtime. Every row is anchored, so one limit can be quoted on its own.
| What is limited | The number | What your server sees |
|---|---|---|
| Requests per scan | 20 max | A hard ceiling counted in code. Probes that do not fit the remaining budget are skipped, and the report says so. |
| Concurrency | 1 request at a time | Strictly sequential fetching; your host is never hit in parallel. |
| Spacing | 500ms minimum gap | The fetcher sleeps between requests to keep the spacing, even when your server responds instantly. |
| Public scans of one domain | 10 per day | Further public submissions that day are refused at intake; nothing is fetched. |
| Repeat scans of one URL | reused for 24 hours | A repeat inside the window returns the existing report instead of fetching again. |
| Stored artifacts | up to 18 months | Then deleted or made inaccessible. The cleanup job reads the same constant this table does. |
Every number above is printed from the same configuration the scanner executes, so this page cannot drift from the code that enforces it.
The same rules, stated as guarantees
- At most 20 fetches per scan from our own fetcher, ever.
- Those fetches run one at a time per host, at least 500ms apart.
- One browser render of the page you asked us to scan, plus one more navigation to capture a screenshot. Both load the page's own subresources the way a visitor's browser does, and the render is repeated if a navigation times out.
- At most 10 scans of any domain per day from the public scan form. Authenticated scans, multi-page runs, and weekly monitoring are bounded by per-key, per-IP, and per-plan budgets instead.
- No link following: one page per scan, plus robots.txt and llms.txt.
- The background Index study reads robots.txt first and does not fetch the page unless it allows
LantadBot. A disallow, a server error, or a file we cannot read all mean we leave the site alone. - Artifacts are kept for up to 18 months, then deleted or made inaccessible.
Which crawler tokens a scan probes
15 tokens are checked on every scan. 12 of them publish a request user agent, so each gets one probe request. The remaining 3 exist only as robots.txt controls and can never appear in a log, so they are evaluated against robots.txt and never probed.
| Token | Vendor | Probed | Published user agent |
|---|---|---|---|
GPTBot |
OpenAI | Yes one request | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot |
OAI-SearchBot |
OpenAI | Yes one request | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot |
ChatGPT-User |
OpenAI | Yes one request | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot |
ClaudeBot |
Anthropic | Yes one request | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) |
Claude-SearchBot |
Anthropic | Yes one request | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com) |
Claude-User |
Anthropic | Yes one request | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) |
anthropic-ai |
Anthropic | No robots.txt only | none published |
PerplexityBot |
Perplexity | Yes one request | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
Perplexity-User |
Perplexity | Yes one request | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) |
Google-Extended |
No robots.txt only | none published | |
Applebot-Extended |
Apple | No robots.txt only | none published |
Meta-ExternalAgent |
Meta | Yes one request | meta-externalagent/1.1 |
Bytespider |
ByteDance | Yes one request | Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/) |
CCBot |
Common Crawl | Yes one request | CCBot/2.0 (https://commoncrawl.org/faq/) |
Amazonbot |
Amazon | Yes one request | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/131.0.0.0 Safari/537.36 |
Free to reuse and republish with attribution and a link to this page. This table is published under CC BY 4.0.
These probes exist so site owners can see their own blocking behavior. Every probe finding in a report carries the caveat that a server verifying crawler IPs may treat the real crawler differently. How probe findings feed the Access score is documented in the methodology.
Verify a request is us
- The exact string
LantadBot/1.0 (+https://lantad.co/bot) - One short burst: robots.txt, llms.txt, the page
- One host at a time, 500ms apart or more
- At most 20 fetches, then it stops
- Up to 12 probes, each a single request
- Following links from page to page
- Sustained traffic outside monitoring or an audit
- Parallel requests to the same host
- A near miss of the string above
- Any claim of a fixed LantadBot IP range
The fetcher identifies itself with this exact string, defined once in the scanner source and sent on every fetch it makes under its own name:
LantadBot/1.0 (+https://lantad.co/bot)
We do not publish a fixed IP list: the scanner runs on Cloudflare Workers and its requests leave from Cloudflare's shared network, so an IP check cannot tell us apart from other Cloudflare traffic. Verify by the string above and by the shape of the traffic instead. A genuine scan is one short burst: robots.txt, llms.txt, and the page itself under our own user agent, sometimes one extra page fetch when content changes between requests, then at most 12 single requests carrying the published user agents of the crawlers listed above. All of it is sequential, at least 500ms apart, and capped at 20 requests. The browser render arrives separately as a normal browser page load; it does not announce LantadBot.
Two of the free tools also fetch on request: What GPTBot sees fetches the page, and the robots.txt tester fetches a domain's robots.txt, each once, using GPTBot's published user agent, because showing what that crawler receives is the point of the tool. Sustained LantadBot-labelled traffic means weekly monitoring re-checks or a whole-site audit run from an account, which reads URLs from the site's own sitemap and checks the opt-out list first. Traffic that claims to be LantadBot while following links across your site is not us.
Opt out
-
Tell us
The complete opt-out: once approved, nothing on Lantad fetches the domain again, on any path. The request and review flow is below.
-
Or block LantadBot at the edge
A CDN or WAF rule on the
LantadBotuser agent stops the baseline fetch immediately, and a scan reports that it was blocked rather than pretending it succeeded. -
robots.txt governs the background sample
The one crawl nobody asked for, the index study, reads robots.txt first and leaves your site alone if it disallows LantadBot. A user-requested scan of a public page is answered to the person who asked, so it is governed by the two routes above.
Site owners can exclude their domains from scanning entirely. Send a
request from an address at the domain in question to
hi@lantad.co
or POST {"domain": "your-domain.com"} to
/api/optout. The API also accepts an optional
proof field for any context that shows you speak for the
domain. Once approved, scans of the domain return an opted-out notice
instead of fetching anything.
What happens next
- Your request lands in a review queue with the domain, any proof you attached, and a timestamp. Processing it fetches nothing from your site.
- A person reviews it. Approval is deliberately manual so a third party cannot de-list a domain they do not control.
- On approval the domain joins the live opt-out list, and every path that would fetch it is refused before the first request: single scans, comparisons, the free fetch tools, the brand check, the opportunity analysis, and both kinds of whole-site audit. Nothing is fetched from an opted-out domain.
- To confirm it took effect, submit your domain to the scan form: you get the opted-out notice back, and no request reaches your server.
Opting out stops new fetching; it does not rewrite history. Artifacts from scans that already ran age out under the retention rule above, 18 months at most.
The baseline fetch always identifies itself honestly as LantadBot. It does not impersonate GPTBot, ClaudeBot or anything else. Per-bot probes are the one exception, and they exist only to detect blocking: each published user agent is sent once, and the result is caveated because servers that verify crawler IPs behave differently for the real crawler.
If a tool that grades honesty were quietly dishonest about its own requests, none of the rest of this site would be worth reading.
Common questions
Why did my logs show GPTBot or ClaudeBot arriving from an IP that does not belong to OpenAI or Anthropic?
That was likely one of our probes. A scan sends one request per AI crawler that publishes a request user agent, using that crawler's exact string, to measure whether your server treats it differently from the baseline. Probes arrive from Cloudflare's network shortly after a request identifying as LantadBot/1.0 (+https://lantad.co/bot), and there is at most one per crawler per scan.
How many requests does one scan send?
At most 20 from our fetcher, a ceiling counted in code. A typical full scan uses fewer: robots.txt, llms.txt, and the page itself under our own user agent, occasionally a second page fetch when content changes between requests, and up to 12 single-request user agent probes. The browser render adds one page load and one screenshot navigation, which fetch the page's subresources the way any visitor's browser does.
Does LantadBot crawl the web on its own?
In one narrow case, and your robots.txt decides that case. Nearly every fetch traces back to something a person asked for: a URL pasted into the scanner, a monitored page being re-checked on its weekly schedule, or a whole-site audit started from an account. Audits read URLs from your sitemap or from a URL list the account supplies, never by following links, and each of these paths is bounded by its own per-account allowance. The exception is the AI Visibility Index study: a background sample that fetches one page per site from a seeded list nobody submitted, so the figures we publish describe the web rather than only the sites that scanned themselves. It reads robots.txt before it fetches the page and leaves the site alone if that file disallows LantadBot, if it answers with a server error, or if it cannot be read at all.
Does opting out remove reports that already exist?
Opting out stops new scanning of the domain once approved, and it also removes the domain from the pages we let search engines index: a side-by-side comparison naming an opted-out domain is served noindex from then on. Reports themselves were never indexed. Artifacts from scans that already ran are governed by the retention rule: kept for up to 18 months, then deleted or made inaccessible. If you want something taken down sooner than that, email us and we will.
Do you respect Crawl-delay?
Requests are rate-limited per host regardless, and a scan is small enough that a crawl-delay directive is not the binding constraint. If our rate is causing you a problem, tell us and we will lower it for your host.
Can I see what LantadBot saw?
Yes. The crawler view tool shows the exact text the baseline fetch extracted from a page, and every scan report shows the same extraction alongside the rendered version it was compared against.
See exactly what our crawler sees.
The whole conduct policy comes down to one thing: nothing we do to your site is hidden from you. The crawler view is the shortest proof of that.