Glossary / AI crawler
AI crawler
Also: AI bot, LLM crawler, AI web crawler, generative AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
An AI crawler is software that AI companies use to collect web content. It requests pages the way a browser does, but instead of showing them to a person it hands the content to a model for training or to an AI assistant answering a live question. Each well-behaved crawler announces itself with a user-agent token, and a site controls it through robots.txt rules that name that token.
The detail that catches most sites off guard is that the common AI crawlers do not run JavaScript. They read the raw HTML your server returns and nothing more. When a page builds its text in the browser after loading, a script-free crawler receives an almost empty shell, so the words a visitor reads never reach the AI system.
What an AI crawler is
An AI crawler is a bot that visits web pages on behalf of an AI company and copies their content for use in a model or an AI answer. Each reputable crawler identifies itself with a user-agent token, and vendors use different tokens for different jobs. OpenAI runs GPTBot, OAI-SearchBot, and ChatGPT-User. Anthropic runs ClaudeBot, Claude-SearchBot, and Claude-User, plus a deprecated anthropic-ai token you may still see listed in older robots.txt files. Perplexity runs PerplexityBot and Perplexity-User. Google uses Google-Extended to govern AI training, separate from Googlebot, which still handles the search index. Apple, Meta, ByteDance, Amazon, and Common Crawl run their own. Some tokens are robots.txt controls only and never send a fetch user-agent, so the token is simply how you name that crawler in your rules.
How AI crawlers differ from Googlebot
The main practical difference is JavaScript. Googlebot renders pages in a headless browser, so text built client-side can still reach the search index. Most AI crawlers do not render JavaScript. A joint analysis by Vercel and MERJ tracked more than 500 million GPTBot fetches and found no evidence of JavaScript execution, and the same held for ClaudeBot, OAI-SearchBot, ChatGPT-User, and PerplexityBot. They read the raw HTML your server returns and stop there. If your content is injected by a framework after the page loads, a crawler that does not run scripts receives a near-empty shell and never sees the words a visitor sees. The second difference is intent. Googlebot builds a search index, while AI crawlers feed training sets and live AI answers, and they respond to their own robots.txt tokens rather than to Googlebot's rules. A page can be fully visible to Googlebot and still be an empty shell to an AI crawler.
Why AI crawlers matter for AI visibility
Being fetched and read is the prerequisite for being cited. An AI system cannot quote, summarize, or attribute a page it could not retrieve or could not parse. Two failures are common and quiet. First, robots.txt or the live server response blocks a named crawler, so it never fetches the page. Second, the crawler fetches the page but receives HTML with little of the visible text in it, because that text depends on JavaScript. In both cases the content is effectively invisible to that AI system, even though the page looks complete in a browser. Reading is necessary, not sufficient. It never guarantees a citation, but without it a citation is impossible.
How to measure and improve AI crawler access
Lantad measures this directly. It fetches a URL two ways: a raw HTTP request that identifies honestly as LantadBot and runs no JavaScript, which is how most AI crawlers fetch, and a full browser render. The gap between the two is content that script-free crawlers never receive. Lantad reports that gap as Prose Parity, the share of rendered, human-visible text present in the raw HTML, and it checks a registry of named crawler tokens against your robots.txt and live server response so you can see which crawlers are allowed. To improve access, serve your main content in the initial HTML rather than injecting it client-side, keep robots.txt from blocking the crawlers you want, return clean responses, and use semantic headings and structured data so machines can parse and label what they read.
Common questions
- Do AI crawlers run JavaScript?
- Most do not. Unlike Googlebot, which renders pages in a headless browser, the common AI crawlers read the raw HTML your server returns and do not execute client-side scripts. Independent analysis of hundreds of millions of GPTBot fetches found no JavaScript execution. Content added to the page by JavaScript after it loads usually does not reach these crawlers, so it will not appear in training data or AI answers.
- How do I allow or block AI crawlers?
- Use robots.txt user-agent rules that name each crawler's token, for example GPTBot, ClaudeBot, or PerplexityBot. Reputable crawlers honor these rules. Note that vendors split their tokens by purpose: blocking a training crawler such as GPTBot does not block the search crawler that fetches pages to answer live questions, so decide token by token.
- How is an AI crawler different from Googlebot?
- Googlebot builds Google's search index and renders JavaScript. AI crawlers fetch pages to train models or answer questions in AI assistants, most do not render JavaScript, and they respond to their own robots.txt tokens such as GPTBot and Google-Extended rather than to Googlebot's rules. A page can be fully visible to Googlebot and still be an empty shell to an AI crawler.
See if AI can read your site
Paste a URL and see exactly what AI crawlers can and cannot read, with a graded report and ranked fixes. Free, no signup.