Get cited by AI / Claude

How to get cited by Claude

Claude reaches your page through named Anthropic crawlers that honor robots.txt and work from your raw HTML, so being cited starts with being fetchable, allowed, and readable without JavaScript.

To be cited by Claude, your page has to be reachable and readable by Anthropic's crawlers first. Citation is never guaranteed, but a page Claude cannot fetch or parse is never a candidate.

Anthropic uses several named crawler tokens, and each works from the HTML your server returns. Like most AI crawlers, these fetchers read that raw markup rather than a fully scripted browser render, so content injected client-side, allowed only for other user agents, or buried in non-semantic markup can reach Claude as less than a visitor sees. This page covers what it takes to be readable and allowed, which is the part you control.

How Claude reads the web

Anthropic operates named crawler tokens including ClaudeBot, Claude-SearchBot, and Claude-User. Per Anthropic's documentation, ClaudeBot collects public web content that may be used to train Anthropic's models, Claude-SearchBot crawls pages to build and improve Claude's search index, and Claude-User handles user-initiated requests, such as when someone asks Claude to read a specific page. Anthropic publishes these user-agent tokens and states that all of them honor robots.txt. Like most AI crawlers, these fetchers work from the raw HTML your server returns; text that appears only after client-side JavaScript runs is at risk of not reaching them. Claude draws on what it can fetch and read, through live fetches and its search index, so being fetched and parsed is a prerequisite for being referenced and not a guarantee of it.

What it takes to be readable and citable

  1. Make the page fetch cleanly

    Claude's crawlers need a clean HTTP response. Return a 200 status, avoid long redirect chains, and keep the page from timing out or blocking on size. Lantad's Retrievability check fetches your URL the way a crawler does and reports the status, redirects, and response size, so you can see whether the page is even reachable before anything else.

  2. Allow the Anthropic tokens in robots.txt

    robots.txt controls whether a given crawler may fetch your pages, and vendors use different tokens. Check that your rules do not block ClaudeBot, Claude-SearchBot, or Claude-User, whether by a named Disallow or an overly broad wildcard. Lantad's Access check reads your robots.txt and the live server response and reports the exact rule that applies to each Anthropic token.

  3. Serve your content in the raw HTML

    Because these fetchers work from the HTML your server returns, text rendered only in the browser can reach them as an empty shell. Server-render or statically generate the words that matter so they appear in the raw markup. Lantad measures this as Prose Parity: it compares the text in a raw HTTP fetch against a full browser render and reports how much of the human-visible content a crawler actually receives. High parity means Claude sees closer to what your visitors see.

  4. Give the content a machine-readable structure

    Semantic HTML and a clear heading hierarchy give your content an outline a machine can follow. Use real headings, lists, and landmarks instead of styled div elements. Add structured data (schema.org / JSON-LD) to label what the page is about. Lantad scores Structure and Schema separately so you can see whether your outline and your labeling are legible to a crawler.

  5. Publish substantive, specific content

    Readable and allowed gets you into the candidate set; substance is what makes a page worth referencing. Answer the question directly, state facts plainly, and keep the useful content in the fetchable HTML rather than behind interactions. Lantad cannot promise a citation, and neither can anyone else, but it can confirm that the substance you wrote is present in what Claude receives.

Common questions

Does allowing ClaudeBot guarantee that Claude will cite my page?
No. Being fetchable, allowed, and readable makes your page a candidate for retrieval and use, but no tool or vendor can guarantee a citation. Lantad measures the prerequisites you control and does not claim to influence how Claude selects sources.
What is the difference between ClaudeBot, Claude-SearchBot, and Claude-User?
Per Anthropic's documentation, ClaudeBot collects public web content that may be used to train Anthropic's models, Claude-SearchBot crawls pages to build and improve Claude's search index, and Claude-User handles user-initiated requests, such as when someone asks Claude to read a specific URL. Your robots.txt can set rules for each token by name, and Lantad reports the rule that applies to each.
My page looks fine in a browser. Why would Claude see something different?
Most AI crawlers work from the HTML your server returns and often do not run JavaScript. If your text is injected in the browser after load, a crawler can receive an empty shell instead. Lantad's Prose Parity check shows the gap between a raw fetch and a full render so you can see exactly what is missing.

See what Claude can read on your site

Run a free scan: paste a URL and see exactly what AI crawlers can and cannot read, with a graded report and ranked fixes.

Scan your site free AI visibility glossary