How to get cited by Claude

How Claude reads the web

Anthropic operates named crawler tokens including ClaudeBot, Claude-SearchBot, and Claude-User. Per Anthropic's documentation, ClaudeBot collects public web content that may be used to train Anthropic's models, Claude-SearchBot crawls pages to build and improve Claude's search index, and Claude-User handles user-initiated requests, such as when someone asks Claude to read a specific page. Anthropic publishes these user-agent tokens and states that all of them honor robots.txt. Like most AI crawlers, these fetchers work from the raw HTML your server returns; text that appears only after client-side JavaScript runs is at risk of not reaching them. Claude draws on what it can fetch and read, through live fetches and its search index, so being fetched and parsed is a prerequisite for being referenced and not a guarantee of it.

What it takes to be readable and citable

1. Make the page fetch cleanly

   Claude's crawlers need a clean HTTP response. Return a 200 status, avoid long redirect chains, and keep the page from timing out or blocking on size. Lantad's Retrievability check fetches your URL the way a crawler does and reports the status, redirects, and response size, so you can see whether the page is even reachable before anything else.

   • Returns HTML Blocking A 4xx or 5xx ends the scan. Nothing downstream is measurable.
   • Crawler allowed Blocking robots.txt and the live server both have to permit the token.
   • Readable without JS Blocking Most AI crawlers do not execute JavaScript.
   • Structured Improves Headings and schema make the content extractable.
   • Cited Not promised The engine decides. No configuration can guarantee it.

2. Allow the Anthropic tokens in robots.txt

   robots.txt controls whether a given crawler may fetch your pages, and vendors use different tokens. Check that your rules do not block ClaudeBot, Claude-SearchBot, or Claude-User, whether by a named Disallow or an overly broad wildcard. Lantad's Access check reads your robots.txt and the live server response and reports the exact rule that applies to each Anthropic token.

   • Claude-SearchBot Costs you You are absent from the index Claude searches.
   • Claude-User Costs you A direct request to read your page returns nothing.
   • ClaudeBot Your choice Training opt-out. Not penalised by Lantad.
   • anthropic-ai Your choice Also training, and robots-only so there is nothing to probe.

3. Serve your content in the raw HTML

   Because these fetchers work from the HTML your server returns, text rendered only in the browser can reach them as an empty shell. Server-render or statically generate the words that matter so they appear in the raw markup. Lantad measures this as Prose Parity: it compares the text in a raw HTTP fetch against a full browser render and reports how much of the human-visible content a crawler actually receives. High parity means Claude sees closer to what your visitors see.

   Sample Illustrative, not a measurement of any real site.

   What the crawler gets

   • <div id="root"></div>
   • a script bundle
   • no product copy
   • nothing to quote

   What the visitor gets

   • the full copy
   • headings and prose
   • assembled in the browser
   • invisible on the left

4. Give the content a machine-readable structure

   Semantic HTML and a clear heading hierarchy give your content an outline a machine can follow. Use real headings, lists, and landmarks instead of styled div elements. Add structured data (schema.org / JSON-LD) to label what the page is about. Lantad scores Structure and Schema separately so you can see whether your outline and your labeling are legible to a crawler.

   • ✓ One h1, then ordered h2s A machine reads the outline before it reads the prose.
   • ✓ Semantic containers main, article and section, rather than nested unlabelled divs.
   • ✓ A recognised business type in JSON-LD Organization, LocalBusiness, SoftwareApplication and similar.
   • ✓ Server-rendered, not injected JSON-LD added by JavaScript is not in the HTML a crawler receives.

5. Publish substantive, specific content

   Readable and allowed gets you into the candidate set; substance is what makes a page worth referencing. Answer the question directly, state facts plainly, and keep the useful content in the fetchable HTML rather than behind interactions. Lantad cannot promise a citation, and neither can anyone else, but it can confirm that the substance you wrote is present in what Claude receives.

   What you control

   • whether the page fetches
   • whether the token is allowed
   • whether the text is in the HTML
   • whether it is structured

   What you do not

   • whether Claude cites you
   • how sources are selected
   • what the model already knows
   • no tool can promise this