BlogFindings
A robots.txt that blocks every citation crawler still grades B
Lantad's stored fixture for a site that disallows the six AI crawlers capable of citing it scores 78.8 out of 100 and grades B, because access is weighted at 25 points and only 20 of them are lost. An audit of our own scale, read out of our own repository.
This post is an audit of our own scoring rather than a measurement of anybody's site. Every figure below is read out of this repository: the four weights that produce the 78.8 are in core/src/config.ts, the rule deciding which AI crawler blocks cost anything is in core/src/bots.ts, and the scores themselves are stored golden fixtures captured on 15 July 2026. None of the pages involved is a real customer site, and no new scan was run for this piece. What follows sets out why the grade lands that high, why the arithmetic is defensible, and why the letter on the front of a report is the wrong field to read if your actual question is whether an AI system can quote you. The full scoring model is on the methodology page. This is the part of it that looks worst under a hostile reading, which is the reason to write it down rather than wait to be asked.
In short
- Lantad's stored golden fixture synthetic-robots-blocked, captured on 15 July 2026, scores 78.8 out of 100 and carries the letter grade B while its robots.txt disallows the six AI crawlers that can produce a citation.
- Access is weighted at 25 of the 100 composite points in core/src/config.ts, and a robots.txt disallow costs 0.4 of that sub-score per penalised crawler class, so blocking both classes removes 20 points and leaves 80 the scale still awards.
- Blocking a training crawler such as GPTBot, CCBot or Google-Extended costs nothing at all on Lantad's access score, because core/src/bots.ts penalises only the search and user_agent purpose classes and records that choice as deliberate.
- The same constructed test page served with an empty HTML body scores 34.4 and grades F, so Lantad's scale punishes a document a crawler cannot read far harder than a robots.txt that switches every citation crawler off.
- A composite grade is a weighted average and cannot express a fatal failure confined to one dimension, so on any Lantad report the access sub-score and the defect list carry information the letter on the front does not.
| Fixture | What differs | Composite | Grade |
|---|---|---|---|
| synthetic-ssr-perfect | Nothing withheld from crawlers | 100 | A |
| synthetic-noai | A noai/nosnippet directive | 95.1 | A |
| synthetic-robots-blocked | Six citation crawlers disallowed | 78.8 | B |
| synthetic-csr-spa | No prose in the served HTML | 34.4 | F |
What the fixture actually scores when the citation crawlers are blocked
A golden fixture in this repository is a frozen scan: a captured page, the facts gathered from it, and the verdict the scoring engine produced, all stored so that a change in the engine which moves a score fails the build until somebody agrees the move was intended. The fixture named synthetic-robots-blocked holds a constructed page whose robots.txt disallows six named crawler tokens and allows everything else. Its stored verdict has a composite of 78.8, a grade of B, and four sub-scores: parity 1, access 0.2, structure 0.923 and schema 1.
The page itself is clean, and that is the whole point of the fixture. Its prose parity is a perfect 1, meaning the text a browser shows and the text a crawler receives are the same document, with 485 main-content tokens in both. It has a title, a meta description, exactly one h1 and a canonical link. It ships valid structured data. Judged as a document, there is close to nothing wrong with it. Judged as a destination for an AI answer engine, it is switched off, because the six tokens it disallows are OAI-SearchBot, Claude-SearchBot and PerplexityBot, which fetch pages to build AI search answers, and ChatGPT-User, Claude-User and Perplexity-User, which fetch a page when a person asks an assistant about it.
Those are the crawlers that produce citations. The nine tokens the fixture leaves allowed are training crawlers, which feed future models and cite nobody today. So the fixture describes a site that has kept everything that cannot quote it and refused everything that can, and our scale returns a B for it.
The stored defect list does say so, and says so in the strongest language the schema allows. It records robots_block_search at critical severity, naming all three blocked search tokens and their exact matching rules, and robots_block_assistants at major severity, naming the three assistant agents. The headline stored with the verdict reads that the HTML is readable but an access issue keeps some AI crawlers from using it. None of that is hidden. It simply does not move the number very far, and the number is what most people read first.
It is worth being precise about what this fixture is not. It is a synthetic page built to exercise one branch of the scoring code, so nothing here is evidence about how common this configuration is in the wild, and Lantad has published no survey of that. The comparison that follows is between constructed cases in a test suite, which is exactly the right instrument for a question about our own arithmetic and the wrong one for a question about the web. Where a real capture is involved, such as the rate-limited fixture later in this post, it is named as one. The engine's behaviour on a page with no server-rendered text at all is visible on our own client-rendered canary page, which exists to be scanned.
Where the twenty points go, and where the other eighty stay
The composite is a weighted sum of four sub-scores, and the weights are settings rather than findings. They are declared once, in core/src/config.ts, as parity 50, access 25, structure 15 and schema 10, out of 100. Somebody chose those numbers. No measurement established that prose parity is worth half of AI visibility, and this post is not evidence that it is.
The access sub-score starts at 1 and has deductions subtracted from it. A robots.txt disallow costs 0.4 per penalised crawler class, a server-level block observed during probing costs 0.2 scaled by how confident the probe was, and a noai or nosnippet directive costs 0.15. The fixture trips the first of those twice, once for the search class and once for the assistant class, so its access score is 1 minus 0.4 minus 0.4, which is the 0.2 stored in the file. Multiply that loss by the access weight and the cost to the composite is 20 points. The scoring engine prices each of those two defects at exactly 10 points when it renders the report, which is the same arithmetic rather than a second opinion.
That accounting is internally consistent, and it is also the whole problem. The other three dimensions are measuring properties of a document that nobody will fetch. Parity asks whether the text survives the trip from server to crawler. Structure asks whether the headings and metadata are in order. Schema asks whether the machine-readable claims are present and valid. Every one of those questions is conditional on a crawler being allowed to make the request, and when the answer to that prior question is no, the scale keeps awarding the 80 points anyway.
There is a defensible reading, which is that a robots.txt line is the cheapest thing on the entire report to reverse. It is one file, usually one deploy, and it changes for new requests as soon as crawlers re-read the file, which is itself slower than most people expect. A page with no server-rendered prose needs an architectural change. On that reading a score is a measure of distance from good rather than a measure of current harm, and a blocked site is genuinely close. The counter-reading is that a buyer looking at a B does not perform that translation, and a scale whose top-line number needs a paragraph of explanation is not doing its job.
Two mechanical details are worth stating because they change what a disallow means. Under RFC 9309, the robots exclusion protocol, a crawler with no group matching its own product token must obey the group whose user-agent line is the asterisk, so a rule written for something else can become the policy for a crawler nobody considered. And an unreachable robots.txt is not a neutral outcome: as covered in a 404 on robots.txt allows every crawler while a 503 blocks them all, the two failure modes mean opposite things. To see how a specific file resolves for a specific agent and path, the robots.txt tester answers that question directly.
| Deduction | Access sub-score cost | Composite points |
|---|---|---|
| robots.txt disallows AI search crawlers | 0.40 | 10.0 |
| robots.txt disallows AI assistant agents | 0.40 | 10.0 |
| Server-level blocking observed on probe | 0.20 | 5.0 |
| noai or nosnippet directive present | 0.15 | 3.8 |
Why blocking GPTBot costs nothing on this scale
The access penalty does not apply to every blocked crawler. It applies to two purpose classes, search and user_agent, and the list of penalised classes is declared once in core/src/bots.ts so that the score and the report's access matrix cannot disagree about it. Adding a class to that list is a compile error in the verdict module until its wording and defect code are decided, which is a deliberate piece of friction. The comment sitting above the declaration states the reasoning plainly: blocking a training crawler stays a deliberate owner choice that costs nothing.
That is a judgement, and it is worth defending explicitly because it is the second place this scale refuses to punish a block. Of the fifteen tokens in our registry, nine are training crawlers, including GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot and Amazonbot. A site can disallow all nine and lose zero points. The reason is that refusing to supply training data is a rights decision with no bearing on whether the site can be quoted in an answer today, and a visibility scanner that docked points for it would be pricing a licensing position as a technical fault.
The vendors themselves draw the same line, and their documentation is the source for which token does what. OpenAI's crawler documentation describes OAI-SearchBot as used to surface websites in search results in ChatGPT's search features, and ChatGPT-User as used for certain user actions in ChatGPT, visiting a web page when a user asks a question; the same page describes GPTBot as crawling content to train generative AI foundation models. Anthropic's crawler support article, carrying a last updated date of 7 April 2026, describes Claude-SearchBot as navigating the web to improve search result quality and Claude-User as accessing websites when individuals ask questions of Claude, while ClaudeBot collects content that could contribute to training. Perplexity's bots guide documents PerplexityBot and Perplexity-User on the same split.
Reading those three pages together is what makes the penalised set defensible: the vendors publish separate tokens precisely so that a site can refuse training and keep citations, and our scale is built to let that succeed. The awkward consequence is the one in this post. A site can also do the exact opposite, refusing citations while permitting training, and the scale barely notices.
There is one crawler this reasoning does not reach at all, which is Googlebot, and that gap is documented separately in the one crawler our scanner does not model. The count of vendors publishing a usable token is itself smaller than the advice assumes, as set out in six of the nine AI vendors we track publish one crawler token. The current registry, with each token's vendor and purpose, is listed on the AI crawlers reference, and the user agent our own scanner sends is documented on the bot page.
| Purpose class | Tokens | Count | Disallow costs score |
|---|---|---|---|
| search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | 3 | yes, 0.4 |
| user_agent | ChatGPT-User, Claude-User, Perplexity-User | 3 | yes, 0.4 |
| training | GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot, Amazonbot | 9 | no, deliberate |
A weighted average cannot say fatal
The general fault is not the size of any one weight. It is that a weighted average of independent dimensions has no way to express that one dimension is a precondition for the others. Averages describe how much of a thing is present. They cannot describe a gate.
The arithmetic makes this concrete. Take the same constructed page and add a noai directive on top of the two robots blocks, so it is refusing citation crawlers in robots.txt and asking, in the markup, not to be used in snippets. Access falls to 1 minus 0.4 minus 0.4 minus 0.15, which is 0.05. Run that through the published weights with parity, structure and schema unchanged and the composite is 75.1, which is still a B, because the B band starts at 75. That is arithmetic on the configured weights rather than a stored fixture, and no such fixture exists, but the weights and the grade bands it uses are both declared in core/src/config.ts and neither is in dispute. A page can say no in every mechanism our scanner reads and stay in the same letter band as a page with a minor structural flaw.
Compare that with the failure the scale does punish. The synthetic-csr-spa fixture holds a page that serves an empty shell and builds its content in the browser: zero main-content tokens in the raw HTML, 485 after rendering. Its parity is 0, which alone removes 50 points, and its structured data is client-injected, which is capped at a quarter of the schema sub-score because a crawler that does not run JavaScript never sees it. Its composite is 34.4 and its grade is F. Its access score, incidentally, is a perfect 1, since it blocks nobody. So the scale awards full access marks to a page that gives a crawler nothing, and a B to a page that gives a crawler everything and then refuses the request.
Both verdicts are individually reasonable and the pair of them is hard to defend. A reader who takes the grade as a ranking of how likely they are to be cited is being misled in both directions at once.
None of this touches whether a block is honoured, which is a separate question this blog has looked at from the outside. A robots.txt rule is a request rather than an enforcement mechanism, and the measured evidence on compliance is not reassuring: see 234 of 592 sites that ban GPTBot served it a 200 anyway. Our access score prices the rule as written, not the behaviour, and it says so. Entity signals sit outside the composite entirely for related reasons, which is set out under entity confidence.
Sample Illustrative, not a measurement of any real site.
When the scanner withholds the grade instead of lowering it
Lantad does have a mechanism for refusing to produce a number, and the interesting thing is where the line falls. The access computation returns a fatal result in one circumstance only, which is a baseline fetch returning HTTP 400 or above. When that happens the scan is marked access_fatal, the composite, grade, parity, structure and schema are all set to null, and the report says what was measured and what was not instead of printing a letter.
There is a real capture in the fixtures showing this. The allbirds-rate-limited fixture, taken on 15 July 2026, records a baseline fetch answered with HTTP 429. Its stored verdict has a composite of null, a grade of null, an access score of 0, and a headline stating that crawlers cannot fetch the page at all and everything else is moot until the URL serves content. Its parity method is recorded as undefined, because there is no document to compare. That is the correct behaviour and the reasoning behind it is set out in why we will not grade a page we could not measure.
Put the two side by side and the asymmetry is clear. A site that answers a crawler with an error code gets no grade, because the measurement genuinely failed. A site that answers every crawler with a clean 200 and a full page, while its robots.txt tells the citation crawlers not to ask, gets a B, because the measurement succeeded. The withholding rule keys on whether the scanner could measure the page, not on whether an AI system can use it. Those are different questions and the current design answers only the first.
Changing that is a scoring change rather than a copy change, which is why this post describes the behaviour instead of announcing a fix. The options are not obviously equal. A hard cap, where a blocked citation class limits the composite to a stated ceiling, is the most direct and also the most disruptive to anyone tracking a score over time. A separate reachability verdict shown beside the grade rather than folded into it keeps the number stable and admits that one number was never enough. Re-weighting is the least honest of the three, because it would move a configured setting to fix a structural problem and leave the same fault at a different threshold. That decision has not been made, and this post is not the announcement of one. What can be said now is that the current fixture value is 78.8, that the value is intended, and that we would rather publish it than have somebody discover it.
To see what a crawler actually receives from a specific URL, including the robots resolution per token, the what GPTBot sees tool runs the same fetch path. On the broader question of whether a block even stops retrieval, the canary evidence in a robots.txt block did not stop 12 of 18 AI chatbots is worth reading before treating any of this as a lever.
-
HTTP 429 on baseline fetchNo grade issued State access_fatal. Composite, grade and three sub-scores stored as null. Fixture allbirds-rate-limited, 15 July 2026. -
Citation crawlers disallowedGraded B at 78.8 State graded. Two access defects recorded at critical and major severity, worth 10 points each. -
noai directive presentGraded A at 95.1 State graded. One access defect worth 3.8 points. Fixture synthetic-noai. -
Empty HTML, content built in browserGraded F at 34.4 State graded. Parity 0 removes 50 points, and client-injected schema is capped at a quarter.
What to read on a report instead of the letter
The practical advice follows from the arithmetic and applies to any AI visibility score, not only this one. A single composite compresses several independent questions into one number, and compression loses the thing you most need to know, which is whether any of the inputs is a gate rather than a quantity.
Read the access sub-score first, and read it on its own scale. On a Lantad report it runs from 0 to 1 and it is the only field that answers whether a crawler is permitted to fetch the page. Anything below 1 means at least one deduction fired, and the deductions are named. A 0.2 means both citation classes are refused, which is the case this post is about. A 0.85 means a noai or nosnippet directive is present. A 0 means either the deductions summed past 1 or the fetch itself failed, and those two are distinguished by whether a grade was issued at all.
Then read the defect list rather than the score, because the defect list is where severity lives. The scoring engine assigns each defect a severity independently of how many points it costs, which is why robots_block_search is recorded as critical while removing 10 points and a missing meta description is minor while removing about 1.2. Severity is the field that carries the judgement. The points carry only the arithmetic, and on the evidence of this fixture the two do not always agree.
The specific check to run on your own site takes a minute. Fetch your robots.txt, find every group whose user-agent line names one of the six citation tokens, and confirm that none of them, and no asterisk group they fall through to, carries a Disallow that matches the paths you want quoted. Sites acquire these rules without deciding to: a token gets renamed and an old group stops matching anything, which is what happened in a renamed crawler token leaves your robots.txt group matching nothing, or a rule written for crawl budget lands on agents nobody was thinking about.
Whether to allow those crawlers at all is a business decision rather than a technical one, and the measured evidence cuts both ways. Publishers who blocked generative AI crawlers saw an estimated traffic decline, summarised in blocking AI crawlers cost large publishers about 7 percent of traffic, and there are good reasons a rights holder might accept that cost. What this post argues is narrower: whichever choice you make, a scanner should not be able to describe the refusing configuration as a B without saying more, and ours currently does. Platform-specific guidance is at how to get cited in ChatGPT and how to get cited in Perplexity, the vocabulary is defined under generative engine optimization, and the aggregate scan data we are willing to publish is on the crawlability study.
Flow: Composite grade to Was a grade issued at all?; Was a grade issued at all? (no) to No grade: fetch failed; Was a grade issued at all? (yes) to Read the access sub-score; Read the access sub-score (below 1) to Below 1: a crawler is refused; Read the access sub-score (equals 1) to Read the defect severities; Below 1: a crawler is refused to Fix robots.txt before anything else; Read the defect severities to Fix robots.txt before anything else.
Lantad
Published .
Blocking the AI search crawlers in robots.txt is the most complete way there is to remove a site from AI answers. It is one edit, it applies to the whole domain, and unlike a rendering fault it is deliberate. So it is reasonable to expect a scanner that sells AI visibility measurement to treat it as the worst thing it can find. Lantad does not. On our own stored fixture for exactly that case, the composite score is 78.8 out of 100 and the letter grade is B.
Common questions
Does blocking AI crawlers in robots.txt lower a Lantad score?
Only if the blocked crawlers are in the search or user_agent purpose classes, and the effect is smaller than most people expect. Disallowing the three AI search crawlers costs 0.4 of the access sub-score and disallowing the three assistant agents costs another 0.4, which is 20 of the 100 composite points in total. The stored fixture for that configuration, captured on 15 July 2026, scores 78.8 and grades B. Disallowing training crawlers such as GPTBot, CCBot or Google-Extended costs nothing, which is a deliberate decision recorded in core/src/bots.ts.
Which crawler tokens actually produce AI citations?
On the vendor documentation, six of the fifteen tokens in Lantad's registry. OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages to build AI search answers, and ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person asks an assistant about it. The other nine are training crawlers, which feed future models rather than today's answers. That split comes from the crawler documentation OpenAI, Anthropic and Perplexity each publish, not from any Lantad measurement of citation behaviour.
Why does a page with no readable text score worse than a page that blocks every crawler?
Because the composite is a weighted average and the two failures land in different dimensions. Prose parity is weighted at 50 points and access at 25, so a page serving an empty HTML shell loses the full 50 and scores 34.4 with a grade of F, while a page serving perfect HTML behind a robots.txt disallow loses 20 and scores 78.8 with a grade of B. Both stored values are correct arithmetic under the published weights. The pairing is the argument of this post: an average cannot express that access is a precondition for the other three dimensions rather than a quantity alongside them.
When does Lantad refuse to issue a grade?
When the baseline fetch returns HTTP 400 or above. That marks the scan access_fatal and stores the composite, grade, parity, structure and schema as null, because a page that could not be fetched cannot be measured. The allbirds-rate-limited fixture, captured on 15 July 2026, is a real example: HTTP 429, no grade, and a headline saying everything else is moot until the URL serves content. A robots.txt disallow does not trigger this path, because the scanner could still fetch and measure the page. That is the asymmetry this post documents.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
AI visibility
AI visibility is whether AI systems can access, read, and faithfully represent your web pages, the prerequisite for being cited in an AI answer.
Why we will not grade a page we could not measure
Every AI visibility tool prints a number. When the measurement failed, that number is a guess wearing a lab coat, so ours refuses to appear.
AI Overviews and the one crawler our scanner does not model
Getting quoted in an AI Overview starts with Googlebot, and Googlebot is not one of the fifteen tokens in our crawler registry. An audit of our own blind spot, read line by line against our own source.
Six of the nine AI vendors we track publish one crawler token
The standard advice is to disallow training crawlers and allow search crawlers. That instruction needs two published names to write, and Lantad's registry records six of nine vendors publishing exactly one, while Cloudflare reported on 1 July 2026 that mixed-use crawlers are over 36 percent of crawler activity on its network.
Prose Parity
Prose Parity is Lantad's metric for the share of a page's rendered, human-visible text that is present in the raw HTML an AI crawler receives, where 100% means a crawler sees everything a human visitor sees.
