BlogFindings
What a crawler actually meets on a real storefront
Six real captures, measured on one day in July. Two stores on the same platform produced opposite outcomes, and the difference was not their content.
What follows is six specific pages, fetched on 15 July 2026, stored as permanent test fixtures so the numbers cannot drift as we change the code. They are useful precisely because they were not chosen to make a point. They were chosen to calibrate a scoring engine, and the pattern they show turned out to be more interesting than the calibration.
In short
- Lantad captured six real pages on 15 July 2026 and stored them as permanent test fixtures, so the figures cannot drift as the code changes.
- Two Shopify storefronts produced opposite outcomes on the same day: gymshark.com scored 94.3 and an A, while allbirds.com returned HTTP 429 and produced no score at all.
- wikipedia.org scored 91.4 and an A on a structure sub-score of 0.615, the lowest structural reading of the five graded captures.
- astro.build and MDN web docs both landed on a B, at 88.8 and 88.3, and both scored zero on schema.
- astro.build was the only capture in the set to score a perfect 1.0 on prose parity.
Two stores on the same platform, opposite outcomes
Both Allbirds and Gymshark run on Shopify. Gymshark scored 94.3 and an A, with prose parity at 0.958 and structured data present. Allbirds produced no score, because the page never arrived: our crawler, identifying itself honestly as LantadBot with a link to a page explaining what it is, received HTTP 429.
The lesson is not that Shopify is a problem. Gymshark is the counterexample in the same data set. The lesson is that the platform is not what decides whether AI can read you. The bot defense configuration in front of it is, and that is usually a setting rather than an architecture.
allbirds.com
- HTTP 429 to an identified crawler
- content type not HTML
- render skipped
- no composite, no grade
gymshark.com
- fetched and rendered
- composite 94.3, grade A
- parity 0.958, access 0.9
- schema 1.0
The fetch that never became a page
It is worth being precise about how little survives a failed fetch, because this is the case where scoring tools quietly invent things. There was no HTML body. The content type was not text/html. The browser render never ran, so there was nothing to compare a crawler view against.
Access still scored, at zero, because access is read from the response and from robots.txt rather than from the body. Everything derived from the body is null. That is the whole report, and it is the correct whole report.
verdict.json
- state access_fatal
- baselineStatus 429
- access 0
- parity / structure / schema null
- renderOutcome skipped
You can score an A with weak structure
Wikipedia is the interesting row. It scored 91.4 and an A on a structure sub-score of 0.615, the lowest structural reading of the five graded captures. Its prose parity was 0.994, effectively perfect, and its access was clean.
This is the weighting working as intended. Structure is a signal about how easy your page is to navigate and extract; parity is a signal about whether the crawler sees what the visitor sees. A page that server renders everything can afford imperfect structure. A page that renders nothing without JavaScript cannot buy its way out with headings.
Structured data is the cheapest points on the board
The two captures that landed on a B rather than an A were astro.build at 88.8 and MDN at 88.3. Both server render beautifully. Astro scored a perfect 1.0 on prose parity, the only capture in the set to do so. Both scored zero on schema, because neither page carried structured data that named a business entity.
Three of the six captures scored a full 1.0 on schema, so this is not a hard problem. It is the one thing in the list you can fix in an afternoon without touching how your site renders, and the structured data guide covers the shape, and in this data set it was the difference between a B and an A.
Lantad
Published .
This is not a survey. Our public index does not yet hold enough scans to publish one honestly, and rounding six captures up into a percentage would be exactly the kind of claim this product exists to catch.
Common questions
Is Shopify bad for AI visibility?
No, and this data set contains its own counterexample: two Shopify storefronts were captured on the same day and gymshark.com scored 94.3 and an A. What differed was the bot-defense configuration in front of the store, which is usually a setting rather than an architecture.
Why did a site block a crawler that identifies itself honestly?
Rate limiting and bot defense generally do not distinguish between a well-behaved identified crawler and an abusive one. Our fetcher sends LantadBot with a link to a page explaining what it is, and it still received HTTP 429, which is a configuration outcome rather than a decision about us.
Do these six captures represent the web?
No, and we do not present them that way. Six pages chosen to calibrate a scoring engine are not a survey. Our public index does not yet hold enough scans to publish one honestly, so nothing here is expressed as a percentage of the web.
Can a page score an A with weak structure?
Yes. wikipedia.org scored 91.4 and an A on a structure sub-score of 0.615, the lowest structural reading of the five graded captures, because its prose parity was 0.994 and its access was clean. A page that server-renders everything can afford imperfect structure.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.
Fix your Shopify (Liquid) site
Shopify themes are server-rendered Liquid, so most of your store is already readable to AI crawlers, and the real gap is the content third-party apps and widgets inject with JavaScript after the page loads.
Prose Parity
Prose Parity is Lantad's metric for the share of a page's rendered, human-visible text that is present in the raw HTML an AI crawler receives, where 100% means a crawler sees everything a human visitor sees.
Structured data for AI (schema markup)
Structured data is schema.org markup, usually written as JSON-LD, that labels a page's facts in a machine-readable format so AI systems can extract and attribute them without having to interpret the surrounding prose.
AI crawler
An AI crawler is an automated bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so an AI system can read, train on, or cite what those pages contain.
