Blog / Findings
What a crawler actually meets on a real storefront
Six real captures, measured on one day in July. Two stores on the same platform produced opposite outcomes, and the difference was not their content.
This is not a survey. Our public index does not yet hold enough scans to publish one honestly, and rounding six captures up into a percentage would be exactly the kind of claim this product exists to catch.
What follows is six specific pages, fetched on 15 July 2026, stored as permanent test fixtures so the numbers cannot drift as we change the code. They are useful precisely because they were not chosen to make a point. They were chosen to calibrate a scoring engine, and the pattern they show turned out to be more interesting than the calibration.
Two stores on the same platform, opposite outcomes
Both Allbirds and Gymshark run on Shopify. Gymshark scored 94.3 and an A, with prose parity at 0.958 and structured data present. Allbirds produced no score, because the page never arrived: our crawler, identifying itself honestly as LantadBot with a link to a page explaining what it is, received HTTP 429.
The lesson is not that Shopify is a problem. Gymshark is the counterexample in the same data set. The lesson is that the platform is not what decides whether AI can read you. The bot defense configuration in front of it is, and that is usually a setting rather than an architecture.
allbirds.com
- HTTP 429 to an identified crawler
- content type not HTML
- render skipped
- no composite, no grade
gymshark.com
- fetched and rendered
- composite 94.3, grade A
- parity 0.958, access 0.9
- schema 1.0
The fetch that never became a page
It is worth being precise about how little survives a failed fetch, because this is the case where scoring tools quietly invent things. There was no HTML body. The content type was not text/html. The browser render never ran, so there was nothing to compare a crawler view against.
Access still scored, at zero, because access is read from the response and from robots.txt rather than from the body. Everything derived from the body is null. That is the whole report, and it is the correct whole report.
verdict.json
- state access_fatal
- baselineStatus 429
- access 0
- parity / structure / schema null
- renderOutcome skipped
You can score an A with weak structure
Wikipedia is the interesting row. It scored 91.4 and an A on a structure sub-score of 0.615, the lowest structural reading of the five graded captures. Its parity was 0.994, effectively perfect, and its access was clean.
This is the weighting working as intended. Structure is a signal about how easy your page is to navigate and extract; parity is a signal about whether the crawler sees what the visitor sees. A page that server renders everything can afford imperfect structure. A page that renders nothing without JavaScript cannot buy its way out with headings.
Structured data is the cheapest points on the board
The two captures that landed on a B rather than an A were astro.build at 88.8 and MDN at 88.3. Both server render beautifully. Astro scored a perfect 1.0 on prose parity, the only capture in the set to do so. Both scored zero on schema, because neither page carried structured data that named a business entity.
Three of the six captures scored a full 1.0 on schema, so this is not a hard problem. It is the one thing in the list you can fix in an afternoon without touching how your site renders, and in this data set it was the difference between a B and an A.
See what AI can read on your site
Run a free scan and get a graded report of exactly what AI crawlers can and cannot read, with ranked fixes.