# What a crawler actually meets on a real storefront > Six real captures, measured on one day in July. Two stores on the same platform produced opposite outcomes, and the difference was not their content. - Canonical page: https://lantad.co/blog/what-a-crawler-meets-on-a-real-storefront - This file: https://lantad.co/blog/what-a-crawler-meets-on-a-real-storefront.md - Last substantive update: 2026-07-25 ## Key facts - **Published:** 2026-07-25 - **Category:** Findings - **Author:** Lantad - **Length:** 860 words - **Takeaway 1:** Lantad captured six real pages on 15 July 2026 and stored them as permanent test fixtures, so the figures cannot drift as the code changes. - **Takeaway 2:** Two Shopify storefronts produced opposite outcomes on the same day: gymshark.com scored 94.3 and an A, while allbirds.com returned HTTP 429 and produced no score at all. - **Takeaway 3:** wikipedia.org scored 91.4 and an A on a structure sub-score of 0.615, the lowest structural reading of the five graded captures. - **Takeaway 4:** astro.build and MDN web docs both landed on a B, at 88.8 and 88.3, and both scored zero on schema. - **Takeaway 5:** astro.build was the only capture in the set to score a perfect 1.0 on prose parity. ## Summary This is not a survey. Our public index does not yet hold enough scans to publish one honestly, and rounding six captures up into a percentage would be exactly the kind of claim this product exists to catch. What follows is six specific pages, fetched on 15 July 2026, stored as permanent test fixtures so the numbers cannot drift as we change the code. They are useful precisely because they were not chosen to make a point. They were chosen to calibrate a scoring engine, and the pattern they show turned out to be more interesting than the calibration. ## Two stores on the same platform, opposite outcomes Both Allbirds and Gymshark run on Shopify. Gymshark scored 94.3 and an A, with prose parity at 0.958 and structured data present. Allbirds produced no score, because the page never arrived: our crawler, identifying itself honestly as LantadBot with a link to a page explaining what it is, received HTTP 429. The lesson is not that [Shopify](https://lantad.co/fix/shopify) is a problem. Gymshark is the counterexample in the same data set. The lesson is that the platform is not what decides whether AI can read you. The bot defense configuration in front of it is, and that is usually a setting rather than an architecture. ## The fetch that never became a page It is worth being precise about how little survives a failed fetch, because this is the case where scoring tools quietly invent things. There was no HTML body. The content type was not text/html. The browser render never ran, so there was nothing to compare a crawler view against. Access still scored, at zero, because access is read from the response and from [robots.txt](https://lantad.co/tools/robots-txt-tester) rather than from the body. Everything derived from the body is null. That is the whole report, and it is the correct whole report. ## You can score an A with weak structure Wikipedia is the interesting row. It scored 91.4 and an A on a structure sub-score of 0.615, the lowest structural reading of the five graded captures. Its [prose parity](https://lantad.co/glossary/prose-parity) was 0.994, effectively perfect, and its access was clean. This is the weighting working as intended. Structure is a signal about how easy your page is to navigate and extract; parity is a signal about whether the crawler sees what the visitor sees. A page that server renders everything can afford imperfect structure. A page that renders nothing without JavaScript cannot buy its way out with headings. ## Structured data is the cheapest points on the board The two captures that landed on a B rather than an A were astro.build at 88.8 and MDN at 88.3. Both server render beautifully. Astro scored a perfect 1.0 on prose parity, the only capture in the set to do so. Both scored zero on schema, because neither page carried structured data that named a business entity. Three of the six captures scored a full 1.0 on schema, so this is not a hard problem. It is the one thing in the list you can fix in an afternoon without touching how your site renders, and the [structured data guide](https://lantad.co/glossary/structured-data) covers the shape, and in this data set it was the difference between a B and an A. ## Questions and answers **Is Shopify bad for AI visibility?** No, and this data set contains its own counterexample: two Shopify storefronts were captured on the same day and gymshark.com scored 94.3 and an A. What differed was the bot-defense configuration in front of the store, which is usually a setting rather than an architecture. **Why did a site block a crawler that identifies itself honestly?** Rate limiting and bot defense generally do not distinguish between a well-behaved identified crawler and an abusive one. Our fetcher sends LantadBot with a link to a page explaining what it is, and it still received HTTP 429, which is a configuration outcome rather than a decision about us. **Do these six captures represent the web?** No, and we do not present them that way. Six pages chosen to calibrate a scoring engine are not a survey. Our public index does not yet hold enough scans to publish one honestly, so nothing here is expressed as a percentage of the web. **Can a page score an A with weak structure?** Yes. wikipedia.org scored 91.4 and an A on a structure sub-score of 0.615, the lowest structural reading of the five graded captures, because its prose parity was 0.994 and its access was clean. A page that server-renders everything can afford imperfect structure. --- Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup. Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot