# Shopify AI crawlers: 38 stores served 1,112 products as open JSON

> Lantad requested /products.json from the 49 hostnames in a 123 host sample whose home pages loaded Shopify storefront assets, on 9 September 2026. 39 answered HTTP 200, 38 of those parsed as a product array holding 1,112 products, and not one of the 49 robots.txt files mentions the path.

- Canonical page: https://lantad.co/blog/shopify-ai-crawlers-1112-products-as-open-json
- This file: https://lantad.co/blog/shopify-ai-crawlers-1112-products-as-open-json.md
- Last substantive update: 2026-09-09

## Key facts

- **Published:** 2026-09-09
- **Category:** Findings
- **Author:** Lantad
- **Length:** 3040 words
- **Takeaway 1:** Shopify AI crawlers are almost always discussed as a robots.txt question, and on 9 September 2026 Lantad measured a second surface: 38 of 49 Shopify signalled hostnames returned a parseable JSON product array at /products.json, carrying 1,112 product objects between them.
- **Takeaway 2:** None of the 49 robots.txt files Lantad read on 9 September 2026 contains the string products.json, while 33 of the 49 disallow /recommendations/products, so those files govern one JSON product endpoint and leave the other unnamed.
- **Takeaway 3:** 889 of the 1,112 product objects carried a non-empty body_html field, holding 62,223 words of description prose once the tags were stripped, a mean of 70 words for each product that carried any.
- **Takeaway 4:** Every one of the 1,112 product objects carried a price on its first variant, 1,066 carried tags and 985 carried a product_type, so the structured fields were near complete on a set where a fifth of the prose was missing.
- **Takeaway 5:** OpenAI's file upload overview, read on 9 September 2026, instructs a merchant to push feeds to OpenAI via SFTP, and Lantad's own scanner requests no product endpoint at all: no file under core/src, worker/src or site/src references that path.

## Summary

Advice about Shopify AI crawlers nearly always ends in the same place, which is the robots.txt file and which tokens it names. That is a reasonable place to end if the only thing an [AI crawler](https://lantad.co/glossary/ai-crawler) can do with a storefront is request HTML pages and read what a theme rendered. It is not the only thing on offer. A storefront on this platform also answers a bulk JSON request for its catalogue, in one round trip, with prices and variants and descriptions already parsed, and it answers it to any client that asks.

On 9 September 2026 we requested the home page of 123 hostnames, once each, sent as LantadBot/1.0 with redirects followed. The list was hand built across direct to consumer retail, news publishing and developer infrastructure, and it was written before any platform detection ran, though it was deliberately stocked with consumer retail, so the share of it that turned out to be Shopify says nothing about the web and everything about how the list was drawn. 49 of the 123 home page responses referenced Shopify storefront assets. For each of those 49 we then requested two more things: /robots.txt, and /products.json. The second request is the one that is rarely made and it is the one this post is about, including what it means for anyone following [the Shopify guide on this site](https://lantad.co/fix/shopify).

## What Shopify AI crawlers can fetch without asking anyone

A page is a rendering decision. A theme takes the product record, chooses which of its fields to print, wraps them in whatever markup the designer preferred, and returns a document that a client then has to take apart again. Every scanner and every crawler that reads a storefront is doing that reconstruction, and [the method behind this scanner](https://lantad.co/methodology) is largely a description of how carefully it is done. The JSON endpoint skips the whole exercise. It returns the record.

Of the 49 hostnames whose home pages referenced Shopify storefront assets, 39 answered a GET of /products.json with HTTP 200. 38 of those returned a body that parsed as a JSON object with a products array in it. One returned HTTP 200 with an HTML document instead, which is a storefront answering a request for a path it does not serve rather than a refusal. Nine returned HTTP 404 and one returned HTTP 403.

One methodological note belongs here rather than in a caption, because it bounds every count in this post. No Shopify specific response header reached this client on any of the 123 requests, so the platform detection rested entirely on asset references in the returned HTML. That is a weaker signal than a header and it is why the endpoint request is treated as the test and the HTML reference only as the reason to run it. The single HTTP 403 was one attempt on one day from one network location and was not retried, so it is recorded as a refusal to this client rather than as a closed endpoint.

The nine 404s matter more than their count suggests, because one of them is a news publication whose home page loads Shopify assets for a merchandise store that does not live at the site root. The HTML signal found it and the endpoint test corrected it. That is the honest reading of the whole sample: an asset reference in a home page says something about how a page was built, and only the request settles what a path returns. It is the same discipline as [reading a real storefront's response rather than its reputation](https://lantad.co/blog/what-a-crawler-meets-on-a-real-storefront), and it is why every figure here comes from a request this scanner made, under [the crawler conduct policy it publishes](https://lantad.co/bot), rather than from an inference about a platform.

## Thirty products is a page, not a catalogue

The 38 stores returned 1,112 product objects between them, and 35 of the 38 returned exactly 30. A number that repeats that precisely across unrelated businesses is not a fact about their catalogues, so we tested it rather than reporting it. On allbirds.com, everlane.com and taylorstitch.com we requested the same path again with a limit parameter of 250, and all three returned 250 products. We then asked for the second page at the same limit, and got 44, 250 and 250 respectively.

So 30 is a default page size and 1,112 is the size of the first page of 38 catalogues. Everlane and Taylor Stitch each published at least 500 products to an unauthenticated client in two requests, and the second page was still full, which means the real figure is higher than anything measured here. This post deliberately does not extrapolate it. The 1,112 is what 38 default requests returned and nothing else, which is the same reason [a sitemap's declared URL count is reported separately from the links a crawl actually found](https://lantad.co/blog/scan-a-website-for-all-pages-985-linked-200712-declared).

That distinction changes what the endpoint is worth to a machine. A crawler working through HTML pays one request per product and has to find the product URLs first. A client using this path pays one request per 250 products and needs no discovery at all, which puts a mid sized catalogue inside a handful of round trips and well inside the kind of byte ceiling that [Googlebot applies to a single document](https://lantad.co/blog/googlebot-reads-the-first-two-megabytes).

## The robots.txt names one JSON endpoint and not the other

All 49 hostnames served a robots.txt at HTTP 200, and every one of the 49 carried a Sitemap line. The files are long: several run past 150 directive lines, and after masking the numeric shop identifier that appears inside the checkout and order paths, no two of the 49 were identical. They are clearly generated from a common template and then diverged.

What the template covers is specific. 46 of the 49 disallow /cart, 33 disallow /recommendations/products, and 19 disallow /search. None of the 49 contains the string products.json anywhere in the file, and none disallows any path ending in .json. So a file that is careful enough to name a JSON product recommendation endpoint, and careful enough to keep crawlers out of faceted collection sorts and preview theme identifiers, leaves the bulk product feed unnamed.

Unnamed is not neutral. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) settles it in the crawler's favour: a path that no Disallow rule matches is allowed, so silence here is permission, and you can see the same evaluation on any file with [the robots.txt tester on this site](https://lantad.co/tools/robots-txt-tester). That is worth holding next to what these files say about AI specifically. Six of the 49 name at least one of the 15 AI crawler tokens in [this scanner's crawler directory](https://lantad.co/tools/ai-crawlers) in a user-agent line. The other 43 name none, which is the same shape found in [the WordPress sample measured the day before](https://lantad.co/blog/wordpress-ai-crawlers-16-of-24-named-none) and the same reason [a wildcard group is doing more work than most site owners think](https://lantad.co/blog/robots-txt-names-the-wildcard-not-the-ai-crawler). The difference on this platform is that the unaddressed crawler is not only being allowed to read the pages. It is being allowed to read the database view behind them.

## How much product prose is actually in there

An open endpoint is only interesting if the thing behind it is worth reading, so we counted what came back. Across the 1,112 product objects, 889 carried a non-empty body_html field. Stripped of markup and entities, those descriptions came to 62,223 words, a mean of 70 words for every product that had any. The distribution is not even: on 19 of the 38 stores every single product carried a description, and on three of them, being kyliecosmetics.com, peakdesign.com and marinelayer.com, not one product did.

The structured fields tell a different story from the prose. All 1,112 objects carried a price on their first variant, 1,066 carried tags and 985 carried a product_type. So the fields a machine can act on without understanding anything were close to complete, while a fifth of the descriptions were absent, which is roughly the split you would expect when merchandising lives in the theme and the record holds only what the admin form required.

That gap is the interesting one for anyone thinking about [prose parity](https://lantad.co/glossary/prose-parity), because it runs the opposite way to the usual finding. The common failure is a page whose text is present for a browser and missing for a crawler. Here the text is missing from the record and present on the page, put there by a theme section that the endpoint never sees. It is a reminder that a catalogue is stored in more than one place, and it sits alongside what we found when [counting the JSON-LD on five captured pages](https://lantad.co/blog/three-of-five-pages-carried-json-ld): [structured data](https://lantad.co/glossary/structured-data) in a page and a product record in a database are different assertions about the same object, and the vocabulary they are written in, such as [the schema.org Product type](https://schema.org/Product), does not make them agree.

## OpenAI asks for the same data over SFTP

The reason to write this down now is that the same fields are being asked for elsewhere, through a much narrower door. OpenAI's commerce documentation sets out a product feed specification, and [its file upload overview](https://developers.openai.com/commerce/specs/file-upload/overview), read on 9 September 2026, states the delivery mechanism in one sentence: push feeds to OpenAI via SFTP. A merchant registers, prepares a file and transmits it. We covered the shape of that file when [twelve of its required fields turned out not to be things a page publishes](https://lantad.co/blog/chatgpt-product-feed-twelve-required-fields), and the delivery method is the part that has not changed.

So there are two routes for one set of facts. On the first, the merchant does nothing and the platform already publishes prices, variants, tags and most descriptions to any client that requests a path no robots.txt names. On the second, the merchant does integration work to hand a curated file to one company over an authenticated channel. The second route is the one being built, and it is the one that carries commercial terms, which is the honest reason it exists rather than an oversight in the first.

What a store owner should take from that is narrow and worth stating plainly. Publishing the endpoint is not a distribution strategy and closing it is not a protection strategy, because neither has been shown here to change what any engine does. Nothing in this measurement observed a single AI crawler requesting the path, and this post makes no claim that any of them do. What it establishes is that the data is reachable, which is a precondition for [being cited](https://lantad.co/how-to-get-cited/chatgpt) rather than evidence of it, and precisely the sort of thing worth knowing before drawing conclusions about [AI visibility](https://lantad.co/glossary/ai-visibility) from page level scans alone.

## What to check on your own store

All of this is checkable in a minute on a site you control, and the checks are worth running in order, because the first two decide whether the rest apply at all.

The first check is the only one that settles anything: request the path and read what comes back. If it returns a products array, everything else in this post applies to your store, and if it returns a document or a 404 then none of it does. The limit and page parameters answer the question the default request hides, which is how much of the catalogue one unauthenticated client can collect and how quickly. Reading a single body_html against the description a shopper actually sees is the check most likely to surprise a merchant, because a theme section that adds copy at render time leaves the record untouched, and the record is what a bulk client receives.

The last check is the one worth writing down somewhere durable. Adding a Disallow line closes the path to a crawler that reads robots.txt and to nothing else, so the decision is about which well behaved clients you want reading the catalogue in bulk, not about whether the data can be taken. Both answers are defensible on those terms.

The inconvenient part of the finding belongs to this scanner rather than to the stores. Lantad does not request this path. No file under core/src, worker/src or site/src references it, which was verified in the repository on 9 September 2026, so every grade this product has ever issued about a storefront was formed from HTML pages alone. That is a real limit and it points at a real gap: a page level scan tells you what a crawler reads from a document, and it says nothing about a bulk endpoint sitting one path away. Seeing [what a page returns to GPTBot](https://lantad.co/tools/what-gptbot-sees) remains the right first question, and it is now visibly not the only one. The same caution applies to the wider claim that an [agent ready storefront performs better](https://lantad.co/blog/agent-ready-storefront-task-success): that result was measured on pages and tasks, not on feeds.

## Questions and answers

**Do AI crawlers fetch /products.json from Shopify stores?**

This measurement does not answer that and does not claim to. Lantad requested the path itself on 9 September 2026 and recorded what 49 hostnames returned. No crawler request was observed, no access log was read, and no vendor documentation examined here names the path. What the measurement establishes is that the data is reachable by an unauthenticated client and that no robots.txt in the sample disallows it, which is a statement about availability and not about use.

**Does robots.txt block products.json on a Shopify store?**

Not on any of the 49 files Lantad read on 9 September 2026. None of them contains the string products.json and none disallows a path by a .json suffix, while 33 of the 49 do disallow /recommendations/products, which is a different JSON product endpoint. Under RFC 9309 a path that no Disallow rule matches is allowed, so an unnamed path is an open one. A store owner who wants it closed has to add the rule, and should expect that a robots.txt rule is a request rather than an enforcement.

**Why did 1,112 products come from 38 stores?**

Because 35 of the 38 returned exactly 30 products, which is the default page size rather than the size of their catalogues. A follow-up probe on allbirds.com, everlane.com and taylorstitch.com returned 250 products each when asked with a limit of 250, and a second page returned 44, 250 and 250. The true catalogue totals are therefore higher than 1,112 and were not measured, so no figure for them appears in this post.

**Does Lantad check this endpoint when it scans a store?**

No. Verified in the repository on 9 September 2026: no file under core/src, worker/src or site/src references products.json, so the scanner has never requested it and no grade it has issued reflects it. Lantad measures what a crawler can read from HTML documents, which is a narrower question than what a storefront publishes in total. That limit is stated here rather than in a footnote because a scan that does not look at something cannot report on it.

---

Lantad measures whether AI crawlers can actually read a page: it fetches as a non-rendering
crawler, renders as a browser, and reports the gap. Free scan, one URL, no signup.

Method and weights: https://lantad.co/methodology | All pages as markdown: https://lantad.co/md | Crawler policy: https://lantad.co/bot