Shopify AI crawlers: 38 stores served 1,112 products as open JSON

What Shopify AI crawlers can fetch without asking anyone

A page is a rendering decision. A theme takes the product record, chooses which of its fields to print, wraps them in whatever markup the designer preferred, and returns a document that a client then has to take apart again. Every scanner and every crawler that reads a storefront is doing that reconstruction, and the method behind this scanner is largely a description of how carefully it is done. The JSON endpoint skips the whole exercise. It returns the record.

Of the 49 hostnames whose home pages referenced Shopify storefront assets, 39 answered a GET of /products.json with HTTP 200. 38 of those returned a body that parsed as a JSON object with a products array in it. One returned HTTP 200 with an HTML document instead, which is a storefront answering a request for a path it does not serve rather than a refusal. Nine returned HTTP 404 and one returned HTTP 403.

One methodological note belongs here rather than in a caption, because it bounds every count in this post. No Shopify specific response header reached this client on any of the 123 requests, so the platform detection rested entirely on asset references in the returned HTML. That is a weaker signal than a header and it is why the endpoint request is treated as the test and the HTML reference only as the reason to run it. The single HTTP 403 was one attempt on one day from one network location and was not retried, so it is recorded as a refusal to this client rather than as a closed endpoint.

The nine 404s matter more than their count suggests, because one of them is a news publication whose home page loads Shopify assets for a merchandise store that does not live at the site root. The HTML signal found it and the endpoint test corrected it. That is the honest reading of the whole sample: an asset reference in a home page says something about how a page was built, and only the request settles what a path returns. It is the same discipline as reading a real storefront's response rather than its reputation, and it is why every figure here comes from a request this scanner made, under the crawler conduct policy it publishes, rather than from an inference about a platform.

Thirty products is a page, not a catalogue

The 38 stores returned 1,112 product objects between them, and 35 of the 38 returned exactly 30. A number that repeats that precisely across unrelated businesses is not a fact about their catalogues, so we tested it rather than reporting it. On allbirds.com, everlane.com and taylorstitch.com we requested the same path again with a limit parameter of 250, and all three returned 250 products. We then asked for the second page at the same limit, and got 44, 250 and 250 respectively.

So 30 is a default page size and 1,112 is the size of the first page of 38 catalogues. Everlane and Taylor Stitch each published at least 500 products to an unauthenticated client in two requests, and the second page was still full, which means the real figure is higher than anything measured here. This post deliberately does not extrapolate it. The 1,112 is what 38 default requests returned and nothing else, which is the same reason a sitemap's declared URL count is reported separately from the links a crawl actually found.

That distinction changes what the endpoint is worth to a machine. A crawler working through HTML pays one request per product and has to find the product URLs first. A client using this path pays one request per 250 products and needs no discovery at all, which puts a mid sized catalogue inside a handful of round trips and well inside the kind of byte ceiling that Googlebot applies to a single document.

The robots.txt names one JSON endpoint and not the other

All 49 hostnames served a robots.txt at HTTP 200, and every one of the 49 carried a Sitemap line. The files are long: several run past 150 directive lines, and after masking the numeric shop identifier that appears inside the checkout and order paths, no two of the 49 were identical. They are clearly generated from a common template and then diverged.

What the template covers is specific. 46 of the 49 disallow /cart, 33 disallow /recommendations/products, and 19 disallow /search. None of the 49 contains the string products.json anywhere in the file, and none disallows any path ending in .json. So a file that is careful enough to name a JSON product recommendation endpoint, and careful enough to keep crawlers out of faceted collection sorts and preview theme identifiers, leaves the bulk product feed unnamed.

Unnamed is not neutral. RFC 9309 settles it in the crawler's favour: a path that no Disallow rule matches is allowed, so silence here is permission, and you can see the same evaluation on any file with the robots.txt tester on this site. That is worth holding next to what these files say about AI specifically. Six of the 49 name at least one of the 15 AI crawler tokens in this scanner's crawler directory in a user-agent line. The other 43 name none, which is the same shape found in the WordPress sample measured the day before and the same reason a wildcard group is doing more work than most site owners think. The difference on this platform is that the unaddressed crawler is not only being allowed to read the pages. It is being allowed to read the database view behind them.

How much product prose is actually in there

An open endpoint is only interesting if the thing behind it is worth reading, so we counted what came back. Across the 1,112 product objects, 889 carried a non-empty body_html field. Stripped of markup and entities, those descriptions came to 62,223 words, a mean of 70 words for every product that had any. The distribution is not even: on 19 of the 38 stores every single product carried a description, and on three of them, being kyliecosmetics.com, peakdesign.com and marinelayer.com, not one product did.

The structured fields tell a different story from the prose. All 1,112 objects carried a price on their first variant, 1,066 carried tags and 985 carried a product_type. So the fields a machine can act on without understanding anything were close to complete, while a fifth of the descriptions were absent, which is roughly the split you would expect when merchandising lives in the theme and the record holds only what the admin form required.

That gap is the interesting one for anyone thinking about prose parity, because it runs the opposite way to the usual finding. The common failure is a page whose text is present for a browser and missing for a crawler. Here the text is missing from the record and present on the page, put there by a theme section that the endpoint never sees. It is a reminder that a catalogue is stored in more than one place, and it sits alongside what we found when counting the JSON-LD on five captured pages: structured data in a page and a product record in a database are different assertions about the same object, and the vocabulary they are written in, such as the schema.org Product type, does not make them agree.

OpenAI asks for the same data over SFTP

The reason to write this down now is that the same fields are being asked for elsewhere, through a much narrower door. OpenAI's commerce documentation sets out a product feed specification, and its file upload overview, read on 9 September 2026, states the delivery mechanism in one sentence: push feeds to OpenAI via SFTP. A merchant registers, prepares a file and transmits it. We covered the shape of that file when twelve of its required fields turned out not to be things a page publishes, and the delivery method is the part that has not changed.

So there are two routes for one set of facts. On the first, the merchant does nothing and the platform already publishes prices, variants, tags and most descriptions to any client that requests a path no robots.txt names. On the second, the merchant does integration work to hand a curated file to one company over an authenticated channel. The second route is the one being built, and it is the one that carries commercial terms, which is the honest reason it exists rather than an oversight in the first.

What a store owner should take from that is narrow and worth stating plainly. Publishing the endpoint is not a distribution strategy and closing it is not a protection strategy, because neither has been shown here to change what any engine does. Nothing in this measurement observed a single AI crawler requesting the path, and this post makes no claim that any of them do. What it establishes is that the data is reachable, which is a precondition for being cited rather than evidence of it, and precisely the sort of thing worth knowing before drawing conclusions about AI visibility from page level scans alone.

What to check on your own store

All of this is checkable in a minute on a site you control, and the checks are worth running in order, because the first two decide whether the rest apply at all.

The first check is the only one that settles anything: request the path and read what comes back. If it returns a products array, everything else in this post applies to your store, and if it returns a document or a 404 then none of it does. The limit and page parameters answer the question the default request hides, which is how much of the catalogue one unauthenticated client can collect and how quickly. Reading a single body_html against the description a shopper actually sees is the check most likely to surprise a merchant, because a theme section that adds copy at render time leaves the record untouched, and the record is what a bulk client receives.

The last check is the one worth writing down somewhere durable. Adding a Disallow line closes the path to a crawler that reads robots.txt and to nothing else, so the decision is about which well behaved clients you want reading the catalogue in bulk, not about whether the data can be taken. Both answers are defensible on those terms.

The inconvenient part of the finding belongs to this scanner rather than to the stores. Lantad does not request this path. No file under core/src, worker/src or site/src references it, which was verified in the repository on 9 September 2026, so every grade this product has ever issued about a storefront was formed from HTML pages alone. That is a real limit and it points at a real gap: a page level scan tells you what a crawler reads from a document, and it says nothing about a bulk endpoint sitting one path away. Seeing what a page returns to GPTBot remains the right first question, and it is now visibly not the only one. The same caution applies to the wider claim that an agent ready storefront performs better: that result was measured on pages and tasks, not on feeds.