An agent-ready storefront passed 89.3 percent of tasks, the baseline 49.3

What the experiment actually ran

The design is the reason the result is worth reporting at all, so it is worth stating before the figures. Two versions of one e-commerce platform were built. The paper calls them Website A, the baseline, and Website B, the agent-ready variant. They shared the same structure, products, pages, pricing, inventory, ratings and checkout procedures. The only difference between them was the set of agent-readiness features listed in the paper's Table 1. Holding the catalogue and the workflow constant is what turns the comparison into something other than an opinion about two different shops.

The agents were driven by browser-use, an open source agentic web framework, which the paper names and links. Three models were evaluated: GPT-4.1, Gemini 2.5 Flash and Grok 4 Fast. Temperature was set to 0 and top p to 1. Each session was capped at thirty steps and three errors, so a run that wandered was cut off rather than allowed to grind. Between trials the local server was restarted, and a temporary browser profile was created at the start of each run and deleted at the end, which removes the obvious contamination path where one run's cookies or cache help the next.

Each of the five tasks was performed ten times per model on each website version. Three models times five tasks times ten repetitions times two versions is 300 independent runs, and that arithmetic is worth doing yourself because it fixes the denominator for every percentage that follows: 150 runs per website version, 50 runs per model per version, 30 runs per task per version.

Scoring was done by two annotators reading the terminal logs and the model's JSON output. A PASS meant all primary task requirements were fully met, and counted as both a strict success and a functional success. A PARTIAL meant the main objective was achieved but a non-critical or moderate issue remained, and counted as a functional success only. A FAIL meant the objective was not achieved or an important instruction was violated. That three-way split is the most useful thing in the paper's method, because a two-way pass or fail would have hidden the finding described below, where the baseline agents usually arrived at roughly the right answer and could not be relied on to arrive at the complete one.

Two of the paper's own framing choices deserve flagging up front. The tasks are shopping tasks, described as covering actions, information retrieval and decision-making, with Task 5 designed as a lower-complexity policy retrieval task to test whether agent-ready features also matter for routine information access. And the agents here are browsing agents in the sense this site has used elsewhere, clients that drive a browser through a task on a live site rather than crawlers that fetch and archive. When that class of client meets ordinary bot defences the outcome is a separate question, and one that a controlled study on the authors' own server does not touch: for that, six of seven browsing agents completed their tasks with both of Cloudflare's free AI bot controls turned on, while a different measurement found headless Chromium soft blocked on 15.2 percent of the top 10,000 sites.

Where the 40 point gap actually sits

The headline is 134 of 150 against 74 of 150, strict success rates of 89.3 and 49.3 percent. The paper reports a chi-square test of the association between website type and outcome at 60.79 with N of 300 and p below 0.001, and a Cramer's V of 0.45, which it describes as a moderate effect. Every model improved, and the improvement was not uniform: GPT-4.1 moved from 32 to 86 percent, Gemini 2.5 Flash from 52 to 88, Grok 4 Fast from 64 to 94. The model that started worst gained most, which is the pattern you would expect if the page was supplying something the weaker extractor could not reconstruct on its own.

Now the number that changes the story. Functional success, which counts PARTIAL as well as PASS, was 78 percent on the baseline against 91.3 percent on the agent-ready version. A 13 point gap, not a 40 point one. The baseline agents were usually getting to a broadly correct answer. What they could not do reliably was extract complete information, supply sufficient evidence, or hold to every constraint at once. PARTIAL outcomes fell from 43 to 3 between the two versions, and that single pair of numbers is the finding: the agent-ready design did not mostly turn failures into successes, it turned near-misses into clean ones.

Per task, the effect concentrates. The paper reports Fisher's exact tests comparing PASS against non-PASS for each task. Task 2 went from 23.3 to 100 percent, a gain of 76.7 points at p below 0.001. Task 3 went from 16.7 to 93.3 percent, 76.6 points, also below 0.001. Task 1 went from 73.3 to 96.7 percent, 23.4 points at p of 0.026. Then it stops. Task 4, described as requiring several constraints satisfied simultaneously, improved by 16.7 points at p of 0.267. Task 5, the simple policy retrieval task, improved by 6.7 points at p of 0.761. Neither reached significance, and the paper says so rather than rounding them into the headline.

Three of five tasks carrying the result is not a weakness to hide, it is the shape of the finding, and reporting the aggregate without it would overstate what was shown. Thirty runs per task per version is a small cell, and small cells are exactly where a difference of a couple of runs decides whether a p value clears 0.05. This site has argued the same point about its own domain when asking how many prompts an AI visibility measurement actually needs, and it is the same discipline that leads it to withhold a grade rather than publish a confident wrong one.

Efficiency moved with success. Average steps per run fell from 9.31 to 6.49, a 30.4 percent reduction, and prompt token consumption fell for every model: 18.72 percent for Gemini 2.5 Flash, 37.43 percent for GPT-4.1 and 40.47 percent for Grok 4 Fast. The paper treats step count as a secondary indicator of execution effort rather than a success criterion, which is the right weight to give it. The practical reading is that an agent that cannot find a field goes looking for it, and looking costs steps and tokens.

Nothing was hidden from the agent, and it still failed half the tasks

This is the part most likely to be misread, so it is worth being slow about. The baseline site is described as holding its product data embedded in JavaScript. The agent-ready site is described as also exposing that data via JSON files and JSON-LD. A reader who works on crawler readability will recognise the first description immediately and reach for the familiar conclusion: the data was invisible, the agent could not see it, adding it to the HTML fixed that. On this experiment, that conclusion is wrong.

The agents were running browser-use, which drives a real browser. The baseline's JavaScript executed. Whatever it wrote into the page was on the screen, in the accessibility tree and in the DOM the agent read. The paper's own error analysis says the same thing from the other side: baseline agents frequently identified the correct page or product and then failed to extract complete information or draw a fully supported conclusion. You cannot identify the correct product on a page whose product data you never received. Access was not the constraint. Comprehension under a deadline was.

That distinction is the whole reason this site treats the two layers separately. A client that does not execute JavaScript is in an entirely different position: it receives the served HTML and nothing else, which is why Common Crawl archived 2.14 billion pages in July without running JavaScript and why what the server returns and what a browser assembles are two different documents. A site can fail that layer completely and never reach the question this paper is asking. It can also pass that layer and still be the baseline site here, serving a browsing agent everything and getting a half-complete answer back.

There is a third position, and it is the one most people are actually in. Not every AI client that arrives at a site is a browser. Some are, some are plain HTTP fetchers, and one logged sample of AI coding agents found that seven of nine arrived without a product name in the user agent at all. Clients of that kind execute nothing. For them the baseline site's product data is genuinely absent, and the fix the paper describes would be the difference between data and no data rather than between two presentations of the same data. The same page can therefore fail for two unrelated reasons depending on who asks, which is why prose parity is measured as its own signal here and why text inside a declarative shadow root reaches a browser and not an extractor.

So the honest summary of this experiment is narrower than the abstract makes it sound, and more interesting. It is not evidence that hiding data in JavaScript costs you agent traffic, because it did not test a client that could not read JavaScript. It is evidence that on one prototype, for agents that could read everything, presentation still decided whether half the tasks completed.

What the agent-ready version added

Table 1 of the paper lists four design aspects and what changed in each. It is short enough to read in full and specific enough to act on, which is more than most framework papers manage.

Product data exposure. The baseline embedded product data in JavaScript; the agent-ready version also exposed it via JSON files and JSON-LD. Note the word also. The JSON-LD was additive rather than a replacement, so the experiment never tested a version with the JavaScript removed. That vocabulary is public: schema.org defines the Product type, and Google's product structured data documentation states that the markup is what lets a search result show price, availability, review ratings and shipping information. This is ordinary structured data, read by a consumer it was not originally written for.

Semantic labels. The baseline had some controls carrying basic labels or attributes; the agent-ready version put aria-label, data attributes and explicit identifiers on more elements. MDN documents aria-label as defining a string value that names an element, for the case where the default accessible name is missing or does not accurately describe what the element contains. That is an accessibility mechanism first and an agent-legibility mechanism second. Google's own tooling already treats the two as connected: Chrome's Lighthouse carries an agentic browsing category whose accessibility group covers names and labels on interactive elements, which this site read when covering the fact that the same audit checks for a machine-readable summary file Google Search says it ignores. What that audit checks for, this paper measured the effect of.

Action controls. The baseline's cart and product actions relied on visible button text and handlers; the agent-ready version added explicit data attributes carrying product IDs, names, stock and availability. This is the one item on the list with no crawler analogue at all. A crawler never clicks anything, so a stable identifier on a button does nothing for it. If you are prioritising work, this is the item that buys agent reliability and nothing else.

Decision reliability signals. The baseline showed reviews, stock text and basic policy information on the page; the agent-ready version added evidence files and pages, temporal fields, and clearer product and use case information. Temporal fields are the interesting half. An agent asked to respect a constraint needs to know whether the thing it is reading is current, and a page that never says when a price or a stock level was set leaves that to inference.

Three of the four are things a non-executing reader also benefits from, which is the practical overlap between this work and crawler readability. But the paper offers no ablation, and says so: it recommends future work isolate the contribution of individual features. Nothing here supports attributing the 40 points to JSON-LD, or to labels, or to any single change. That matters because structured data is the field where the gap between markup present and markup correct is widest, which is why this site measures structured data twice, once in the served HTML and once after rendering, and why counting the published vocabulary showed only sixteen schema.org types reaching ten million domains.

What the experiment does not show

The authors are direct about this and the paper is better for it. Their limitations section states that the study is a controlled proof of concept rather than a full validation of the framework, and that the results should not be generalised to all domains, websites or agent systems. They call for evaluation on real-world websites across broader domains, a wider range of agent systems, and ablation studies isolating individual features. Take that at face value, because the alternative reading, in which a 40 point gap on a prototype becomes a number to quote about your own shop, is how a finding turns into folklore.

Several specific limits follow from the design rather than from any failing. There is one site. Not one sector, not a sample of sites: a single prototype, built by the authors, in two versions. Both versions were therefore authored by the people measuring the difference, and the size of the gap depends on how the baseline was written. The paper describes the baseline as a human-oriented design, and its features are recognisable ones, but nothing establishes that it represents a typical storefront, and the paper does not claim it does.

The scope is e-commerce, and the tasks are shopping tasks. Whether the same design changes move the needle for documentation, a booking flow or a support site is untested here. Three models were evaluated at temperature 0 and top p 1, and no other model or configuration was tested. Two annotators scored the runs, and no inter-annotator agreement figure appears in the version read on 5 August 2026. The paper is a preprint on arXiv and carries no statement of peer review or of a publication venue.

The most important limit is about what the outcome variable is. This study measures task completion by an agent on a site it has been pointed at. It says nothing about whether an agent chooses your site, whether a model cites you, or whether any of this moves referral traffic. Agent task success and AI visibility are different quantities with different measurement problems, and a survey of the field found the same confusion at scale, with 45 GEO studies placing crawling second among stages and few of them observing it directly. Treating a task-completion result as a generative engine optimization result is precisely the substitution this blog exists to refuse.

And the closing declaration, because this site holds itself to it: Lantad ran none of this. No agent suite, no browser-use harness, no prototype storefront, no 300 runs. Everything above is read from a preprint. The nearest thing this site has published from its own measurements is what a crawler actually meets on a real storefront, which captured pages and read what came back, and which measured a different thing on different sites for a different reason.

What to check on your own site

None of the following requires you to accept the paper's headline number. Each item is checkable in a few minutes, and each one either holds or does not on your own pages.

Start below the layer this paper worked at, because it is the one that can fail absolutely. Request a product URL the way a crawler does, without a browser, and read what comes back. If the price, the stock state and the product name are not in that response, then every client that does not execute JavaScript has nothing, and no amount of agent-readiness above that point helps them. Fetching the page as a named crawler and reading the result answers that directly, and the methodology page sets out what the scan does and does not do before you rely on it.

Then check whether the structured data is present and complete rather than merely present. A Product block with a name and no offers, price or availability tells an agent the item exists and nothing about whether it can be bought, which is the exact shape of the incompleteness the paper's PARTIAL outcomes describe. On a hosted platform this is usually a theme-level question rather than a page-level one, which is why the stack guides exist and why the Shopify guide addresses the template rather than the product.

Check your interactive controls for stable identifiers. This is the item with no crawler benefit and it is worth doing anyway if agent traffic matters to you: a button whose only identity is its visible text is one copy change away from breaking an automated flow. The accessibility work most teams have already done covers a good part of this, since an accessible name is a machine-readable name.

Check whether your pages say when they were last true. Stock levels, prices and policies read very differently to a client that can see a date on them. This is cheap and it is skipped almost everywhere.

Finally, be clear with yourself about which question you are answering. Whether an agent can complete a purchase on your site is a usability question with a testing answer: point an agent at it and watch. Whether an AI crawler can read your pages at all is a measurement question with a fetching answer, and it is the one this product exists for, described in full in the crawlability study. They are related, they are not the same, and a tool that claims to answer the second should not be quoted as if it had answered the first.