A New Benchmark for Agentic Commerce: Are Merchants Ready for AI Shopping?

Title

A New Benchmark for Agentic Commerce: Are Merchants Ready for AI Shopping?

Meta Description

A paper proposes a merchant-side Agentic Commerce readiness benchmark built around three questions: access, decision sufficiency, and evidence. In tests of four large merchants, none could confirm that a buyer mandate was fulfilled; in a census of 555 sites, 18.9% of homepages were unreadable to declarative agents; among 330 evaluated merchants, only 35% met the rankability threshold, and total cost was the single biggest shortfall.

Article Content

When AI agents browse and place orders on behalf of humans, most attention goes to how smart the agent is. But at the other end of the transaction—whether a merchant's website lets agents in, whether its terms are written precisely enough, and whether it leaves behind traceable evidence—almost no one is measuring.

A paper titled Get In, Decide, Prove: A Merchant-Side Readiness Benchmark for Agentic Commerce proposes a merchant-side evaluation benchmark and publishes a pilot round of results. Its core purpose is not to rank merchants, but to validate whether this measurement methodology can actually run end to end.

Three Questions: Get In, Decide, Prove

The benchmark is built around three actions: In (access), Decide (decision), and Prove (evidence). Mapped to the merchant side, these correspond to whether the website allows agents to crawl and purchase, whether the published product terms are precise enough to support a concrete purchase decision, and whether evidence is left behind that can prove "what both parties agreed to."

The most critical measurement metric is called decision sufficiency. Its approach is to take a concrete buyer mandate and check it item by item—total spend, cancellation or return terms, service dates, product model identifiers—judging each item as determinate, indeterminate, or unknown based on the evidence the merchant itself publishes.

The significance of this design is that AI shopping failures are often not because the agent isn't capable enough, but because the merchant page simply doesn't provide the facts needed to decide.

Buyer mandate is the key concept for understanding the entire benchmark: it refers to a structured instruction a user grants to an AI agent, explicitly specifying hard constraints such as total budget, acceptable return or cancellation conditions, required service dates, and designated product models. Every choice the agent makes on the user's behalf while shopping must be checked back against this mandate to determine whether it exceeds authority. The reason decision sufficiency is measured by "checking against the mandate item by item" is precisely that the mandate itself is a decidable checklist: each item either has a determinate answer found in the evidence the merchant publishes, or is marked indeterminate or unknown. This design turns the fuzzy question of "is the page information sufficient" into a reproducible quantitative question of "what is the coverage rate across N constraints."

Declared bot and declared headless browser are two types of agent access identities that websites can recognize: the former actively declares itself as an automated program through User-Agent or similar signals, while the latter exposes automation traces through headless browser runtime characteristics but may not actively declare itself. The importance of distinguishing these two identities is that merchants' blocking strategies often follow two separate rule sets—one for "honestly self-reported bots" and one for "automation disguised as human"—so the same agent may shift from being rejected to being allowed through just by changing identity. Reporting access results by identity is the only way to expose this strategic difference, rather than giving a misleading binary conclusion of "crawlable / not crawlable."

Real-World Tests of Four Large Merchants: None Could Confirm Mandate Fulfillment

The paper conducted a full evaluation of four large merchants, and the results were not encouraging. Three directly rejected declared bots and declared headless browser configurations on their quote pages; and no single buyer mandate could be confirmed as "satisfied," because each merchant left at least one total-spend item or refund term in an indeterminate state.

In other words, just because a website can be accessed does not mean its terms can be understood; just because terms can be read does not mean they are sufficient to support an executable purchase.

The benchmark also distinguishes three access identities: declared bot, declared headless browser, and browser with a human-operated interface. Reporting access permissions separately, rather than lumping everything into a single "can it crawl" measure, captures the most realistic point of friction in agentic commerce—many merchants use two different logics for blocking bots versus allowing mixed human-machine traffic.

In addition, every observed term is mapped to the RAILS admissibility classification—that is, whether this piece of evidence could be admitted if a dispute arose in the future. This upgrades "did the product page state it" to "if stated, can it be used."

The RAILS admissibility classification is concerned not with whether a term was "written down," but with whether the term "can be used as evidence" in a dispute scenario. The same return policy, if placed in an HTML fragment that can be changed at any time, with no version snapshot, no explicit effective date, and no binding to merchant identity, would be very difficult to recognize in arbitration or litigation as "the content agreed upon by both parties." By mapping every observed term to this classification, the benchmark is in effect measuring the evidentiary strength of the terms merchants display, not just information density. For the agent side, this means that even if the page text is successfully scraped, it may still be unable to assert rights when a transaction fails due to lack of admissibility; for merchants, this suggests that beyond "writing it clearly," they also need to "keep it preserved and recognizable." This dimension corresponds directly to Prove in In/Decide/Prove, and it is also the core distinction between this benchmark and ordinary product-page crawl-quality assessments.

Census of 555 Sites: Nearly One-Fifth of Homepages Are Unreadable to Agents

The paper conducted an access census of 555 merchant sites publicly listed as clients by two payment processors, and found that 105 of them (18.9%) had homepages unreadable to both declarative configurations mentioned above.

This is an easily overlooked proportion. It means that even as agents' browsing and purchasing capabilities continue to improve, a substantial share of merchants still block them at the entrance. For merchants hoping to capture agent traffic, this is a signal they can act on directly; for those building agents, this is a real-world constraint that must be incorporated into failure handling.

Rankability Is Determined Not by Score, but by Evidence Coverage

In another evaluation of 330 merchants, the paper used deterministic rule-based scoring with no human raters. The results produced a counterintuitive conclusion: what determines who is eligible to be ranked is not the score, but evidence coverage.

• Only 115 merchants (35%) met the rankability threshold.
• The median score of rankable merchants was between 89 and 90.
• Among the four pillars, material terms was notably lower than the others, because only 18 merchants had a determinate "total cost."

This dataset shows that a high score does not mean complete information—many merchants perform similarly on the tested items, and what truly distinguishes them is "whether they have clearly written down the key facts." And ambiguity in total cost is a systemic weakness: fees beyond the listed price (shipping, taxes, service charges) are scattered across different pages, and agents cannot assemble a determinate answer.

Evidence coverage refers to the proportion of a merchant's total measurement items in the benchmark that can be judged as "determinate." The reason it determines rankability more than score does is that scores typically apply weights to convert outcomes at different levels into a relative value; even if most items are indeterminate or unknown, as long as the few items that are clearly measured score well, the total score can still look good. Coverage, by contrast, does no conversion—it directly exposes "how many facts were never measured at all." When coverage falls below the threshold, it means too much information is missing from the sample, and any score ranking is built on large blank spaces and lacks comparability—this is precisely why only 35% of the 330 merchants evaluated met the ranking threshold. Understanding this makes it clear why the sparsity of the "determinate total cost" item systematically drags down the coverage of the entire merchant group: it is the hardest constraint in a shopping decision, and without it, no amount of completeness in other terms can compensate.

What This Benchmark Can Do and Cannot Do

The paper's authors take a notably restrained stance: the pilot proved that these measurements can be executed, but it did not validate their validity, nor did it use them to rank merchants. The weighting scheme and the predicted clearing band still need to be checked against actual transaction outcomes before they can be confirmed.

This self-limitation matters. In the early stage of agentic commerce, an easy bias is to treat an uncalibrated set of scores as a conclusion. This work instead focuses on reproducible measurement methods—how to define "determinate," how to distinguish access identities, how to map terms to admissibility in dispute scenarios—and these have more long-term value than a ranking list.

For merchants, actionable conclusions have already emerged: let declared agents in, write total cost and refund terms as determinate values, and preserve traceable evidence. These three things do not need to wait for benchmark validation to be completed.

One-line Summary

A paper proposes a merchant-side agentic commerce readiness benchmark, and pilot tests show systematic gaps in access, terms, and evidence across most merchants.

Paragraph Summary

A paper proposes a merchant-side benchmark for evaluating agentic commerce, structured around three questions: In (can agents access the site), Decide (are the terms sufficient to support a purchase decision), and Prove (is admissible evidence left behind). Its core metric, decision sufficiency, judges each item in a buyer mandate as determinate, indeterminate, or unknown. Pilot results are not encouraging: three of four large merchants rejected declared bots and headless browsers on quote pages, and no mandate could be confirmed as satisfied; in a census of 555 sites, 18.9% of homepages were unreadable to the two declarative configurations; among 330 deterministic evaluations, only 35% met the rankability threshold, and only 18 merchants had a determinate total cost. The authors stress that the pilot only validates that the measurement can be executed, not its validity, and does not rank merchants.