Benchmarks / Methodology

How Ottom measures category benchmarks

An Ottom category benchmark is the aggregate of every store Ottom scanned in that category over the trailing 90 days. Each store contributes one observation, its most recent completed scan. The page publishes the middle of that distribution, never an individual store. This page states every rule that produces those figures, so a number you disagree with can be argued with rather than only doubted.

methodology bench-v5 · scoring score-v2 · taxonomy stall-v3

What is measured

Ottom runs an agent through the four stages of a purchase against a real storefront and scores each stage on what the agent actually managed. The overall score is the weighted average of those stages:

Buy carries the most weight because a store an agent can find but not buy from has failed the thing this measures. The buy stage is scored on how far the agent got, not on how many defects were found: a journey that did not reach the payment step can never score that stage as passing, however tidy the page it stopped on. How one store’s score is made, finding by finding, is on the scoring page.

A stage counts as passing at 80 out of 100. That is the same threshold that stamps PASS on an individual report, so a benchmark and a report never disagree about what passing means. The overall readiness score has no pass mark of its own. A store’s verdict comes from its worst stage, so a store can score well overall and still stall, which is why the score charts draw no line at 80.

Which stores are counted

• Every store Ottom has scanned, paying or not. A free scan counts the same as a monitored one. A category benchmark counts the stores in that category; the all stores benchmark counts every store, including the ones in no category.
• The category is read from the store itself, from the product types, product names and collection pages it publishes. Nothing a merchant tells us, and nothing from any private or app-granted data. A store whose catalogue does not clearly fall into one of the eight is counted in none of them.
• One observation per store, its most recent completed scan inside the window. A merchant who re-scans daily cannot weight their own category, and a store Ottom surveyed is measured again at most once every 60 days.
• Categories are a fixed list. A store whose category is not on that list is counted in no category benchmark, rather than creating a category of one. It is still counted in the all stores benchmark.
• A store’s findings are those of that same latest scan, from both of its runs: the retrieval and the browser journey are two runs of one scan, and reading only the last would drop half of them. A finding an earlier scan raised and a later one did not is no longer counted. Until bench-v5 findings were taken across the whole window, so a fixed problem, or one our agent had reported wrongly, stayed in the stall shares until the scan that found it aged out.

The all stores benchmark

Alongside the categories, Ottom publishes one benchmark that pools every store it has scanned: every category, and every store whose category could not be determined. It is at /benchmarks/all and it is the figure the benchmarks index opens with.

• The same rules as a category. The same window, the same minimum of 20 stores measured at every stage, the same two cohorts, the same floor under a stall, and the same methodology version on every figure. Adding it changed no rule, which is why the version did not move.
• Each store once. A store counts once in the pool and once in its own category, never twice in either.
• Not an average of the categories. Every store carries the same weight, so a category with more stores in it moves the pooled figure more than one with fewer. Read it as the typical store Ottom has scanned, not as the typical category.
• One scoring version at a time. Like a category, the pool scores only the stores measured under the version of the scoring rules that most of them were scored under. Stores on another version are left out of the scores, still count in the stall shares, and the page says how many.

The platform comparison

/benchmarks/platforms groups the stores in the all stores benchmark by the platform their storefront is built on, as Ottom detected it from markers the platform itself puts in the page, such as its own script paths and response headers.

• Named platforms only. Shopify, WooCommerce and BigCommerce can each appear. A storefront whose page carries no platform marker we recognise, a platform we fold under one label, and a store we could not read are not platforms, so none of them is ever shown. Those stores are still in the all stores benchmark.
• The same rules as a category. A platform is shown once at least 20 of its stores have been measured at every stage inside the window, all under one version of the scoring rules, with the same floor under a stall.
• Two platforms, or nothing. The comparison is published once at least 2 platforms meet that rule for the same window and the same version of the scoring rules. Below that the page publishes no figure and no store count for any platform.
• Confounded with how stores are chosen. The survey admits a store only if its catalogue can be read in one request, which in practice means Shopify and WooCommerce storefronts (see where the stores come from). So the platform decides whether a store is counted as well as how it performs, and a difference between platforms is not a verdict on either platform. The page says so above its figures.

The segment benchmark

/benchmarks/segments groups the stores in the all stores benchmark by what kind of store they are. Its figures need a free Ottom account, are not open data and are not covered by the CC BY licence on this page. The rules below are public. Rules version segments-v2.

• From the store’s latest scan, and nothing else. Every segment is read from what the storefront showed during the scan the benchmark describes: its own markup, its own catalogue feed or sitemap, and the response headers it sent. Ottom makes no extra request to decide a segment. A store last scanned before segments existed is placed only from what that scan already recorded: its currency, from the currency its own product questions were asked in; who ran the checkout its journey reached; and a bot manager, only when a challenge page met it. It joins the other segments when it is next scanned.
• Storefront currency. The ISO currency the store’s product markup states, then its catalogue, then what its home page states. Australian, Canadian and US dollars, euros and pounds can each appear.
• Price tier. The median price of the products we read from the store’s own catalogue, at least 5 of them, against the median of that figure across stores in the same category that price in the same currency. That peer median needs at least 10 stores, and prices are never compared across currencies. Luxury is 3 times the peer median or more, premium 1.5 times or more, mainstream anything below.
• Catalogue size. How many products the store lists, from its feed or its sitemap. Where we read only the first page of a feed, that is a lower bound, so a store is placed in the small band (under 50) only from a complete count, and in the large band (500 or more) from any count that reaches it.
• Checkout. Who runs the checkout page the agent reached: the store’s platform (recognised by the platform’s own checkout address, on whichever domain the store checks out on), a known third-party checkout service, or a checkout the store built itself. Read from the scan’s own journey, so a store whose agent never reached checkout is in no checkout segment, and every store in one has reached it. Read these figures with that in mind: they describe stores an agent got as far as checkout on, not every store.
• Express wallets. Whether the checkout the agent reached, or failing that its cart, offered an express wallet button: Shop Pay, Apple Pay, Google Pay, PayPal, Amazon Pay or Klarna. The agent looks and presses nothing. Only buttons count, not a row of payment logos. Wallets are offered by the browser as much as by the store (Apple Pay never appears in the browser our agent uses), so “no express wallet seen” means none was offered to an agent like ours, not that the store does not take one. A store whose agent never reached its cart is in neither group.
• Bot protection. Whether a bot management product answers in front of the store, named by its own headers, cookies or script hosts, or whether a challenge page met the scan. A content delivery network alone does not count: one fronts every Shopify store and says nothing about whether an agent is challenged.
• The same rules as a category. Each segment is shown only under the floors a category is held to, under one version of the scoring rules, with the same floor under a stall. An account does not lower either floor.

Can AI agents read these stores?

Beside the all stores benchmark, the benchmarks index publishes how often an AI agent is turned away, or finds nothing it can read, at the first step of a scan. It is counted over the same stores as the all stores benchmark, from each store’s latest scan in the window, and it reads findings only, never a score.

• Each figure has its own denominator, and the sentence names it. A rule is only counted over the stores it was actually checked on. robots.txt: every store scanned. Bot protection: the stores whose robots.txt let the agent in. No product feed and no llms.txt: the stores whose storefront the agent could open, meaning no bot protection stopped it and it found products. No schema.org Product markup and no shipping costs before checkout: the stores whose product pages the agent read.
• The same floors. Nothing is published under 20 stores, a figure whose own denominator is under 20 is withheld, and a share is withheld when fewer than 5 stores show it. Whole percentages only, and no store is named anywhere in the data.
• One taxonomy version. Only stores whose findings were written under the current stall taxonomy (stall-v3) are counted. Stores scanned under an older version are left out rather than mixed in, and a store with no findings is counted if it was scanned after the current version was in use.
• What is left out, and why. Missing prices, stock status and GTINs are only checked on product pages that carry structured data, and a scan does not record which pages did, so an honest denominator cannot be rebuilt. A missing sitemap is not shown yet. The rule behind it was corrected on 24 September 2026, after it reported stores whose sitemap sits on the www or bare domain, or behind a sitemap index, and the figure will be added once the stores in the window have been measured under the corrected rule. How an agent reached each catalogue is not published either: the survey chooses stores for a readable catalogue, so that mix describes our sampling, not online retail.

Two cohorts, and why the numbers differ

Not every scan reaches checkout. Bot protection, a robots.txt rule that excludes agents, or product pages that could not be read will all stop a scan at the compare stage. That store was measured, and what was found on it is real. What it does not have is a comparable overall score, because the buy stage carries 40% of the weight and an unmeasured stage counts as zero. Its score cannot exceed 60 however good the store is.

So each figure is computed over the cohort it can honestly describe:

• Score quartiles and stage medians come from the stores measured at every stage. Mixing in a score that cannot exceed 60 would make a category median a blend of two different scales with nothing on the page to say which is which.
• Stall prevalence is computed over every store scanned. The stalls that stop a scan early are the most important ones in the category, and counting them only among the stores they did not stop would report them at close to zero.

Every category page shows both cohort sizes, in that order. Before bench-v2 the two were one number and the score quartiles were computed over both, which understated every category by an amount that varied with how many of its stores had bot protection.

When the scoring changes

The scoring that turns what an agent found into a number is versioned, and it is currently score-v2. A change to it can move a store’s score with nothing about the store changing, so scores from two versions never share a median. When a category’s window holds stores scored under two versions, its scores come from the version most of its stores were scored under, and the rest are left out of the scores. They still count in the stall shares, like a store that stopped before checkout, and the category page says how many were left out.

So a category does not jump the night the scoring changes. It keeps its figures on the old version until more of its stores have been scored under the new one, then moves. Until bench-v4 a category holding two versions published nothing at all, which with a ninety day window meant ninety days of nothing after every change.

Where the stores come from

A category holds two kinds of store, and the difference changes what its figures mean. Some are merchants who came to Ottom and scanned their own storefront. Others Ottom scanned on its own initiative, because a benchmark nobody can fill is not a benchmark.

The surveyed stores are chosen by a rule, not by a list of names. A domain enters the survey if Ottom can read a catalogue of at least 10 products from it, reachable from the fixed egress our agents scan from, robots.txt permitting us. Being well known qualifies nothing.

That rule has a bias, and the bias is in what Ottom can read rather than in what merchants publish. Read it carefully: the test is what we can read, not what a store makes available. Ottom reads a whole catalogue in one request from three places, and three only. Shopify’s products endpoint, the WooCommerce Store API, and a feed a store advertises in its own markup or robots.txt. Every other storefront is read a product page at a time, and that path is capped at 3 pages per scan, so it cannot reach 10 however many products the shop actually sells.

So the threshold is doing something it does not look like it is doing. A storefront on Adobe Commerce, BigCommerce or a custom stack can publish a complete, correct, machine-readable catalogue and still fail to qualify, because we cannot collect enough of it in one pass to compare with the rest. Of the surveyed stores whose catalogue we have been able to read so far, 46 are Shopify and 2 are WooCommerce. That is a fact about our reach, and reporting it as a fact about those platforms would be the more flattering mistake.

The surveyed cohort is therefore not a random or representative sample of online retail, and a median computed over it should not be read as the truth about the category as a whole. It is stated here rather than left to be inferred, because a benchmark whose sampling is undisclosed is a number dressed as a measurement.

This section has been corrected twice. It first described the survey as drawn from a list of recognisable UK retailers, which is how it began; selecting by recognition produced one complete four-stage measurement in ten, because the retailers a person can name are the ones running bot protection and custom storefronts. It then said the rule excluded storefronts that serve no feed for anything to read, which was wrong about the cause: they are read, a page at a time, and it is the page budget that keeps them under the threshold. Both corrections are stated rather than quietly applied, because a sampling rule that changes without saying so is how a benchmark stops being checkable. The stores themselves are not published: a list of names would turn an aggregate study into a report card on individual shops, which is the thing this page exists to avoid.

When a category is published

A category is published only once at least 20 stores in it have been measured at every stage inside the window. Stores whose scan stopped short of checkout do not count toward it, however many there are. This is enforced twice: in the code that builds a snapshot, and as a constraint in the database that rejects the row. It is the rule most likely to be argued away under deadline.

Twenty, specifically, because:

• Nobody should be identifiable. A merchant who knows a few competitors use Ottom can back an individual score out of a small cohort.
• The number has to hold still. These pages refresh nightly. Below about twenty stores, a median jumps several points when one store re-scans, which is noise presented as a trend.
• It has to be reachable. A threshold of a hundred per category would leave every page empty long past the point where the data is defensible.

Below the threshold the category page publishes no figure at all, is not indexed, and does not appear in our sitemap or llms.txt. Advertising a URL with nothing behind it is how a placeholder becomes a claim.

How the figures are calculated

• Quartiles, not extremes. Lower quartile, median, upper quartile. The best and worst store in a category are each one identifiable merchant.
• Nearest-rank percentiles. Every published score is a score some store actually received, not a number interpolated between two of them.
• Whole percent. Shares are rounded to whole numbers; a share quoted to one decimal place leaks the size of the cohort it came from.
• Stalls are withheld below 5 stores, and the number withheld is shown, so the list on a page is visibly a floor rather than the whole truth.
• A stage nobody measured is absent, never zero. If most of a cohort never had a browser journey run, the page shows no buy-stage figure rather than a failure rate built from a handful of stores.

How often it refreshes

The pipeline runs nightly and each snapshot covers the trailing 90 days, aligned to UTC. If the pipeline stops, a page keeps serving its last snapshot for 3 days and then stops publishing, rather than serving an indefinitely old figure under a “refreshed nightly” claim. A category that falls back below the threshold returns to saying there is not enough data; its history is kept, not shown.

A surveyed store is measured again at most once every 60 days, so the benchmark stays current. A reading counts for 90 days, and the next one is taken before it leaves the window, so a surveyed store stays in the cohort and its reading is replaced rather than fixed on the day it was first surveyed. It is not continuous monitoring: no surveyed store is visited more often than that. A store that has asked us to stop, that blocks our agents in robots.txt, or that its own merchant has scanned in the last 60 days is not measured again by us. Until 23 September 2026 a surveyed store was scanned once and never again, which is why this paragraph used to say its observation could not be refreshed.

bench-v3 lengthened that window from thirty days to 90, on 20 September 2026. At thirty days, with every surveyed store scanned only once, the arithmetic was self-defeating: a category reached the threshold, then lost its oldest stores a month later with no way to replace them except by scanning new shops faster than the old ones expired. The cost is stated rather than hidden. A 90 day median describes storefronts as they were up to three months ago, so a merchant who fixed something in the second week still appears at their old score until the window carries them out.

What this cannot tell you

• It is a sample of stores that were scanned, not of the category. Stores arrive because someone ran a scan, and that is not a random draw. It says what Ottom has seen, and nothing wider.
• It is not a revenue figure. A stage score is a measurement of what an agent could do, not a measured drop-off rate. The ROI calculator turns one into the other and is explicit about which parts of that are assumptions.
• Agents change. Scores move when models and shopping protocols move, not only when stores do. Model versions are pinned per run and a change to them is a versioned event, so a shift can be attributed rather than guessed at.
• Your own scan is the number that matters. A category median does not tell you what to fix on Monday; your report does.

Disputing a figure

Every benchmark page shows three things beside its numbers: the cohort size, the window, and the methodology version. Quote those and any figure can be reproduced exactly. If the scan of your own store is what you dispute, your report shows what the agent saw at each step and can be re-run. That is the more useful argument, and the one we would rather have.

Changing any rule on this page changes numbers that have already been published, so it is treated as a versioned event: the methodology version is bumped, the change is stated here, and it is recorded in our decision log. Figures published under an older version keep their own stamp.

Report versions

Every published category report carries a version number, such as v1.2.3. The first number changes when the methodology on this page changes. The second changes when the scoring behind each store’s score changes. The third changes when a nightly refresh changes any figure on the report. A night that leaves every figure the same keeps the version it had.

Earlier versions stay readable at their own address, for example /benchmarks/apparel/v1.2.3, for 90 days after they were issued, with the same figures as JSON beside them. Quote the version with a figure and anyone can check it against the report it came from.

Common questions

Read next

• All category benchmarks

  Every category Ottom measures, and which have enough data to publish.

• What is agentic commerce?

  The definition behind every number these pages report.

• How our agents identify themselves

  The published user agent, and the static addresses every scan behind these numbers came from.

• The agentic readiness checklist

  What the stages above look like as a list of things to fix.

Run a scan on your own store