Documentation

How an Ottom score is calculated

An Ottom score is a weighted average of four stages an agent must get through to buy from a store: discover, evaluate, compare and buy. This page gives the weights, what each finding costs, how the buy stage is walked, and the two rules that keep noise out of alerts: a noise floor on score changes, and two signals before a Critical.

The four stages

• Discover

  Whether an agent is let in at all, by robots.txt and bot protection, and what your store publishes for machines to read: sitemaps, product feeds, structured data and llms.txt. Scored from the findings it raises. The stall library lists 10 findings for this stage.

• Evaluate

  Shopper requests are matched against your product data alone: price, stock, variants, identifiers and policies, as an agent can read them without running the page. Rules decide what they can, and a language model judges only the matches rules cannot settle. Scored from the findings it raises. The stall library lists 7 findings for this stage.

• Compare

  Checks the facts a comparison rests on: that your feed and your storefront agree on price, that shipping cost can be known before checkout, and that the catalogue has categories to compare within. Scored from the findings it raises. The stall library lists 3 findings for this stage.

• Buy

  A browser walks the purchase path on your live store, one step at a time, and records how far it gets. Steps can be taken by AI agents running on Claude models from Anthropic. Before a stall is reported, the store can be walked again, and the further of the two walks is the result. It stops at the payment step and never pays. The score comes from how far the walk got, and its findings say where and why it stopped. The stall library lists 20 findings for this stage.

What a finding costs

A data stage starts at 100 and loses points for each finding. At 80 or above it is PASS, from 50 to 79 it is SLOW, and below 50 it is STALLED. It never goes below zero.

How the buy stage is scored

The buy stage is PASS, at 100, only when the walk reaches the payment step. Anything short of that is STALLED, scored by how far it got, up to 45: a store that reached checkout scores above one that stopped at the size picker, and neither passes. Two cases are neither. When every product the scan picked was sold out or not yet on sale, there was nothing to buy, so the buy stage is not scored on that scan and an Info finding, which costs nothing, says why. When your store does not allow checkout below a minimum order and the one item the agent added is under it, the buy stage is not scored either, and a Medium finding says so: the agent never adds more to reach a minimum.

The walk takes these steps, in order:

1. Open the product
2. Choose an option
3. Add to cart
4. Open the cart
5. Start checkout
6. Reach the payment step

How the walk works

Our agent reads each page as it is and chooses from the controls on it, one action at a time, the way a careful shopper would. Steps can be taken by AI agents running on Claude models from Anthropic. Our code, not a model, decides whether a step is done, and every run records how it was walked. It never attempts a CAPTCHA, never works around the store’s own checkout error, and stops at the payment step.

Some steps can only be got past by reasoning about the page. When the walk gets past one that way, the report adds a Low finding, “A simpler agent stopped at a step an AI agent got past”, naming the step and what got the walk past it. The step can be done, but a less capable agent would stop there.

What that finding means and how to fix the step is in the stall library. What our agent may and may not do, whatever a page tells it, is on our agent page.

What the walk works around

A walk that passes may still have had to work around something in its way. When it closes a popup or an offer that covers a button it has to press, answers a cookie banner that covers the page, or has to press checkout a second time because the first press did nothing, the report says so in a finding of its own, even when the walk reaches the payment step. Each is reported at most once a scan, on the buy stage, which is scored by how far the walk got, so none of them changes a score or a verdict, and benchmarks do not count them as stalls. They are there because a less capable agent would have stopped where ours found a way past. A cookie banner that leaves the page usable is never reported.

• A popup covers a button on the way to checkout

  Medium

• A cookie banner covers the page until it is answered

  Low

• The first press of checkout does nothing

  Medium

Before a stall is published

Before a stall is published, our agent can walk your store again, from the product page, within the same journey. If it gets further, its walk is the result: the stop was ours, not your store’s. If it stops at the same step, having tried that step itself, the stall is published and the finding records two concurring signals: the walk that stopped, and the second walk. If the two walks give different reasons for that stop, the finding gives the reason your store’s own response shows, such as an add to cart it refused, rather than one we could only infer. Otherwise, for example when it stopped somewhere else or could not run, we hold the result and walk the store again about ten minutes later. A stall that still cannot be confirmed is then published with one signal, and a Critical one is shown as Critical, unconfirmed. A stop at the product page itself, such as a page that does not open, is published as found, because a second walk would open it the same way.

The score itself

The score is the weighted average of all four stages. A scheduled check reads your pages and feeds and does not attempt checkout, so when a scan did not measure a stage, the score uses the most recent measurement of it from the 7 days before, and the report says when that was. Checkout counts for as long as your plan waits between agent journeys, never less than 7 days: 30 days on Coverage. If a stage was not measured in that time, there is no score: the report says how many stages were checked rather than scoring the missing one as zero or as a pass. The exception is a scan that could not read a single product page, because the store blocked the agent or had nothing it could open: the stages after Discover were not reached, they score zero, and the report says so.

The noise floor

Two scans of an unchanged store do not always score the same, because stores and networks vary from minute to minute. So an alert reports a score change only when it is larger than 5 points and larger than twice the run to run variation measured for that kind of scan. Smaller changes still go into your score history. They are not announced as a regression or an improvement.

Two signals before a Critical

A Critical finding is alerted on, and counted as open in the weekly digest, only when two signals agree: two agent journeys, or one journey backed by a retrieval check. The second journey can come from another scan of the same store, run within 7 days of the first and finding the same thing. A retrieval check alone does not confirm a Critical, because it cannot see the purchase. Three Criticals are the exception, because only the retrieval check can see them: robots.txt blocking AI agents, bot protection stopping the agent before the store loads, and no products for an agent to read. When an agent is stopped there no journey runs, so each of these is confirmed when the retrieval check of a different scan of the same store, run within 7 days, finds the same thing. Two checks in the same scan never confirm each other. Until a second signal agrees, the report shows the finding as Critical, unconfirmed.

Versions

Scores are calculated by scoring version score-v2 and findings are named by stall-v3 of the stall taxonomy, which has 40 codes. Every run records the version that scored it, and a change to the weights, the costs or a severity is released as a new version rather than made silently. Every code is published at /taxonomy.json.

Questions

How scores are combined across stores is on the benchmark methodology. How often each plan scans is on the plans page.