---
title: "How an Ottom score is calculated"
source: "https://ottom.io/docs/scoring"
summary: "An Ottom score is a weighted average of four stages an agent must get through to buy from a store: discover, evaluate, compare and buy. This page gives the weights, what each finding costs, how the buy stage is walked, and the two rules that keep noise out of alerts: a noise floor on score changes, and two signals before a Critical."
---

# How an Ottom score is calculated

An Ottom score is a weighted average of four stages an agent must get through to buy from a store: discover, evaluate, compare and buy. This page gives the weights, what each finding costs, how the buy stage is walked, and the two rules that keep noise out of alerts: a noise floor on score changes, and two signals before a Critical.

## The four stages

| Stage | Weight | What it asks |
| --- | --- | --- |
| Discover | 20% | Can an agent find the store and read its catalogue at all? |
| Evaluate | 25% | Does the store’s own data answer what the shopper asked for? |
| Compare | 15% | Can an agent weigh this store against another on price and shipping? |
| Buy | 40% | Can an agent get a chosen product all the way to the payment step? |

### Discover

Whether an agent is let in at all, by robots.txt and bot protection, and what your store publishes for machines to read: sitemaps, product feeds, structured data and llms.txt. Scored from the findings it raises. The stall library lists 10 findings for this stage.

### Evaluate

Shopper requests are matched against your product data alone: price, stock, variants, identifiers and policies, as an agent can read them without running the page. Rules decide what they can, and a language model judges only the matches rules cannot settle. Scored from the findings it raises. The stall library lists 7 findings for this stage.

### Compare

Checks the facts a comparison rests on: that your feed and your storefront agree on price, that shipping cost can be known before checkout, and that the catalogue has categories to compare within. Scored from the findings it raises. The stall library lists 3 findings for this stage.

### Buy

A browser walks the purchase path on your live store, one step at a time, and records how far it gets. Steps can be taken by AI agents running on Claude models from Anthropic. Before a stall is reported, the store can be walked again, and the further of the two walks is the result. It stops at the payment step and never pays. The score comes from how far the walk got, and its findings say where and why it stopped. The stall library lists 20 findings for this stage.

## What a finding costs

A data stage starts at 100 and loses points for each finding. At 80 or above it is PASS, from 50 to 79 it is SLOW, and below 50 it is STALLED. It never goes below zero.

| Severity | Points off the stage |
| --- | --- |
| Critical | 60 |
| High | 25 |
| Medium | 12 |
| Low | 5 |
| Info | 0 |

## How the buy stage is scored

The buy stage is PASS, at 100, only when the walk reaches the payment step. Anything short of that is STALLED, scored by how far it got, up to 45: a store that reached checkout scores above one that stopped at the size picker, and neither passes. Two cases are neither. When every product the scan picked was sold out or not yet on sale, there was nothing to buy, so the buy stage is not scored on that scan and an Info finding, which costs nothing, says why. When your store does not allow checkout below a minimum order and the one item the agent added is under it, the buy stage is not scored either, and a Medium finding says so: the agent never adds more to reach a minimum.

1. Open the product
2. Choose an option
3. Add to cart
4. Open the cart
5. Start checkout
6. Reach the payment step

## How the walk works

Our agent reads each page as it is and chooses from the controls on it, one action at a time, the way a careful shopper would. Steps can be taken by AI agents running on Claude models from Anthropic. Our code, not a model, decides whether a step is done, and every run records how it was walked. It never attempts a CAPTCHA, never works around the store’s own checkout error, and stops at the payment step.

Some steps can only be got past by reasoning about the page. When the walk gets past one that way, the report adds a Low finding, “A simpler agent stopped at a step an AI agent got past”, naming the step and what got the walk past it. The step can be done, but a less capable agent would stop there.

What that finding means and how to fix the step: https://ottom.io/learn/stalls/agent-needed-to-recover. What our agent may and may not do: https://ottom.io/agent.

## What the walk works around

A walk that passes may still have had to work around something in its way. When it closes a popup or an offer that covers a button it has to press, answers a cookie banner that covers the page, or has to press checkout a second time because the first press did nothing, the report says so in a finding of its own, even when the walk reaches the payment step. Each is reported at most once a scan, on the buy stage, which is scored by how far the walk got, so none of them changes a score or a verdict, and benchmarks do not count them as stalls. They are there because a less capable agent would have stopped where ours found a way past. A cookie banner that leaves the page usable is never reported.

- **A popup covers a button on the way to checkout** Medium. https://ottom.io/learn/stalls/popup-covers-control
- **A cookie banner covers the page until it is answered** Low. https://ottom.io/learn/stalls/consent-overlay-covers-page
- **The first press of checkout does nothing** Medium. https://ottom.io/learn/stalls/checkout-press-dropped

## Before a stall is published

Before a stall is published, our agent can walk your store again, from the product page, within the same journey. If it gets further, its walk is the result: the stop was ours, not your store’s. If it stops at the same step, having tried that step itself, the stall is published and the finding records two concurring signals: the walk that stopped, and the second walk. If the two walks give different reasons for that stop, the finding gives the reason your store’s own response shows, such as an add to cart it refused, rather than one we could only infer. Otherwise, for example when it stopped somewhere else or could not run, we hold the result and walk the store again about ten minutes later. A stall that still cannot be confirmed is then published with one signal, and a Critical one is shown as Critical, unconfirmed. A stop at the product page itself, such as a page that does not open, is published as found, because a second walk would open it the same way.

## The score itself

The score is the weighted average of all four stages. A scheduled check reads your pages and feeds and does not attempt checkout, so when a scan did not measure a stage, the score uses the most recent measurement of it from the 7 days before, and the report says when that was. Checkout counts for as long as your plan waits between agent journeys, never less than 7 days: 30 days on Coverage. If a stage was not measured in that time, there is no score: the report says how many stages were checked rather than scoring the missing one as zero or as a pass. The exception is a scan that could not read a single product page, because the store blocked the agent or had nothing it could open: the stages after Discover were not reached, they score zero, and the report says so.

## The noise floor

Two scans of an unchanged store do not always score the same, because stores and networks vary from minute to minute. So an alert reports a score change only when it is larger than 5 points and larger than twice the run to run variation measured for that kind of scan. Smaller changes still go into your score history. They are not announced as a regression or an improvement.

## Two signals before a Critical

A Critical finding is alerted on, and counted as open in the weekly digest, only when two signals agree: two agent journeys, or one journey backed by a retrieval check. The second journey can come from another scan of the same store, run within 7 days of the first and finding the same thing. A retrieval check alone does not confirm a Critical, because it cannot see the purchase. Three Criticals are the exception, because only the retrieval check can see them: robots.txt blocking AI agents, bot protection stopping the agent before the store loads, and no products for an agent to read. When an agent is stopped there no journey runs, so each of these is confirmed when the retrieval check of a different scan of the same store, run within 7 days, finds the same thing. Two checks in the same scan never confirm each other. Until a second signal agrees, the report shows the finding as Critical, unconfirmed.

## Versions

Scores are calculated by scoring version score-v2 and findings are named by stall-v3 of the stall taxonomy, which has 40 codes. Every run records the version that scored it, and a change to the weights, the costs or a severity is released as a new version rather than made silently. Every code is published at https://ottom.io/taxonomy.json.

## Questions

### Why does the buy stage count for the most?

Because a store an agent can find but cannot buy from has failed the thing that matters. Buy is 40% of the score, and the three stages before it share the rest.

### I fixed something. Why was there no alert?

Either the next scan has not run yet, or the change was inside the noise floor. A move of 5 points or less is recorded in your score history without an alert, so clearing a single low finding will usually not announce itself.

### Why is a Critical finding in my report but not in my alert?

The report shows what the scan found. An alert about a Critical waits for a second signal to agree with the first, so one unlucky journey cannot page you. Until one does, the report marks the finding Critical, unconfirmed. The second signal can come from your next scan, if it runs within 7 days and finds the same thing. If the agent could not get into your store at all, because of robots.txt, bot protection or no products to read, no journey runs, and the retrieval check of that next scan is the second signal.

### The walk got past a step. Why is there a finding?

Because not every agent that shops for a customer reasons about the page. Some follow the patterns most stores use and stop when a page does something different. The finding is Low, a walk that goes on to reach the payment step still passes, and fixing the step lets the simpler agents through too.

### My store passed. Why does the report still list findings?

Because the walk can pass and still have had to work around something: a popup over the checkout button, a cookie banner over the page, or a checkout button that needed a second press. Each is reported so you can remove it before it stops an agent less capable than ours. None of them changes your score or your verdict.

### Can I compare a score from last month with one from today?

Yes, within one scoring version. Alerts never compare a score across a version change, so a jump caused by a new version is not reported as your store getting better or worse.