Release gates: failing a build when AI agents stall

An Ottom release gate fails a CI build when a release makes your store harder for AI agents to buy from. Your pipeline starts a scan of your store or its staging origin through the Ottom API, waits for it, and reads one field: gate.result. This page gives the rules, the two API calls and a recipe for GitHub Actions, GitLab CI and CircleCI.

How the gate decides

Each run is compared with the last completed scan of your production store. A run on your staging origin is compared with production too, so a release is judged against what shoppers have now.
For the overall score and for each stage you choose the largest drop a release may cause, in points. The gate fails when the score falls by more than that.
A drop only counts once it is larger than the noise floor: twice the measured variation of the AI agents that scored the run, and never less than 5 points. A threshold of 0 therefore means any real drop, not any wobble.
You can also fail the gate when a stage stalls, or when any stage stalls. A stall needs no earlier scan to compare with.
When our scoring changes version, scores either side of the change are not compared, so drops are skipped for that run and stalls are still checked.

The noise floor is the same one alerts use. How it is measured is on how a score is calculated.

What gate.result can be

pass
Every threshold that could be checked held. The recipes pass the build.
fail
At least one threshold tripped. gate.tripped says which. The recipes fail the build.
pending
The scan is still running, its buy attempt included. Poll again.
skip
The gate did not run: no thresholds apply to this store, or none could be checked this run. gate.reason says which. The recipes print it and pass.
error
The scan could not finish, so there was nothing to judge. The recipes print it and pass. Change that line if you would rather fail.

Setting it up

1. Mint a CI token
On your workspace page, an owner or admin mints a CI token for the store. It is shown once. A token reaches one store and that store’s staging origin, nothing else.
2. Store it as a secret
Save the token in your CI system as a secret named OTTOM_TOKEN, and the address to scan as OTTOM_STORE_URL: your store, or its staging origin.
3. Add the script and the job
Commit the script below as ci/ottom-gate.sh and add the job for your CI system after the step that deploys.

Who can do what in a workspace is on the workspaces page.

The two API calls

Both take the token as Authorization: Bearer. The base address is https://ottom.io/api/ci/runs.

POST /api/ci/runs
Starts a scan. The body is JSON with url, the address to scan. Answers 202 with runId. Each scan counts against the store’s daily limit, and a store at its limit answers 429 with code rate_limited.
GET /api/ci/runs/{runId}
Reads the scan: its status, score, stages, findings your plan can see, cost, and gate. gate.result is the field a pipeline acts on, gate.tripped lists the thresholds that failed, and gate.receiptUrl links the receipt for a ticket.

The daily limit and what happens at it are on the limits page.

The script

Every recipe below runs this. It starts a scan, polls every 30 seconds for up to 30 minutes, prints the result and the receipt link, and exits 1 when the gate fails.

#!/bin/sh
# Ottom release gate. Needs curl, jq, OTTOM_TOKEN and OTTOM_STORE_URL.
set -eu
API="https://ottom.io/api/ci/runs"
AUTH="Authorization: Bearer $OTTOM_TOKEN"

START=$(curl -sS --fail-with-body -X POST "$API" -H "$AUTH" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg url "$OTTOM_STORE_URL" '{url: $url}')") || { echo "$START"; exit 1; }
RUN_ID=$(echo "$START" | jq -r '.runId')
echo "Ottom scan $RUN_ID started"

RESULT=pending
i=0
while [ "$RESULT" = pending ] && [ "$i" -lt 60 ]; do
  sleep 30
  i=$((i + 1))
  REPORT=$(curl -sS --fail-with-body "$API/$RUN_ID" -H "$AUTH") || { echo "$REPORT"; exit 1; }
  RESULT=$(echo "$REPORT" | jq -r '.gate.result // "pending"')
done

echo "$REPORT" | jq -r '"Gate: \(.gate.result). \(.gate.reason)", "Receipt: \(.gate.receiptUrl)"'
case "$RESULT" in
  fail) exit 1 ;;
  pending) echo "No gate result after 30 minutes."; exit 1 ;;
  *) exit 0 ;;
esac

GitHub Actions

In .github/workflows/ottom-gate.yml. Runs when your staging deploy workflow finishes. Add OTTOM_TOKEN under Settings, Secrets and variables, Actions. Change the workflow name to yours.

name: Ottom release gate
on:
  workflow_run:
    workflows: ["Deploy staging"]
    types: [completed]
jobs:
  gate:
    if: github.event.workflow_run.conclusion == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Ottom release gate
        env:
          OTTOM_TOKEN: ${{ secrets.OTTOM_TOKEN }}
          OTTOM_STORE_URL: https://staging.example.com
        run: sh ci/ottom-gate.sh

GitLab CI

In .gitlab-ci.yml. A job after your deploy job. Add OTTOM_TOKEN as a masked variable under Settings, CI/CD, Variables.

ottom_gate:
  stage: test
  needs: [deploy_staging]
  image: alpine:3.20
  variables:
    OTTOM_STORE_URL: https://staging.example.com
  before_script:
    - apk add --no-cache curl jq
  script:
    - sh ci/ottom-gate.sh

CircleCI

In .circleci/config.yml. A job that requires your deploy job. Add OTTOM_TOKEN as a project environment variable or in a context. The cimg/base image already has curl and jq.

version: 2.1
jobs:
  ottom-gate:
    docker:
      - image: cimg/base:stable
    environment:
      OTTOM_STORE_URL: https://staging.example.com
    steps:
      - checkout
      - run:
          name: Ottom release gate
          command: sh ci/ottom-gate.sh
workflows:
  release:
    jobs:
      - deploy-staging
      - ottom-gate:
          requires: [deploy-staging]

Questions

Does a release gate cost extra scans?

Each gate run is one scan of the store or its staging origin, and it counts against that store’s daily limit like any other. A store at its limit answers 429 and the script fails the step, so a busy pipeline should gate once per release rather than once per commit.

Why did a drop of a few points not fail the gate?

AI agents do not score a store identically every time. A drop has to be larger than the noise floor, at least 5 points, before the gate treats it as a change your release made. How the floor is measured is on the scoring page.

What is the difference between gate and verdict in the response?

verdict compares the run with the previous run of the same store and fails on new findings at a severity you pick. gate compares it with your production store and fails only on the thresholds you set. The recipes read gate. Both are in every response.

Can the gate scan a store that is not public yet?

Yes, if it is a staging origin set up for the store. A token for the store can scan that origin, and the run is judged against the production store.

Your stores and their gates are on your dashboard.