Stall library · Discover
Bot protection blocks the agent before the storefront loads
A bot wall is a challenge served by bot management or a WAF before the storefront renders, so an AI shopping agent receives an interstitial rather than a product page. Unlike robots.txt, this layer does not read a file and obey it: it scores the request on network reputation, TLS and browser fingerprint, and a permissive robots.txt has no effect on the outcome.
Why the agent stops
Bot management assigns a score to the request before your theme is involved. An agent driving a headless browser from a data centre address looks, on every one of those signals, like the scraping traffic the product was bought to stop. The challenge that follows expects a human interaction the agent cannot perform.
The loss is invisible in analytics, and that is the property that makes this stall dangerous. A request that is challenged before the page loads produces no session, no pageview and no add to cart, so it never appears in your conversion rate. It appears as an absence.
How often we see this
Not stated here. A frequency means nothing without the cohort it was counted over, and this page is about one stall rather than one category. Where a category has enough stores measured at every stage, the share of them showing this stall is published on its benchmark page, with the methodology beside it.
See it for yourself
- From a machine outside your office network, request a product page with a plain HTTP client rather than a browser, for example curl with a custom user agent.
- Read the status code and the first part of the body. An interstitial page, a 403, or a challenge script instead of your product markup is the wall.
- Repeat from a cloud server. Residential addresses are often scored more leniently than data centre ones, so an office test can pass while every agent fails.
- Check your bot management logs for the same timestamps and see which rule matched.
What to change
- Allowlist identified agents by user agent, and by their egress addresses where the operator lists them, rather than trying to infer intent from behaviour.
- Move the challenge to a risk based trigger on the checkout path instead of applying it to every product page request.
- Where your provider supports verified or signed agent traffic, prefer that signal over a blanket category block.
- Ottom publishes its own user agent at /agent, with its static egress addresses to follow, so you can allowlist a scan and measure your store rather than your bot protection.
What Ottom looks at
Ottom flags a bot wall when a storefront request returns a challenge or block response rather than the page, and records the URL and the evidence. When the wall stops everything, the scan deliberately suppresses downstream findings: telling you your prices are missing when the truth is that we never saw a product page would be a false report with a real fix attached. The buy attempt raises it too, when every product it opens is answered by a Cloudflare challenge, or is sent to the same other site moments after loading by a page that declares a bot blocker set to turn automated browsers away.
Questions
Why does my store block agents when I never chose to?
Bot management ships with category defaults, and an automated browser from a data centre falls into a category most defaults treat as unwanted. The setting that blocks a shopping agent is usually one nobody at the store selected deliberately.
How do I tell a bot wall apart from a slow site?
Look at what came back rather than how long it took. A wall returns a challenge document or a block status quickly and consistently, from any address you test, while a slow site returns your own markup eventually.
Does allowlisting an agent expose me to scrapers?
Allowlisting a named user agent from a published, static set of addresses is narrow by construction. A scraper can copy a user agent string, but it cannot originate from addresses it does not control, which is why the address list is the part that matters.