Stall library · Evaluate
The store refused the product page to the agent
A refused product page is a product the store will not show an AI shopping agent. The agent opens the product in a browser and the store answers 403, 429 or 503 instead of the page, with no challenge to solve and no message to read. Asked again later, it refuses again. The product is listed, but nothing about it can be read: not its price, not its stock, not its options.
Why the agent stops
An agent opens a product the way a shopper does, in a browser with no account and no history on the store. Rules in front of a storefront often treat that visitor differently from a returning customer: a firewall rule that refuses an unfamiliar browser, a rate limit set for a person rather than an assistant comparing several products, a country block, or a rule that answers 503 on the origin’s behalf.
Nothing on the page says which rule fired, so the agent has nothing to work around. It does not wait and retry for minutes the way a determined person might. It moves on to a store that answered.
A refusal is not a bot wall. A bot wall shows a challenge an agent cannot solve, and names itself in its markup or its headers. A refused page names nothing, which is why we look twice before reporting it: a single refusal can be a moment of load, and only a refusal that comes back the same later is reported as the store’s.
How often we see this
Not stated here. A frequency means nothing without the cohort it was counted over, and this page is about one stall rather than one category. Where a category has enough stores measured at every stage, the share of them showing this stall is published on its benchmark page, with the methodology beside it.
See it for yourself
- Copy the product URL the report names.
- Open it in a private window with no extensions, from a network your store has not seen you on, such as a phone off wifi.
- Fetch the same URL with curl and the published Ottom user agent, and note the status it returns.
- Request it five or six times within a minute, the way an agent comparing products would, and check the later answers are still 200.
What to change
- Find the rule that answered. Your firewall, CDN or bot manager logs the request by URL and time, and the report gives both.
- Allow the published Ottom user agent, and verified agent traffic in general, through that rule.
- Set rate limits for product pages high enough for an assistant comparing several products in a minute.
- Where the origin answers 503 under load, serve a cached product page rather than an error.
What Ottom looks at
Ottom opens a product the catalogue scan picked, in a real browser, and records the status it gets back. A 403, 429 or 503 with no challenge page and no challenge header is held and the product is opened again at least ten minutes later, with nothing else from us in between. This finding is raised only when the second visit is refused the same way. A refusal written by our own network is never reported.
Questions
Why does the product page open for me but not for the agent?
Most rules that refuse a page look at who is asking: a new browser with no cookies, a network your store has not seen, several requests close together. You arrive as a customer the store knows. The agent arrives as a stranger, which is exactly how a shopper’s assistant arrives too.
Could this be a fault in the scan rather than in my store?
That is why we check twice. A refused page is held, not reported, and opened again at least ten minutes later. We report it only when the second answer matches the first, and never when the refusal came from our own network.