Stall library · Discover
robots.txt blocks AI shopping agents
A robots.txt block stops an AI shopping agent before it reads a single product, because the file tells well behaved agents which paths they may fetch and a broad Disallow rule covers the catalogue as well as the cart. An agent that respects robots.txt treats the rule as final, so the store is not compared badly against a rival. It is not compared at all.
Why the agent stops
Most shopping agents fetch /robots.txt first and cache the result for the session. A rule that names their user agent, or a wildcard rule with a broad Disallow, removes the catalogue from what the agent is willing to request. Nothing is rendered, nothing is parsed, and no error is raised: the agent simply has no products to reason about.
The reason this is easy to ship by accident is that the same file is where stores block training crawlers. Blocking a crawler that ingests your copy for model training and blocking an agent that is shopping for a named customer are different decisions, and one wildcard collapses them into the same rule.
How often we see this
Not stated here. A frequency means nothing without the cohort it was counted over, and this page is about one stall rather than one category. Where a category has enough stores measured at every stage, the share of them showing this stall is published on its benchmark page, with the methodology beside it.
See it for yourself
- Open https://yourstore.com/robots.txt in a browser.
- Read every User-agent block, not only the first one. A wildcard block near the bottom still applies.
- Look for a Disallow line that covers your product URL prefix, commonly /products/ or /collections/, or a bare Disallow: / under a wildcard.
- Check for named agent user agents. A rule naming a shopping agent is the clearest form of this stall.
What to change
- Narrow the Disallow rules to the paths that genuinely should not be crawled, which is usually /cart, /checkout, /account and internal search.
- Leave product, collection and policy paths fetchable. Those are the pages an agent needs to answer a shopper.
- If you want to block model training but allow shopping, separate the rules by user agent rather than using one wildcard for both.
- Reference your sitemap from robots.txt so an allowed agent can find the catalogue quickly.
What Ottom looks at
Ottom fetches robots.txt as the first request of every scan, using the published Ottom user agent. If the file is reachable and does not allow that agent, the scan stops there and reports only this finding, because every other conclusion would be guesswork about pages we were told not to read.
Questions
Do AI shopping agents actually obey robots.txt?
The identified ones generally do, and Ottom does. That is precisely why this stall is expensive: the agents that obey your robots.txt are the ones acting for a real customer, and they are the ones you lose. Traffic that ignores the file was never the traffic you were trying to keep.
Can I block AI training crawlers but not shopping agents?
Yes. They send different user agents. Write the Disallow rules against the named crawlers you want to exclude rather than under a wildcard, and check the file again after any theme or platform upgrade, because generated robots.txt files are frequently overwritten.
Will fixing robots.txt make my store agent ready?
It removes one blocking condition, and that is worth doing, but it is the first of four stages rather than the whole test. robots.txt decides whether an agent is allowed to look. Whether it can then read your prices, tell your sizes apart and get to a payment screen are three separate questions, and each is answered by attempting a purchase rather than by reading a file. A store that unblocks robots and stops there has made itself visible and not necessarily buyable.