Learn / Start here
Why discovery comes first for AI shopping agents
Discovery is the first stage of an AI agent’s purchase attempt: the agent has to be allowed to fetch your store, find the products in it and read what each one is. A store that fails discovery is never evaluated, compared or bought from, so no improvement later in the journey can make up for it.
That makes discovery the cheapest stage to get right and the most expensive to get wrong. Most of the fixes are a setting or a file. The cost of missing one is every sale an agent would have sent you.
What discovery means to an agent
A person finds a store through search, an ad or a recommendation, then browses. An agent does something narrower and more literal. It needs three things, in this order.
- Permission. Your robots rules, your CDN and your bot protection decide whether the request is answered at all. See how to configure robots.txt for AI shopping agents and how to let honest agents through bot protection.
- A list of products. The agent has to learn which products exist without guessing at URLs. A sitemap lists your pages. A product feed lists your products and what they contain. See how to publish a product feed for AI agents.
- Readable pages. Once a product page arrives, its name, price, stock and variants have to be readable in the HTML your server sends, ideally as structured data. See how to add product schema for AI agents.
Why a discovery failure is total
Every later stage is a judgement about something the agent has already found. Evaluate asks whether the product fits the request. Compare asks whether it beats the alternatives. Buy asks whether the checkout can be operated. Each of those can go partly wrong and still leave you in the running. A discovery failure cannot, because there is no product to judge.
It is also silent. A shopper who cannot find their size leaves a trace in your analytics. An agent that is refused at the CDN, or that reads a page with no products on it, leaves one short request at most, and the shopper it was working for is shown another store. Nothing tells you it happened.
That is why four of the discovery stalls in the stall library are rated Critical: robots blocks agent, bot wall, no catalogue and catalogue requires JavaScript. Any one of them ends the attempt before evaluation starts.
Ranking in search is not the same thing
It is tempting to assume that a store which ranks well is discoverable. The two overlap less than they appear to. Search crawlers are often allowed by name while agent traffic falls under a general bot rule. Google renders JavaScript before it indexes a page, and many agents read only the HTML the server sends. And a search result is a link for a person to follow, while an agent needs the price, stock and variants in a form it can read with no person in the loop.
Is my store visible to AI shopping agents? walks through those differences one at a time.
The common ways stores fail discovery
A rule written for scrapers
The most common cause is a rule nobody remembers adding: a robots.txt disallow for a broad group of user agents, or a CDN setting that challenges anything that is not a browser. It was written to stop scrapers and it stops shoppers’ agents too. Cloudflare settings for AI shopping agents covers the most common case.
Products that only exist after JavaScript runs
A storefront built as a JavaScript application can send a nearly empty page and fill it in inside the browser. A person never notices. An agent that reads the served HTML finds a shell with no products in it and concludes, correctly from what it was given, that there is nothing to buy.
A catalogue with no map
Without a sitemap or a feed, an agent has to find products by following links from your home page. Products reachable only through filters, infinite scroll or on-site search are never found. How to structure collections for agent discovery shows the fix.
Pages with nothing machine-readable on them
An agent that reaches a product page with no structured data has to work out the price and stock from the layout. When it gets that wrong it quotes the wrong figure, or moves on to a store whose page says it plainly.
Discovery is a set of decisions you already control
Every one of these is a setting, a file or a template on your own site. None of them needs an assistant’s cooperation or a partnership. That is unusual in agentic commerce, where much of what decides an outcome sits inside someone else’s product. Work through them in this order.
- Check that your robots rules and your CDN answer agent requests rather than challenging them.
- Publish a sitemap, and a product feed if your platform can produce one. Then keep it current: see how to keep a product feed fresh.
- Make sure product names, prices and stock appear in the HTML your server sends, with Product schema on every product page.
- Add an llms.txt that points agents at your catalogue and your policies. It is a small signal, not a fix for the steps above. See how to write an llms.txt for an online store.
How to check
Reading your own settings tells you what you intended. It does not tell you what an agent receives, because the CDN, the theme and the apps in between all have a say. The reliable test is to make the requests an agent makes, from outside your network, with no saved session, and look at what comes back.
Agent view shows one product page the way an agent receives it. An Ottom scan runs the whole journey, discover, evaluate, compare and buy, and tells you the stage where it stalled and what to forward to your developer.
Questions
If an agent cannot read my store, will it tell the shopper?
Usually not in those words. It answers with the stores it could read. From the shopper’s side your store was simply not among the options, which looks exactly the same as not being a good match.
Is an llms.txt file enough to make my store discoverable?
No. An llms.txt points an agent at the pages worth reading, which helps once the agent is allowed in and the pages are readable. It cannot undo a robots rule, a bot challenge or a catalogue that only appears after JavaScript runs. Treat it as the last step, not the first.
Why are discovery stalls the most severe ones in an Ottom report?
Because nothing after them is reached. A store with a clumsy checkout can still be found, chosen and nearly bought from. A store an agent cannot read is never chosen at all, and a lenient score for it would describe a store no agent can buy from as nearly fine.
How often should I check discovery?
After any change to your theme, your apps, your CDN or your bot protection. Those are where discovery breaks without anyone deciding that it should, and none of them announces the change in your analytics.