Stall library · Discover
No sitemap for AI agents to crawl
A missing sitemap forces an AI shopping agent to discover your catalogue by following links, which means the products it finds are the ones your navigation happens to surface rather than the ones a shopper asked for. A sitemap is the only cheap, complete statement a store makes about which URLs exist, and without it an agent works from a partial view of the catalogue and does not know that it is partial.
Why the agent stops
Link following is expensive for an agent and it is bounded. Every hop costs a fetch and a parse, and agents work under a budget, so deep or paginated catalogues get truncated. The products that fall outside the crawl are not ranked lower. They are absent.
A sitemap changes the shape of the problem from search to lookup. With one, an agent can confirm in a single request that a product exists and where it lives, which is the difference between answering a specific shopper question and offering the nearest thing on the homepage.
How often we see this
Not stated here. A frequency means nothing without the cohort it was counted over, and this page is about one stall rather than one category. Where a category has enough stores measured at every stage, the share of them showing this stall is published on its benchmark page, with the methodology beside it.
See it for yourself
- Open https://yourstore.com/sitemap.xml and check it returns XML rather than a 404 or your homepage.
- Follow the child sitemaps and confirm product URLs are listed, not only pages and blog posts.
- Pick three products that are not linked from the homepage or main navigation and check each appears.
- Open robots.txt and confirm it carries a Sitemap: line pointing at the file.
What to change
- Publish /sitemap.xml covering product, collection and policy URLs. Most platforms generate this for you once it is enabled.
- Reference the sitemap from robots.txt so an agent that reads one file finds the other.
- Keep it current on publish rather than on a weekly schedule, so newly listed products are discoverable the day they go live.
- Include your shipping and returns policy pages. Agents read policy terms as part of comparing you, not as legal boilerplate.
What Ottom looks at
Ottom looks for a sitemap at the conventional locations and via the Sitemap directive in robots.txt, and records whether product and collection URLs were found in it. Sitemap coverage is also what later lets the scan say something about your category structure.
Questions
Do I need a sitemap if my products rank well in search?
Ranking well means a search engine has crawled you over months. An agent answering a shopper right now has no such history and no index of its own to fall back on, so it relies on what your site states about itself in the moment.
Should I check the sitemap my platform generates?
Yes, once, and then rarely. Generated sitemaps are commonly capped at a product count, split across child sitemaps where only the index is advertised, or excluded by a robots rule somebody added later for an unrelated reason. None of those failures produce a visible error: the sitemap still exists, still validates and still returns 200, it simply describes a fraction of your catalogue. Open it, count the URLs, and compare that against how many products you sell.