Ottom

Checklist · Discover

Agentic readiness checklist: the discover stage

An AI agent can only consider products it can find. At the discover stage it reads what your site publishes about itself, and anything that turns it away or hides your catalogue ends the visit before your products are ever considered.

What the agent is trying to do here: Finding out that your product exists at all, and that it is allowed to look.

  1. robots.txt blocks AI shopping agents

    critical

    Narrow the Disallow rules to the paths that genuinely should not be crawled, which is usually /cart, /checkout, /account and internal search.

    • Leave product, collection and policy paths fetchable. Those are the pages an agent needs to answer a shopper.
    • If you want to block model training but allow shopping, separate the rules by user agent rather than using one wildcard for both.
    • Reference your sitemap from robots.txt so an allowed agent can find the catalogue quickly.
    See it on your own store
    1. Open https://yourstore.com/robots.txt in a browser.
    2. Read every User-agent block, not only the first one. A wildcard block near the bottom still applies.
    3. Look for a Disallow line that covers your product URL prefix, commonly /products/ or /collections/, or a bare Disallow: / under a wildcard.
    4. Check for named agent user agents. A rule naming a shopping agent is the clearest form of this stall.

    Why this stops an agent

  2. Bot protection blocks the agent before the storefront loads

    critical

    Allowlist identified agents by user agent, and by their egress addresses where the operator lists them, rather than trying to infer intent from behaviour.

    • Move the challenge to a risk based trigger on the checkout path instead of applying it to every product page request.
    • Where your provider supports verified or signed agent traffic, prefer that signal over a blanket category block.
    • Ottom publishes its own user agent at /agent, with its static egress addresses to follow, so you can allowlist a scan and measure your store rather than your bot protection.
    See it on your own store
    1. From a machine outside your office network, request a product page with a plain HTTP client rather than a browser, for example curl with a custom user agent.
    2. Read the status code and the first part of the body. An interstitial page, a 403, or a challenge script instead of your product markup is the wall.
    3. Repeat from a cloud server. Residential addresses are often scored more leniently than data centre ones, so an office test can pass while every agent fails.
    4. Check your bot management logs for the same timestamps and see which rule matched.

    Why this stops an agent

  3. No products found for an agent to read

    critical

    If this is not an online store, do nothing. The scan is describing the site accurately.

    • Publish a product feed, or make sure the one your platform serves is not blocked.
    • List product pages in the sitemap and reference the sitemap from robots.txt.
    • Link at least the main collections and a few products from the home page in plain HTML.
    See it on your own store
    1. Open your home page and look for a link to any product page in the HTML itself, not one that appears only after scrolling or clicking.
    2. Open /sitemap.xml and check whether any URL in it is a product page.
    3. On Shopify open /products.json; on WooCommerce open /wp-json/wc/store/v1/products. On anything else, check whether a product feed exists at all.

    Why this stops an agent

  4. Products only appear after JavaScript runs

    critical

    Send product links, names and prices in the HTML the server returns, on the home page and on every category page.

    • If the site is built with a framework such as Next.js or Nuxt, use its server rendering or static generation for category and product pages. The pages look the same; the difference is that the HTML arrives filled in.
    • List every product page in the sitemap.
    • Add schema.org Product markup to each product page, and answer a product that does not exist with a 404 status rather than a normal page.
    See it on your own store
    1. Open a category page, then view its source (not the inspector, which shows the page after scripts ran). Search the source for one of the product names you can see on the page.
    2. Open /sitemap.xml and check whether any URL in it is a product page rather than a category or information page.
    3. Turn JavaScript off in your browser and reload a category page. What is left is what a reader that does not run scripts gets.

    Why this stops an agent

  5. Product pages did not answer when the agent asked for them

    high

    Make every URL your catalogue states answer 200 with the product on it, for a request carrying no cookies and running no scripts.

    • Regenerate the feed and the sitemap whenever a product URL changes, so neither keeps naming a route the storefront no longer serves.
    • Where a redirect is unavoidable, point it at the product itself rather than at a collection page or the home page.
    • Check the rate limits and firewall rules in front of the storefront, and allow the published Ottom user agent through them.
    See it on your own store
    1. Open your product feed or your sitemap and copy three product URLs exactly as they are written there.
    2. Fetch each one in a private window with JavaScript turned off, or with curl, and check the status is 200.
    3. Follow any redirect to the end and confirm it lands on that product rather than on a collection or the home page.
    4. Request all three within a second or two, the way an agent would, and check the third still answers.

    Why this stops an agent

  6. No sitemap for agents to crawl

    high

    Publish /sitemap.xml covering product, collection and policy URLs. Most platforms generate this for you once it is enabled.

    • Reference the sitemap from robots.txt so an agent that reads one file finds the other.
    • Keep it current on publish rather than on a weekly schedule, so newly listed products are discoverable the day they go live.
    • Include your shipping and returns policy pages. Agents read policy terms as part of comparing you, not as legal boilerplate.
    See it on your own store
    1. Open https://yourstore.com/sitemap.xml and check it returns XML rather than a 404 or your homepage.
    2. Follow the child sitemaps and confirm product URLs are listed, not only pages and blog posts.
    3. Pick three products that are not linked from the homepage or main navigation and check each appears.
    4. Open robots.txt and confirm it carries a Sitemap: line pointing at the file.

    Why this stops an agent

  7. No machine-readable product feed

    high

    Expose a product feed. On Shopify /products.json is served automatically unless it has been disabled by an app or a theme rule.

    • Include price, currency, availability, variant options and an identifier such as a GTIN or MPN for every variant.
    • Regenerate the feed when a product changes rather than on a long schedule.
    • Make sure robots.txt does not disallow the feed path. Blocking it is a common side effect of blocking scrapers.
    See it on your own store
    1. If you are on Shopify, open https://yourstore.com/products.json and check it returns product objects rather than an error.
    2. Confirm the response includes price, availability and a variants array for each product.
    3. Count the products returned against your real catalogue size. Feeds are commonly paginated and a first page is not a catalogue.
    4. On other platforms, look for a merchant feed endpoint or a generated feed file and check the same fields.

    Why this stops an agent

  8. Product pages have no schema.org markup

    high

    Add schema.org Product JSON-LD to every product page, including offers.price, offers.priceCurrency and offers.availability.

    • Include an identifier: a GTIN, EAN, UPC or MPN, so an agent can match your product to the one a shopper named elsewhere.
    • Emit the markup in the server response rather than injecting it with a script after load.
    • Describe variants with hasVariant entries, each carrying its own offer, so a size or colour choice is readable without running the page.
    See it on your own store
    1. Open a product page and view source, or fetch the URL with curl so you see the HTML as delivered rather than after scripts run.
    2. Search the source for "application/ld+json".
    3. If a block exists, check it contains an object with "@type": "Product" and an "offers" object carrying "price", "priceCurrency" and "availability".
    4. Repeat on a product added by a different app or template. Markup is often present on one template and absent on another.

    Why this stops an agent

  9. No llms.txt to orient agents

    low

    Publish /llms.txt with a one line site description followed by your key URLs, each with a short description.

    • Regenerate it when you publish, so it never advertises a page that has moved.
    • Do not put anything in it you would not put on a public page. It is a public file.
    • Treat it as a finishing touch after the feed, the structured data and the checkout path are right.
    See it on your own store
    1. Open https://yourstore.com/llms.txt and check it returns plain text rather than a 404.
    2. Confirm it lists your key pages: the catalogue, the main collections, shipping, returns and contact.
    3. Check each URL in it resolves. A file advertising pages that 404 is worse than no file.

    Why this stops an agent

Scan your store to find out which of these 9 it actually hits, rather than working through all of them.