Which AI agents can shop your store?
An AI robots.txt checker reads the robots.txt file at the root of a store and reports, for each AI agent and crawler by name, whether it may read the home page, a product page, the cart and the checkout. Enter your store address to see yours.
How to read the result
Each row is one agent, and each column is one part of the buy path. A product page that says BLOCKED is the row to fix first: an agent that cannot read your products cannot recommend them.
A store with no robots.txt is open to every AI agent: RFC 9309 says a robots.txt that answers 404 has no rules in it. #
A robots.txt that answers with a server error closes the whole store: RFC 9309 tells crawlers to read a 5xx answer as a rule to fetch nothing. #
A group that names an agent replaces your rules for all crawlers, for that agent. It does not add to them, so an agent named in its own group ignores the User-agent: * group entirely. #
To block training crawlers and keep shopping agents, name the crawlers rather than writing the rule under the wildcard. How to block training crawlers but not shopping agents covers it, and the robots.txt stall shows what a block costs a store.
The agents we check
Every token below is the one its operator publishes, linked to the operator's own page about it.
| User agent | Operator | What it does |
|---|---|---|
| ChatGPT-User | OpenAI | Visits a page when a shopper asks |
| OAI-SearchBot | OpenAI | Builds the index an assistant searches |
| GPTBot | OpenAI | Collects pages to train models |
| Claude-User | Anthropic | Visits a page when a shopper asks |
| Claude-SearchBot | Anthropic | Builds the index an assistant searches |
| ClaudeBot | Anthropic | Collects pages to train models |
| Perplexity-User | Perplexity | Visits a page when a shopper asks |
| PerplexityBot | Perplexity | Builds the index an assistant searches |
| Google-Extended | Collects pages to train models | |
| Applebot | Apple | Builds the index an assistant searches |
| Applebot-Extended | Apple | Collects pages to train models |
| Amzn-User | Amazon | Visits a page when a shopper asks |
| Amzn-SearchBot | Amazon | Builds the index an assistant searches |
| Amazonbot | Amazon | Collects pages to train models |
| Meta-ExternalFetcher | Meta | Visits a page when a shopper asks |
| Meta-WebIndexer | Meta | Builds the index an assistant searches |
| Meta-ExternalAgent | Meta | Collects pages to train models |
| CCBot | Common Crawl | Collects pages to train models |
| OttomAgent | Ottom | Tests a purchase like a shopping agent |
What this does not tell you
robots.txt decides whether an agent may read a page. It says nothing about whether the agent can use the page once it is there: a price that only appears after a script runs, a size picker that needs a mouse, a checkout that insists on an account. Agent view shows what one page looks like to an agent, and a free scan walks a real purchase as far as the payment step.
Frequently asked questions
What is robots.txt?
A plain text file at the root of a site, at /robots.txt, that tells crawlers which paths they may read. Each group names one or more user agents and lists the paths they may and may not fetch. The rules are defined by RFC 9309, and every agent in the table above is published by its operator with the token a store writes to address it.
Should I block AI crawlers?
Block the ones you have a reason to block, and know which kind each one is. Training crawlers collect pages to train models, and blocking them is a policy choice that does not stop a shopper reaching you. Search and user agents are how an assistant finds your products and reads them when a shopper asks. Blocking those takes your store out of the answer.
Does blocking GPTBot stop ChatGPT from recommending my store?
Not by itself. OpenAI publishes GPTBot, OAI-SearchBot and ChatGPT-User as separate tokens: the first collects training data, the second builds the index ChatGPT search reads, and the third fetches a page when a person asks. Anthropic and Perplexity split theirs the same way. Each one follows its own group in robots.txt, which is why this tool checks every token on its own.
Why are my cart and checkout blocked?
Most likely on purpose. Shopify’s default robots.txt disallows /cart and /checkout for every crawler, which keeps those pages out of search results, so most Shopify stores show BLOCKED in those two columns. A product page that is blocked is the row to worry about: an agent that cannot read it cannot tell anyone what you sell.
Is robots.txt enforced?
No. It is a request, not a lock, and each operator says on its own page how its agents treat it. Each token in the table links to that page. robots.txt is still the one place where a store states its rules to every agent at once, which is why getting it right matters.
What do you send to my store?
One request for /robots.txt and nothing else, from our published user agent and a fixed set of static IP addresses listed on our agent page. The result is kept for half an hour, so reloading or sharing it does not send us back, and we cap how often any one store is checked however many people ask.
See how our own agent identifies itself, and how to opt out, on our agent page.
Allowed in is not the same as able to buy
A scan walks your store the way an AI shopper would and tells you where it stalls on the way to checkout. One free scan per store, per email address, every 30 days.