Learn / Fixes
How to configure robots.txt for AI shopping agents
Search crawling, training data collection and agentic shopping are three different uses with three different consequences, and most robots.txt rules about AI were written with only the second in mind. The result is stores that intended to keep their content out of a training set and also removed themselves from assistant driven purchases, without ever making that decision.
Search crawling, training data collection and agentic shopping are three different uses of your store with three different consequences. Most robots.txt rules about AI were written thinking about the second one only, and quietly decided the third.
This is the most common cause of robots blocks agent, and it is almost never deliberate.
The three decisions
- Search. Being crawled so you appear in results. Nearly every store wants this and nearly every store permits it.
- Training. Your content being used to train a model. Legitimate to refuse, and refusing costs you nothing in sales.
- Shopping. An agent reading your catalogue to answer a shopper’s question and potentially buy. Refusing this removes you from a sales channel.
The second and third are the ones that get conflated, because the guidance available when most of these rules were written treated all non-search AI traffic as one thing.
Read your own file first
Before changing anything, open your robots.txt and write down every rule mentioning an AI or agent token, along with what you believe it was for. Most stores find at least one rule nobody currently on the team wrote.
Get the current tokens from each operator’s own published documentation rather than from a list in an article. Republished lists start decaying immediately, and a rule written against a token that no longer exists does nothing while looking like a policy.
Write rules you will not have to maintain
A file that names each assistant individually needs editing every time a new one appears, and until somebody edits it the new assistant is treated by whatever your default rule says. If your default is a disallow, every new entrant is blocked and nothing tells you.
Prefer the broadest expression your rules allow, and treat the specific tokens as exceptions rather than as the policy itself.
Remember the layer above
Your robots.txt is a request. Your CDN is enforcement, and it can block an agent that has read and respected your rules perfectly. If the two disagree, the CDN wins and your robots.txt is describing a policy you do not actually have.
See how to let honest agents through bot protection, and check both before concluding anything about your visibility.
Questions
If I block training crawlers, am I blocking shopping agents?
Frequently, yes, because the tokens were listed together in whatever guide was followed. They are different agents with different purposes. Read your own file and check which is which rather than assuming the rule does what it was written to do.
Is robots.txt even enforced?
It is a convention that well behaved operators follow and badly behaved ones ignore. That asymmetry is worth being clear eyed about: a disallow mostly stops the agents you might have wanted, since those are the ones that read it.
Should I allow by name or by category?
Naming tokens by hand means every new assistant arrives blocked by default and you find out slowly. Decide the category and express it as broadly as your rules allow, then revisit when the landscape changes rather than when something breaks.