Learn / Strategy
AI agent traffic vs bot traffic
Non-human traffic is not one category. Search crawlers, training collectors, shopping agents and abusive scrapers have different purposes and different consequences, and the tooling most stores use groups them together. That grouping is the mechanism by which a store intending to protect its content removes itself from a sales channel.
Four categories, four consequences
- Search crawlers. Make you findable in search. Almost every store permits these.
- Training collectors. Use your content to train models. Refusing costs you nothing in sales.
- Shopping agents. Read your catalogue to answer a shopper and potentially buy. Refusing removes you from a channel.
- Abusive scrapers. Take your data at your expense with nothing offered back. Everyone wants these gone.
The second and third are the pair that get conflated, because the guidance most rules were written from treated all non-search AI traffic as one thing.
How to tell them apart
Two signals, used together. Identity: a declared user agent, corroborated against address ranges the operator publishes, since the string alone is self declared and trivially copied. Behaviour: a shopping agent fetches narrowly, taking your robots.txt, your feed and a handful of products, while a crawler sweeps.
See how to identify AI agent traffic in server logs.
The cost argument, honestly
Bandwidth is the usual objection and it is usually small here, because the fetch pattern is narrow. A current product feed reduces it further, since a catalogue read in one request is cheaper for you than the same information reconstructed page by page.
That is worth noticing: the thing that makes you easier to buy from also makes you cheaper to read.
What to do about it
Decide the categories separately and write the decision down, so the next person does not collapse them again. See should you block or allow AI shopping agents.
Questions
How do I tell a shopping agent from a scraper?
By behaviour and by verifiable identity. A shopping agent declares itself, comes from published address ranges, respects robots rules and fetches narrowly. A scraper hides, ignores robots and sweeps broadly.
Is agent traffic a cost I should worry about?
Rarely at current volumes, and the shape helps: an agent answering a question fetches a handful of pages rather than surveying your catalogue. A feed reduces it further by answering in one request.
Can I allow shopping agents without allowing training crawlers?
Yes, and it is a reasonable position. They are distinct categories with distinct tokens, and separating them is exactly the decision most stores have not made.