Learn
How to identify AI agent traffic in server logs
Server logs are the only place an AI agent’s visit is recorded, because the request is served whether or not any JavaScript runs. Identify agents by user agent token against each vendor’s published list, corroborate with the address the request came from, and treat any hardcoded list as something that decays, because new agents appear and tokens change without notice.
Server logs are the only place an AI agent’s visit is recorded, because your server answers the request whether or not any JavaScript ever runs. If you want to know that agents are reaching your store at all, this is where you look, and it is the check most stores have never run.
Match on the token, then corroborate
Start by matching the user agent string against the tokens the assistant operators publish. Get those from each operator’s own documentation rather than from a list in an article, including this one: any republished list is a snapshot that begins decaying the moment it is written, and a stale list quietly under reports.
A user agent is self declared, so treat a match as a hypothesis. Operators that publish their address ranges give you something to check it against, and a request that is cryptographically signed gives you something stronger. This is the problem web bot auth exists to solve.
Read the shape, not just the count
The sequence tells you more than the total. A shopping agent answering a question fetches narrowly: your robots.txt, your sitemap or feed, then a handful of product pages. A crawler surveys broadly and steadily. If you are seeing wide sweeps and calling it agentic shopping traffic, you are counting the wrong thing.
Two patterns are worth flagging when you see them, because both are failures rather than traffic:
- A request for your robots.txt followed by nothing. You told it to leave and it left. See robots blocks agent.
- Repeated requests answered with a challenge response rather than your page. It arrived, met a wall, and never saw your store. See bot wall.
What to record so this stays useful
Keep the user agent, the address, the path, the status code and the response size. Status code and size are the two people forget, and they are what separate "an agent read my product page" from "an agent was served an interstitial", which look identical if you only count requests.
The limit of what logs can tell you
Logs prove arrival, permission and what was served. They do not tell you whether the agent could use what it got. A page served with a clean status code and no price in it is a successful request and a failed read.
For that you have to look at the data itself, which is what happens when an AI agent visits your store. Pair the logs with the analytics view in how to find AI agent traffic in Google Analytics and you have both halves: the agent that arrived, and the person it sent afterwards.
Questions
Can I trust the user agent string?
As a starting point, not as proof. It is self declared and trivially copied. An operator that publishes its address ranges gives you something to corroborate against, and a signed request gives you something better still. Match on the token, then verify.
Where do I get the current list of tokens?
From each operator’s own published documentation, which is the only source that stays correct. Any list republished in an article, including this one, is a snapshot that starts decaying immediately. Build the check so adding a token is a config change rather than a deployment.
What does a healthy pattern look like?
Requests for your robots.txt, your sitemap and your feed, followed by product pages rather than a broad sweep of everything. An agent answering a question fetches narrowly. A crawler surveys. The shape tells you which one you are looking at.