Learn / Fixes
How to let honest agents through bot protection
Bot protection is the earliest and bluntest way a store disappears from agentic shopping: a challenge that asks the visitor to enable JavaScript and wait is unanswerable to an agent, so every other quality of the store stops mattering. The work is separating identifiable, well behaved agents from the abusive traffic the protection was bought for, which is a policy question rather than a technical one.
This is bot wall, and unlike most failures it is total rather than partial.
The distinction that does the work
The instinct is to treat this as a dial between security and openness. It is better understood as identity. The agents worth admitting share three properties that abusive traffic does not:
- They declare who they are in a stable, published user agent.
- They come from address ranges the operator publishes, so the claim can be checked.
- They read and respect your robots rules, which means your stated policy is actually applied.
Admitting traffic that satisfies all three is a narrow opening, and it is precisely the opposite of what a scraper trying to look like a browser does.
Verify rather than trust
A user agent string is self declared and copied in seconds, so treat a match as a hypothesis and corroborate it against the operator’s published addresses. Where signed requests are available they are stronger, which is what web bot auth is for.
Get the tokens and ranges from each operator’s own documentation. A list republished in an article, including this one, begins decaying immediately, and a stale allowlist blocks new entrants silently.
Check what you are actually serving
The only trace of this failure is in your logs, where you are looking for repeated requests answered with a challenge rather than your page. Status code and response size are what separate an agent that read your product page from one that was served an interstitial, and counting requests alone shows them as identical.
See how to identify AI agent traffic in server logs.
Two layers, one policy
Your robots.txt is a request and your edge is enforcement. When they disagree the edge wins, which means a store can publish a welcoming robots.txt and block every agent that honours it.
Read both before concluding anything: robots.txt and your CDN settings.
Questions
If I let agents through, am I opening the door to scrapers?
Not if you admit by identity rather than by relaxing the whole rule. The agents worth admitting declare themselves, publish their address ranges and respect robots. That is a much narrower opening than turning protection down, and it is the opposite of what abusive traffic does.
How do I know a claimed agent is genuine?
Corroborate the user agent against the operator’s published addresses, since the string alone is self declared and trivially copied. Signed requests are better where available, which is the problem Web Bot Auth exists to address.
Will this show up as a problem before I look for it?
No, and that is the difficulty. A blocked agent does not retry, complain or appear in your analytics. Your logs record a challenge served instead of a page, which is the only trace there is.