Learn / Fixes
Cloudflare settings for AI shopping agents
Cloudflare separates bot traffic into categories, and the setting that governs shopping agents is usually one somebody accepted once rather than chose. That matters because this layer enforces where robots.txt only requests, so a store can publish a welcoming robots file and still block every agent that honours it.
Cloudflare separates bot traffic into categories, and the setting governing shopping agents is usually one somebody accepted once. It is worth reading, because this layer enforces where robots.txt merely requests.
The change worth knowing about
Cloudflare wrote on 1 July 2026 that from 15 September 2026, for new domains onboarding to Cloudflare, bots in its Training and Agent categories will be blocked by default on pages that display ads, that the Search category stays allowed, and that customers can opt out in their Security settings before that date.
Most storefronts do not display ads, so most are not directly affected by that specific default. The reason it matters anyway is what it demonstrates: these categories exist, they have defaults, and the defaults move. A setting nobody on your team chose is still your policy.
Read the categories separately
Search, training and agent traffic are three decisions with three different consequences, and the interface makes it easy to treat them as one. Refusing training costs you nothing in sales. Refusing agents removes you from a channel.
Write down what each is currently set to before changing anything, including anything set by a default rather than by a person. That record is what makes the next review quick.
Make the two layers agree
Your robots.txt states a policy and your edge enforces one. When they differ, the edge wins and your robots file is describing something that is not true.
The commonest version is a store that deliberately permits shopping agents in robots.txt while the agent category is blocked at the edge, so every well behaved agent reads the invitation and is then turned away. See how to configure robots.txt for AI shopping agents.
Verify from the outside
Fetch your store from a network you have never used it from, without your usual browser, and see what comes back. Then check your logs for challenge responses rather than page responses.
Both halves are needed: the first tells you what a stranger gets, and the second tells you what agents have been getting while nobody was looking. See how to identify AI agent traffic in server logs.
Questions
Does this affect my search rankings?
The search category is governed separately from the agent category, so blocking one does not imply blocking the other. That separation is the entire reason it is worth looking: the decisions are independent and are frequently made as though they were the same decision.
I did not change anything. Can this still have changed?
Yes. Vendor defaults change, and they apply to you without a deploy on your side. That is exactly why this belongs in a periodic check rather than in a one off configuration task.
Where do I see whether agents are being challenged?
In your own logs, by looking at what was served rather than what was requested. A challenge response and a product page are both requests. Only the status code and response size tell them apart.