Questions · Discover

Can I block AI training crawlers but not shopping agents?

Yes. They send different user agents. Write the Disallow rules against the named crawlers you want to exclude rather than under a wildcard, and check the file again after any theme or platform upgrade, because generated robots.txt files are frequently overwritten.

Why it happens

Most shopping agents fetch /robots.txt first and cache the result for the session. A rule that names their user agent, or a wildcard rule with a broad Disallow, removes the catalogue from what the agent is willing to request. Nothing is rendered, nothing is parsed, and no error is raised: the agent simply has no products to reason about.

The reason this is easy to ship by accident is that the same file is where stores block training crawlers. Blocking a crawler that ingests your copy for model training and blocking an agent that is shopping for a named customer are different decisions, and one wildcard collapses them into the same rule.

Run a free scan and find out which of these your store actually hits.