Questions · Discover

Will fixing robots.txt make my store agent ready?

It removes one blocking condition, and that is worth doing, but it is the first of four stages rather than the whole test. robots.txt decides whether an agent is allowed to look. Whether it can then read your prices, tell your sizes apart and get to a payment screen are three separate questions, and each is answered by attempting a purchase rather than by reading a file. A store that unblocks robots and stops there has made itself visible and not necessarily buyable.

Why it happens

Most shopping agents fetch /robots.txt first and cache the result for the session. A rule that names their user agent, or a wildcard rule with a broad Disallow, removes the catalogue from what the agent is willing to request. Nothing is rendered, nothing is parsed, and no error is raised: the agent simply has no products to reason about.

The reason this is easy to ship by accident is that the same file is where stores block training crawlers. Blocking a crawler that ingests your copy for model training and blocking an agent that is shopping for a named customer are different decisions, and one wildcard collapses them into the same rule.

Run a free scan and find out which of these your store actually hits.