Glossary

Crawl budget

Crawl budget is the limited number of requests an automated client will spend on a site before moving on, whether that limit comes from politeness, cost or time.

An agent answering one shopper’s question has a far smaller budget than a search crawler building an index. It is not surveying your catalogue, it is trying to answer something, so it takes the cheapest path to an answer and stops.

That is why enumeration cost decides so much. A catalogue readable in one request is affordable; one that must be reconstructed page by page frequently is not, and the difference is not ranking lower, it is not being considered.

Why it matters to a store

Being expensive to read is a form of invisibility that no amount of good product data fixes, because the data is never reached. It is the argument for a feed over relying on pages.

Scan your store