Glossary

User agent

A user agent is the string a client sends to identify itself when requesting a page, and it is how a store distinguishes a browser from a search crawler from a shopping agent.

It is self declared, which is the whole difficulty. Anything can claim to be anything, so a match tells you what the requester says it is rather than what it is.

Operators that publish their address ranges give you something to corroborate against, which turns a claim into a check. A cryptographically signed request is stronger again, and is the problem Web Bot Auth exists to address.

Why it matters to a store

Every decision a store makes about admitting agents currently rests on this string. Treating it as proof admits anything that copies it; treating it as a hypothesis and verifying the address is what makes a narrow allowlist defensible.

Scan your store