StorefrontSurvey crawler

StorefrontSurvey is a research crawler. It visits the public homepages of online stores to record which e-commerce platform and technologies each store uses. The results feed aggregate studies of e-commerce sites and a directory of online stores.

What it fetches

• /robots.txt, then the homepage, following redirects.
• Nothing else: no forms, logins, carts, checkouts, search or product crawling. It keeps the homepage as publicly served, which is what it reads to detect the platform and technologies.
• It requests one page at a time from each site. If robots.txt answers 429 or a 5xx error, it skips the site until a later crawl; a throttled homepage is not kept and is retried later.
• Requests to stores on shared platforms (Shopify, Squarespace) are spaced out, slowed further on 429 (or Shopify's 430), and paused for Retry-After or up to an hour.

How to recognise it

• User-Agent: StorefrontSurvey/1.0 (+https://storefrontsurvey.com; research crawler; reads robots.txt and the homepage only)
• Requests come from 62.83.97.123.
• Every request is signed with Web Bot Auth (Signature-Agent: "https://storefrontsurvey.com"). The public keys are at /.well-known/http-message-signatures-directory.

Opting out

The crawler follows your robots.txt group for User-agent: StorefrontSurvey. If there is none, it follows the * group. To block it, add:

User-agent: StorefrontSurvey
Disallow: /

Or email bot@storefrontsurvey.com with the domains to exclude, and we stop visiting them.

Contact

Questions or problems: bot@storefrontsurvey.com.