HOPE-arastirma crawler

This page describes the automated agent that identifies itself with the User-Agent Mozilla/5.0 (compatible; HOPE-arastirma; +https://bot.roboshop.com.mx) and signs its requests with Web Bot Auth (Signature-Agent: "https://bot.roboshop.com.mx").

Purpose

E-commerce market research. The crawler reads publicly available product information (product title, description, images, price, variants) from public online store pages in order to analyse product assortments and pricing across stores. The data is used internally by BRECCIA LLC for its own product decisions. It is not resold, republished or shared outside our organisation.

What it does not do

• No login, no checkout, no cart actions, no form submissions.
• No collection of personal or customer data.

Request rate

• Product page / product JSON requests: at most 1 request every 10 seconds per store, and at most 200 requests per store per day.
• Sitemap reading: at most 1 request every 2 seconds per store, typically a few requests per store per crawl.

The crawler backs off when it receives HTTP 429 and honours Retry-After.

robots.txt

The crawler fetches and follows each store's robots.txt, including Crawl-delay when present.

Opt-out and contact

To exclude your store, disallow HOPE-arastirma in your robots.txt, or email info@brecciallc.org with your store domain and we will stop crawling it.

Verification

Public keys: /.well-known/http-message-signatures-directory