Stop Competitors Scraping Your Prices

If a competitor always seems to price just below you within hours of a change, you're probably being scraped. Automated bots crawl your catalog, extract prices and stock, and feed them into repricing engines or comparison sites. Scraping isn't as damaging as payment fraud, but it erodes margin, distorts your analytics, and hands rivals a real-time view of your business.

How price scrapers work

A scraper is a script that requests your product pages on a schedule and parses the response. Shopify makes this easier than most platforms because product data is highly structured:

• Product JSON endpoints. Appending .json to a product or collection URL, or hitting /products.json, returns clean structured data — no HTML parsing needed. This is the single most-abused vector.
• Sitemaps. /sitemap.xml hands over a complete list of every product URL to crawl.
• Rendered pages. More sophisticated scrapers run headless browsers (Puppeteer, Playwright) to execute JavaScript and mimic a real visitor.

Simple scrapers are noisy and easy to spot. Advanced ones rotate IPs, randomize timing, and forge headers to blend in — those need layered detection.

Detection signals

No single signal is definitive, but scrapers accumulate tells.

Signals.

• User-Agent. Default library UAs (python-requests, Go-http-client, curl, Scrapy, axios, empty or ancient browser strings) are obvious. Sophisticated scrapers spoof a real Chrome UA, so treat UA as one input, not a verdict.
• ASN. Bulk scraping runs on servers. Traffic from hosting/cloud ASNs hitting many product pages is high-confidence bot activity. Residential-proxy scrapers evade this, pushing detection to behavior.
• Velocity. A human views a handful of products per minute. A scraper requests dozens or hundreds. Watch requests-per-IP and requests-per-session over rolling windows.
• Access pattern. Sequential crawling of every product, repeated hits to .json endpoints, requests to /sitemap.xml followed by a systematic sweep, and zero asset loading (no CSS, images, or fonts).
• No interaction. No scroll, no mouse events, no add-to-cart — pure fetch-and-leave.

Best control. Combine ASN classification with per-IP velocity limits and endpoint monitoring. Datacenter IP + high velocity + .json endpoint hits is a scraper with near certainty; block it. Residential IP + high velocity + no interaction is a proxy-based scraper; challenge it.

Rate limiting the abused endpoints

Scraping is fundamentally a volume activity, so rate limiting is your most effective lever:

• Throttle .json and product endpoints harder than regular pages. Real shoppers rarely hit product JSON directly at volume.
• Set per-IP and per-session ceilings. Allow a generous burst for real browsing, then throttle sustained request rates that no human would produce.
• Watch the sitemap. A request for /sitemap.xml followed by rapid sequential product hits from the same source is a crawl in progress — clamp it.
• Escalate, don't just deny. First a challenge, then throttle, then block on continued abuse. This limits damage to real users caught in the crossfire.

Content protection layers

Rate limiting slows scrapers; these measures make your data less useful and harder to reach:

• Block by category. Deny datacenter, VPN, proxy, and Tor traffic from the endpoints scrapers depend on. This alone stops the majority of unsophisticated scraping.
• Filter bad User-Agents. Block known scraping libraries and empty UAs outright — cheap and effective against low-effort bots.
• Detect headless browsers. Automation frameworks leak fingerprint tells (missing plugins, automation flags, unusual rendering). Flag and challenge them.
• Monitor for anomalies. Alert when a single IP or ASN suddenly requests your entire catalog.

Shieldy Fraud Filter handles the network layer for this: it classifies visitors by ASN and blocks datacenter, VPN, proxy, and Tor traffic before it reaches your catalog, and because it runs on Shopify Functions the rules apply consistently rather than only on pages a bot can bypass. Pairing that with UA filtering and velocity limits closes most of the scraping surface.

Avoiding false positives

Some legitimate bots must be allowed, or you'll hurt yourself:

• Search engines. Googlebot and Bingbot need access for SEO. Verify them by reverse DNS or official IP ranges rather than trusting the UA string, then allowlist.
• Your own tools. Analytics crawlers, uptime monitors, and price-sync integrations you rely on should be exempted by IP or ASN.
• Shopping feeds. If you submit to Google Shopping or affiliates, ensure their fetchers aren't caught.
• Real power users. A genuine shopper comparing many products can trip velocity limits — set thresholds well above normal human browsing and prefer a challenge over an immediate block.

The payoff

You will never make scraping impossible — a determined competitor with a browser can copy a price. The goal is to make it expensive and unreliable: block the easy vectors, throttle the abused endpoints, and force scrapers onto costly residential-proxy infrastructure that still trips behavioral flags. Most repricing bots aren't worth that effort and will move on.

Curious how many bots are crawling your catalog right now? Turn on monitoring in Shieldy Fraud Filter and watch the ASN and velocity data — then tighten controls as needed from the pricing page.