Blocking AI Scrapers & LLM Bots

A new class of crawler has arrived on your store. Beyond the familiar search engine spiders, AI companies now run bots that harvest web content to train large language models and to answer user questions in real time. Your product descriptions, prices, and images are exactly the kind of structured content they want.

For merchants, this raises a real question: is an AI bot reading your store a helpful new referral channel or an uncompensated data grab? The answer depends on the bot and your goals—and either way, you should be the one deciding, not the crawler.

Meet the AI crawlers

AI bots identify themselves through user-agent strings, much like Googlebot does. The major ones you'll encounter include:

• GPTBot — OpenAI's crawler for gathering training data.
• ClaudeBot and anthropic-ai — Anthropic's crawlers.
• Google-Extended — Google's token for AI/Gemini training, separate from regular Googlebot.
• CCBot — Common Crawl, whose massive dataset feeds many AI models.
• PerplexityBot and OAI-SearchBot — crawlers that power AI search answers, which may cite and link back to you.
• Bytespider — TikTok/ByteDance's aggressive crawler.

A key distinction: some of these gather training data (they take your content, you get nothing back), while others power live AI answers (they may cite your store and send referral traffic). That difference should drive your policy.

Why robots.txt isn't enough

The conventional way to manage crawlers is robots.txt. You can add directives like:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

This tells compliant bots to stay out. On Shopify you can edit robots.txt.liquid to add these rules. And for well-behaved crawlers from reputable companies, it genuinely works—the major AI firms honor it.

But robots.txt has a fundamental weakness: it's a request, not a barrier. It's a note on the door asking bots to behave. Consider what it doesn't do:

• It can't stop non-compliant bots. Scrapers that ignore the file—including many operating under AI-training pretenses—crawl anyway.
• It can't stop spoofers. A bot can claim any user agent it likes, including pretending to be Googlebot to slip past your rules.
• It offers no enforcement. There's no penalty for ignoring it; the file is purely advisory.

So robots.txt is a policy statement. It sets your stated preference and turns away the honest crawlers. It does nothing to the ones that matter most—the aggressive, disguised, or bandwidth-hungry scrapers.

When to allow versus block

Blanket-blocking every AI bot can be a mistake. Think in categories:

Consider allowing:

• AI search crawlers that cite sources (PerplexityBot, OAI-SearchBot). If AI-driven search sends shoppers to your store, blocking it cuts off an emerging discovery channel—the AI equivalent of blocking Google.

Consider blocking:

• Pure training crawlers (GPTBot, CCBot, Google-Extended) if you don't want your content feeding models without attribution or referral value.
• Aggressive crawlers (Bytespider) known for heavy request volume that strains your store.
• Anything hammering your server regardless of its stated purpose—bandwidth and performance come first.

There's no universally correct answer. A content-heavy brand protecting original copy leans toward blocking training bots. A store chasing every discovery channel might allow the citation-driven ones. Decide deliberately, then enforce your decision.

Enforcement beyond robots.txt

Here's the crux: stating a preference and enforcing it are different things. Once you've decided which AI bots to keep out, you need a mechanism that actually stops the ones ignoring your robots.txt.

That means detecting and blocking at the access level—by user agent, by network, and by behavior. This is where Shieldy — Fraud Filter turns policy into enforcement. It blocks bots, VPNs, proxies, and Tor traffic, so crawlers that ignore your robots.txt or spoof a friendly user agent get stopped before they reach your content. Its content protection features specifically guard the product data and imagery that AI scrapers are after.

The combination is what works: use robots.txt to declare your policy to compliant crawlers, and use Shieldy to enforce it against the ones that don't listen. A note on the door plus a lock on it. For stores serious about controlling how their catalog is used, the Enterprise plan at $8.99/mo adds the country and network-level filtering that makes enforcement airtight.

A practical setup

Put the pieces in order:

1. Decide your policy. List which AI bots to allow (citation crawlers) and which to block (training and aggressive crawlers).
2. Update robots.txt.liquid with your Disallow rules to turn away compliant bots and formalize your stance.
3. Deploy enforcement to catch non-compliant and spoofing crawlers that ignore the file.
4. Monitor your logs. Watch for user agents and network origins that shouldn't be crawling, and adjust as new AI bots appear—this space changes fast.
5. Review quarterly. New crawlers launch regularly, and the allow/block calculus shifts as AI search matures.

Own your content's future

AI crawlers aren't going away, and the volume of AI-driven scraping only grows. You don't have to accept it passively. Decide which bots serve your interests, declare that policy through robots.txt, and enforce it with real access-level blocking for everything that ignores the rules.

Your catalog is your asset. Ready to control who gets to read it? See how Shieldy enforces your policy and take back the decision.