Skip to content

AI Crawlers

ClaudeBot on Shopify: the three Anthropic bots explained

Published

The three Anthropic bots that visit Shopify stores

Anthropic runs ClaudeBot, Claude-User, and Claude-SearchBot. ClaudeBot is the training crawler. Claude-User is the live retrieval bot — it fetches pages when a Claude.ai user references a URL during a conversation. Claude-SearchBot powers the Claude.ai search surface. Each can be allowed or blocked independently in robots.txt.liquid. The three roles map to three different visibility outcomes for a Shopify store.

Anthropic's bot policy1 documents all three. The robots.txt.liquid syntax to control each is the standard Disallow-per-User-agent pattern documented in Shopify's editing-robots help page2.

Per-bot role on a Shopify store

ClaudeBot affects future Claude model training. Block it and the next-generation Claude won't know about your store; allow it and the future model includes your store in its training corpus. Claude-User affects mid-conversation product Q&A — when a buyer says 'check yourstore.com/products/abc' in a Claude chat, Claude-User fetches the page. Claude-SearchBot affects whether your store shows up in Claude.ai's search index.

BotAffectsIf blocked
ClaudeBotFuture Claude model trainingStore excluded from next-gen training data
Claude-UserMid-conversation page fetchesClaude can't fetch your URLs when users reference them
Claude-SearchBotClaude.ai search indexStore excluded from Claude.ai search results

Robots.txt.liquid directives for Claude bots

Standard User-agent + Disallow pattern. Per-bot directives are case-sensitive. To allow all three, omit Disallow rules for them (Shopify's defaults allow). To block one specifically, add a User-agent block with Disallow: /. To allow most paths but block specific ones, add granular Disallow rules under the user-agent block.

Most stores benefit from allowing all three. The argument for blocking ClaudeBot is the training opt-out argument; the argument for blocking Claude-User or Claude-SearchBot is harder to construct because doing so directly reduces current-state AI visibility.

Per-bot decision tree

Four questions, in order. The default answer for each is allow; deliberate blocks should be defensible.

  1. Do you want future Claude versions to know about your store? Yes → allow ClaudeBot. No → Disallow ClaudeBot.
  2. Do you want Claude to fetch your URLs when a user references them in a Claude chat? Yes → allow Claude-User. No → Disallow Claude-User.
  3. Do you want your store to appear in Claude.ai's search results? Yes → allow Claude-SearchBot. No → Disallow Claude-SearchBot.
  4. Default for most stores: allow all three.

ClaudeBot audit checklist

Five checks. The audit overlaps with the broader robots.txt.liquid audit but treats each Claude bot independently.

  1. Open theme code editor. Look for templates/robots.txt.liquid.
  2. Search for 'ClaudeBot' (case-sensitive). Note any Disallow rules.
  3. Search for 'Claude-User' and 'Claude-SearchBot' separately.
  4. Confirm any blocks are intentional (training opt-out) rather than inherited from 2024 templates.
  5. Verify the live file: curl https://yourstore.com/robots.txt.

Keep reading