Skip to content

Free tool · Technical SEO

Shopify robots.txt generator and validator

Build a templates/robots.txt.liquid that keeps Shopify's default rules and adds yours: AI crawler policy, extra Disallow paths, custom groups and sitemaps. Paste any robots.txt.liquid or live /robots.txt to check it, and test a URL the way Googlebot reads it.

Free, no signup Built by Billy Reiner Updated

1 · Build robots.txt.liquid

AI crawlers

Blocking training crawlers costs no search or AI-answer visibility. Blocking search and user fetchers takes you out of those AI answers.

Choose crawlers one by one
Model training
AI search and citations
User-triggered fetchers
Shopify's default rules
Extra Disallow paths (all crawlers)

Already blocked by Shopify's defaults, so not offered: /collections/*sort_by* (sorted collection URLs), /collections/*+* (tag combinations (red+large)), /collections/*filter*&*filter* (two or more filters combined), /*preview_theme_id* (theme preview links).

Added inside Shopify's * group. Full URLs are turned into paths.

Write them as robots.txt. A crawler with its own group ignores the * group, Shopify's defaults included.

Your main /sitemap.xml is already printed by the loop.

Advanced: remove another default rule

Shopify Support can't help with edits to this file. Remove a rule only when you know why it's blocking something you need crawled.

Uses Shopify's documented unless pattern. If the rule isn't in your defaults, nothing changes.

With nothing selected you don't need a robots.txt.liquid: Shopify's managed file already does this job. Pick an option to build one.

templates/robots.txt.liquid 
{% for group in robots.default_groups -%}
{{ group.user_agent }}
{% for rule in group.rules -%}
{{ rule }}
{% endfor -%}
{% if group.sitemap != blank -%}
{{ group.sitemap }}
{% endif %}
{% endfor -%}

Shopify's loop and nothing else.

    What the template prints (approximation) 
    User-agent: *
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /[shop-id]/checkouts
Disallow: /[shop-id]/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*
Disallow: */collections/*filter*&*filter*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*/blogs/*+*
Disallow: /*/blogs/*%2B*
Disallow: /*/blogs/*%2b*
Disallow: /*?*oseid=*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*
Disallow: /policies/
Disallow: /*/policies/
Disallow: /*/*?*ls=*&ls=*
Disallow: /*/*?*ls%3D*%3Fls%3D*
Disallow: /*/*?*ls%3d*%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/*.js
Disallow: /recommendations/products
Disallow: /*/recommendations/products
Disallow: /services/login_with_shop
Disallow: /products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Sitemap: https://your-store.com/sitemap.xml

User-agent: adsbot-google
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /orders
Disallow: /[shop-id]/checkouts
Disallow: /[shop-id]/orders
Disallow: /*?*oseid=*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*
Disallow: /cdn/wpm/*.js
Disallow: /products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /services/login_with_shop

User-agent: Nutch
Disallow: /

User-agent: AhrefsBot
Crawl-delay: 10
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /[shop-id]/checkouts
Disallow: /[shop-id]/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*
Disallow: */collections/*filter*&*filter*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*/blogs/*+*
Disallow: /*/blogs/*%2B*
Disallow: /*/blogs/*%2b*
Disallow: /*?*oseid=*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*
Disallow: /policies/
Disallow: /*/policies/
Disallow: /*/*?*ls=*&ls=*
Disallow: /*/*?*ls%3D*%3Fls%3D*
Disallow: /*/*?*ls%3d*%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/*.js
Disallow: /services/login_with_shop
Disallow: /products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Sitemap: https://your-store.com/sitemap.xml

User-agent: AhrefsSiteAudit
Crawl-delay: 10
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /[shop-id]/checkouts
Disallow: /[shop-id]/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*
Disallow: */collections/*filter*&*filter*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*/blogs/*+*
Disallow: /*/blogs/*%2B*
Disallow: /*/blogs/*%2b*
Disallow: /*?*oseid=*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*
Disallow: /policies/
Disallow: /*/policies/
Disallow: /*/*?*ls=*&ls=*
Disallow: /*/*?*ls%3D*%3Fls%3D*
Disallow: /*/*?*ls%3d*%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/*.js
Disallow: /services/login_with_shop
Disallow: /products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Sitemap: https://your-store.com/sitemap.xml

User-agent: MJ12bot
Crawl-delay: 10

User-agent: Pinterest
Crawl-delay: 1

    The loop's rules as templated stores printed them on 16 September 2026. Shopify can change them; your store's may differ.

      2 · Check a robots.txt.liquid or robots.txt

      Nothing leaves your browser.

      Paste a file and press Check file.

        3 · Test a URL path

        Pick what to test against, a crawler and a path. If the checker is empty, the tester uses Shopify's managed file.

        What robots.txt.liquid is

        Every Shopify store serves a /robots.txt file whether you touch it or not. robots.txt.liquid is the theme template that lets you change what it says. It lives at templates/robots.txt.liquid, it can't be a JSON template, and no theme ships with it: you add it when you need it2. The glossary entry for robots.txt.liquid has the short definition.

        Shopify puts a caution on the feature: it's an unsupported customization, Support can't help with it, and "Incorrect use of the feature can result in loss of all traffic"4. That's why this generator never lets you disallow / for every crawler and always keeps Shopify's default loop.

        Shopify's two default files

        On 16 September 2026 we fetched /robots.txt from 38 Shopify storefronts. Apart from three with fully custom files, they split cleanly in two.

        • Stores without a template (23 of them, including Shopify's own Dawn demo store) served a new managed file5. It opens with "# Shopify storefront. Public product, collection, page, blog, policy, cart, and localized HTML is crawlable", then a block of comments for AI agents. It has 87 rule lines in two groups (* and adsbot-google), starts with Allow: /, and doesn't block /search or /policies/.
        • Stores with a template (the other 12) printed an older set through the loop: /search and /policies/ blocked, and extra groups for Nutch, AhrefsBot, AhrefsSiteAudit, MJ12bot and Pinterest. Seven of them printed the same default lines, word for word, including recent additions, so this is what robots.default_groups returns today, not stale copies.

        Shopify hasn't announced the change that we could find, and the docs haven't caught up. The help page still lists /search and /policies/ among the key default entries4, and the developer docs say the default rules "are mirrored through the Liquid robots object"2. On live stores, they aren't mirrored: adding a template switches a store from the managed file to the older rules.

        One more detail explains why the managed file stopped blocking search. Shopify's /search?q= pages now send X-Robots-Tag: noindex, nofollow, on stores with and without a template (checked on four stores). Google can only obey that noindex if it's allowed to fetch the page: "the page or resource must not be blocked by a robots.txt file"8. A Disallow: /search hides the noindex, and blocked URLs can still be indexed from links. The opposite advice exists too: Google's crawl-budget guide prefers robots.txt over noindex for pages you never want crawled, because a noindexed page still gets fetched9. That guide is written for sites with a million-plus pages, or 10,000-plus changing daily. For most stores, letting the noindex work is the cleaner choice. The default rules guide covers the reasoning behind each block.

        Show both files side by side (approximation, 2026-09-16)
        robots.txt No template: Shopify's managed file (comment block left out) 
        User-agent: *
Allow: /
Allow: /products/account
Allow: /products/orders
Allow: /products/checkout
Allow: /*/products/account
Allow: /*/products/orders
Allow: /*/products/checkout
Allow: /collections/account
Allow: /collections/orders
Allow: /collections/checkout
Allow: /*/collections/account
Allow: /*/collections/orders
Allow: /*/collections/checkout
Allow: /pages/checkout
Allow: /*/pages/checkout
Allow: /blogs/*account
Allow: /blogs/*orders
Allow: /blogs/*checkout
Allow: /*/blogs/*account
Allow: /*/blogs/*orders
Allow: /*/blogs/*checkout
Disallow: /admin
Disallow: /cart/
Disallow: /*/cart/
Disallow: /checkout
Disallow: /*/checkout
Disallow: /checkouts/
Disallow: /*/checkouts/
Disallow: /orders
Disallow: /*/orders
Allow: /account/login
Allow: /*/account/login
Disallow: /account
Disallow: /*/account
Disallow: /[shop-id]
Disallow: /cdn/wpm/*.js
Disallow: /services
Disallow: /sf_*
Disallow: /cart.js
Disallow: /*/cart.js
Disallow: /recommendations/products
Disallow: /*/recommendations/products
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*
Disallow: /collections/*filter*&*filter*
Disallow: /*/collections/*filter*&*filter*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*/blogs/*+*
Disallow: /*/blogs/*%2B*
Disallow: /*/blogs/*%2b*
Disallow: /*?*ls=*&ls=*
Disallow: /*?*ls%3*ls%3*
Disallow: /*?*oseid=*
Disallow: /*?*preview_theme_id=*
Disallow: /*?*preview_script_id=*

User-agent: adsbot-google
Allow: /products/
Allow: /*/products/
Allow: /collections/
Allow: /*/collections/
Allow: /pages/
Allow: /*/pages/
Allow: /blogs/
Allow: /*/blogs/
Allow: /pages/checkout
Allow: /*/pages/checkout
Allow: /blogs/*checkout
Allow: /*/blogs/*checkout
Disallow: /checkout
Disallow: /*/checkout
Disallow: /checkouts/
Disallow: /*/checkouts/
Disallow: /orders
Disallow: /*/orders
Disallow: /services
Disallow: /sf_*
Disallow: /[shop-id]
Disallow: /cdn/wpm/*.js

Sitemap: https://your-store.com/sitemap.xml
        robots.txt With a template: what the loop prints (158 lines) 
        User-agent: *
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /[shop-id]/checkouts
Disallow: /[shop-id]/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*
Disallow: */collections/*filter*&*filter*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*/blogs/*+*
Disallow: /*/blogs/*%2B*
Disallow: /*/blogs/*%2b*
Disallow: /*?*oseid=*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*
Disallow: /policies/
Disallow: /*/policies/
Disallow: /*/*?*ls=*&ls=*
Disallow: /*/*?*ls%3D*%3Fls%3D*
Disallow: /*/*?*ls%3d*%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/*.js
Disallow: /recommendations/products
Disallow: /*/recommendations/products
Disallow: /services/login_with_shop
Disallow: /products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Sitemap: https://your-store.com/sitemap.xml

User-agent: adsbot-google
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /orders
Disallow: /[shop-id]/checkouts
Disallow: /[shop-id]/orders
Disallow: /*?*oseid=*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*
Disallow: /cdn/wpm/*.js
Disallow: /products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /services/login_with_shop

User-agent: Nutch
Disallow: /

User-agent: AhrefsBot
Crawl-delay: 10
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /[shop-id]/checkouts
Disallow: /[shop-id]/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*
Disallow: */collections/*filter*&*filter*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*/blogs/*+*
Disallow: /*/blogs/*%2B*
Disallow: /*/blogs/*%2b*
Disallow: /*?*oseid=*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*
Disallow: /policies/
Disallow: /*/policies/
Disallow: /*/*?*ls=*&ls=*
Disallow: /*/*?*ls%3D*%3Fls%3D*
Disallow: /*/*?*ls%3d*%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/*.js
Disallow: /services/login_with_shop
Disallow: /products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Sitemap: https://your-store.com/sitemap.xml

User-agent: AhrefsSiteAudit
Crawl-delay: 10
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /[shop-id]/checkouts
Disallow: /[shop-id]/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*
Disallow: */collections/*filter*&*filter*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*/blogs/*+*
Disallow: /*/blogs/*%2B*
Disallow: /*/blogs/*%2b*
Disallow: /*?*oseid=*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*
Disallow: /policies/
Disallow: /*/policies/
Disallow: /*/*?*ls=*&ls=*
Disallow: /*/*?*ls%3D*%3Fls%3D*
Disallow: /*/*?*ls%3d*%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/*.js
Disallow: /services/login_with_shop
Disallow: /products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Disallow: /*/collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote
Sitemap: https://your-store.com/sitemap.xml

User-agent: MJ12bot
Crawl-delay: 10

User-agent: Pinterest
Crawl-delay: 1

        [shop-id] stands for your store's numeric ID. Both files change when Shopify changes them.

        Do you need a template?

        Probably not, if all you want is sensible defaults. The managed file already allows AI crawlers, blocks checkout, cart, account and the filter traps, and lists your sitemap. You need a template when you want a rule it doesn't have:

        If you already have a template that only repeats Shopify's loop, deleting it moves the store to the managed file. Shopify documents the delete steps on the help page4. If you keep one, the generator's "match Shopify's managed file" option skips the three older blocks so your file behaves like the managed one on search and policy pages.

        The loop, line by line

        This is the loop exactly as Shopify's template reference prints it2:

        liquid Shopify's documented robots.txt.liquid 
        {% for group in robots.default_groups %}
  {{- group.user_agent -}}

  {% for rule in group.rules %}
    {{- rule -}}
  {% endfor %}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}
        LineCodeWhat it does
        1{% for group in robots.default_groups %}Loops over Shopify's rule groups. Each group is one user-agent and its rules.
        2{{- group.user_agent -}}Prints the group's User-agent: line. The dashes are Liquid whitespace control: they strip the spaces and line breaks around the tag.
        4{% for rule in group.rules %}Loops over the group's rules. Each rule has a directive (Allow or Disallow) and a value (the path)3.
        5{{- rule -}}Prints one rule, for example Disallow: /admin.
        8{%- if group.sitemap != blank -%}Shopify returns a blank sitemap for groups that don't need one, so only the others print it.
        9{{ group.sitemap }}Prints Sitemap: https://your-store.com/sitemap.xml.
        10–11{%- endif -%} {% endfor %}Close the sitemap check and the groups loop.

        Lines 8 to 10 are the ones people drop when they copy a snippet, and then robots.txt stops telling crawlers where the sitemap is. The validator flags that.

        The generator prints the same objects but lays the loop out differently, with one printed item per line and no dash that could strip a line break:

        liquid The generator's loop, with nothing selected 
        {% for group in robots.default_groups -%}
{{ group.user_agent }}
{% for rule in group.rules -%}
{{ rule }}
{% endfor -%}
{% if group.sitemap != blank -%}
{{ group.sitemap }}
{% endif %}
{% endfor -%}

        The reason is line breaks. Shopify doesn't document where its objects put them3, and on live stores lines run together. A large DTC store's robots.txt currently contains …-remoteDisallow: /collections/*-loop and …-remoteSitemap: https://…/sitemap.xml: a custom rule and a Sitemap line glued to the last default rule, so all three are broken. A 2021 GitHub report describes the same thing with Shopify's own one-line example20. The pattern fits objects that bring a line break before a user-agent or rule line but not before a Sitemap line, which would mean the documented loop above can glue the Sitemap line too. Ending every line with its own break sidesteps the question. If an object also adds one, you get a blank line, which Google and RFC 9309 crawlers ignore.

        Four safe customisations

        Shopify documents three kinds of change: add a rule to an existing group, remove a default rule, and add rules outside the loop1. Each pattern below is the generator's output for that change, so it keeps the loop and its line breaks.

        1. Add a rule to Shopify's * group

        Check for the * group inside the loop and print your rule there, which keeps one group per crawler.

        liquid Adds Disallow: /apps/old-review-app/ to the * group 
        {% for group in robots.default_groups -%}
{{ group.user_agent }}
{% for rule in group.rules -%}
{{ rule }}
{% endfor -%}
{% if group.user_agent.value == '*' -%}
Disallow: /apps/old-review-app/
{% endif -%}
{% if group.sitemap != blank -%}
{{ group.sitemap }}
{% endif %}
{% endfor -%}

        2. Remove a default rule

        Skip the rule inside the rules loop with unless, Shopify's documented method1. This one drops the loop's Disallow: /search, so Google can fetch search pages and read their noindex header. When a rule isn't there, the unless does nothing.

        liquid Skips Disallow: /search in every group 
        {% for group in robots.default_groups -%}
{{ group.user_agent }}
{% for rule in group.rules -%}
{% unless rule.directive == 'Disallow' and rule.value == '/search' -%}
{{ rule }}
{% endunless -%}
{% endfor -%}
{% if group.sitemap != blank -%}
{{ group.sitemap }}
{% endif %}
{% endfor -%}

        3. Block a crawler or add a group

        Crawlers that aren't in the default set get their own group, written as plain text after the loop. Shopify's example blocks discobot1. The same shape blocks an AI training crawler.

        liquid Plain-text groups after the loop 
        {% for group in robots.default_groups -%}
{{ group.user_agent }}
{% for rule in group.rules -%}
{{ rule }}
{% endfor -%}
{% if group.sitemap != blank -%}
{{ group.sitemap }}
{% endif %}
{% endfor -%}

# AI model training crawlers. Blocking these doesn't remove you from search or AI answers.
User-agent: GPTBot
Disallow: /

User-agent: discobot
Disallow: /

        Don't do this for Googlebot or Bingbot unless you mean it. A crawler with its own group follows only that group7, so a Googlebot group holding just a crawl-delay would let Googlebot into /checkout and the cart.

        4. List another sitemap

        Add a Sitemap: line after the loop. It needs the full URL with https:// and the host, and it can point at another host7. Your main /sitemap.xml already comes from {{ group.sitemap }}.

        liquid An extra sitemap after the loop 
        {% for group in robots.default_groups -%}
{{ group.user_agent }}
{% for rule in group.rules -%}
{{ rule }}
{% endfor -%}
{% if group.sitemap != blank -%}
{{ group.sitemap }}
{% endif %}
{% endfor -%}

# Extra sitemaps
Sitemap: https://your-store.com/apps/image-sitemap/sitemap.xml

        AI crawlers and what blocking costs

        Two decisions get mixed up here. Training crawlers feed future models, and blocking them costs no visibility. Search crawlers and user-triggered fetchers put your pages into AI answers, and blocking them takes you out. Shopify adds a third point: if you sell through agentic storefronts such as ChatGPT or Microsoft Copilot, your product data reaches them through Shopify Catalog "independently of /robots.txt", so a robots block affects only open-web discovery4. For the policy side, read AI bots in Shopify's robots.txt and AI crawlers on Shopify.

        Model training

        TokenWhat it doesWhat blocking costsFollows robots.txt
        GPTBot
        OpenAI        		 Collects pages that may be used to train OpenAI's foundation models.13 Opts your content out of OpenAI training. ChatGPT search is a separate bot, and OpenAI says each setting is independent. Yes
        ClaudeBot
        Anthropic        		 Collects web content for Anthropic's model training.14 Signals that your future content should be left out of Anthropic's training data. Claude-SearchBot and Claude-User are separate. Yes
        Google-Extended
        Google        		 A robots.txt token, not a crawler. It controls whether content Google already crawls can be used to train Gemini models and to ground Gemini answers.10 Google says it doesn't affect inclusion or ranking in Google Search. AI Overviews are part of Search and follow Googlebot. Token only, read by the vendor's main crawler
        Applebot-Extended
        Apple        		 A token, not a crawler. It controls whether Apple can use your pages to train its foundation models.16 Apple says pages that disallow it can still appear in its search results. Token only, read by the vendor's main crawler
        CCBot
        Common Crawl        		 Builds Common Crawl's open web archive, a dataset many AI labs train on.17 Keeps new pages out of future Common Crawl snapshots. No search product depends on it. Yes
        meta-externalagent
        Meta        		 Meta says it's used for training foundation AI models or improving products by indexing content directly.18 Opts out of Meta AI training. Meta also uses it for product indexing, so the block reaches past training. Yes
        Amazonbot
        Amazon        		 Amazon says it improves Amazon's products and services and may be used to train Amazon AI models.19 Opts out of Amazon AI training. It also feeds Amazon's products, so the block reaches past training. Yes

        AI search and citations

        TokenWhat it doesWhat blocking costsFollows robots.txt
        OAI-SearchBot
        OpenAI        		 Finds pages to show in ChatGPT's search features.13 Your pages stop appearing in ChatGPT search answers. Yes
        Claude-SearchBot
        Anthropic        		 Crawls to improve the quality of Claude's search results.14 Anthropic says blocking it stops your content being indexed for search, which can cut your visibility in Claude's answers. Yes
        PerplexityBot
        Perplexity        		 Surfaces and links sites in Perplexity's search results. Perplexity says it isn't used to train models.15 Your pages stop being cited as Perplexity sources. Yes
        meta-webindexer
        Meta        		 Meta says it improves Meta AI search result quality.18 Less of your store in Meta AI's search answers. Yes
        Amzn-SearchBot
        Amazon        		 Improves search in Amazon products. Amazon says it doesn't crawl for model training.19 Less of your store in search experiences inside Amazon products. Yes

        User-triggered fetchers

        TokenWhat it doesWhat blocking costsFollows robots.txt
        ChatGPT-User
        OpenAI        		 Fetches a page for certain user actions in ChatGPT and Custom GPTs.13 OpenAI says robots.txt rules may not apply to it, because a user started the request. May ignore it
        Claude-User
        Anthropic        		 Fetches a page when a Claude user asks a question that needs it.14 Claude can't read your page when a shopper asks it about your store. Yes
        Perplexity-User
        Perplexity        		 Fetches a page when a Perplexity user asks a question.15 Perplexity says this fetcher generally ignores robots.txt, because a user asked for the page. May ignore it
        meta-externalfetcher
        Meta        		 Fetches individual links at a user's request.18 Meta says it may bypass robots.txt rules. May ignore it
        Amzn-User
        Amazon        		 Handles user actions, such as answering Alexa questions that need current information.19 Amazon says it may not follow all robots.txt directives, because a user triggered the request. May ignore it

        Three facts trip people up. Google-Extended controls Gemini training and grounding, and Google says it "does not impact a site's inclusion in Google Search"10. AI Overviews are part of Search, so Googlebot's rules and the snippet controls decide those11. Anthropic's current page lists ClaudeBot, Claude-SearchBot and Claude-User; guides that block anthropic-ai or claude-web target tokens that page no longer mentions14. And OpenAI, Perplexity, Meta and Amazon all say their user-triggered fetchers may not follow robots.txt, because a person asked for the page.

        What the validator catches

        Paste a template and the checker renders it against the loop's current rules, then runs the same checks it runs on a live /robots.txt. Blocking checks use the real matcher, so a wildcard that happens to cover /products/ gets caught too. Paste a live file and it also tells you which of Shopify's two default files it's looking at.

        CheckSeverityWhy it matters
        Default loop removed, or no {{ rule }} inside itCriticalShopify's rules for checkout, cart and the filter traps stop printing, and never update again.
        {{ group.user_agent }} missingCriticalThe rules print without their User-agent line and attach to the wrong group or none.
        Unbalanced Liquid tagsCriticalA missing {% endfor %} or {% endif %} means Liquid can't parse the file.
        Disallow: / for *, Googlebot or BingbotCriticalThe whole store drops out of crawling. This is the "loss of all traffic" edit.
        Homepage, /products/, /collections/ or theme CSS/JS under /cdn/ blockedCriticalGoogle can't crawl what you sell, or can't render the pages it does crawl.
        File over 500 KiBCriticalGoogle ignores everything after the first 500 KiB7.
        Two directives on one line, or a line the template doesn't end with a breakWarning"…-remoteSitemap: https://…" breaks both the rule and the sitemap line.
        group.sitemap lines dropped, or no Sitemap lineWarningCrawlers lose the pointer to your sitemap.
        A Disallow over /search, which already sends a noindex headerWarning (info when it's the loop's own rule)Google can't see a noindex on a page it isn't allowed to fetch8. Very large stores may still keep the block for crawl budget.
        AI search or user bots blockedWarningNamed with what each one costs. Training bots are listed as information, not a problem.
        noindex, nofollow in robots.txtWarningGoogle reads only user-agent, allow, disallow and sitemap7.
        Duplicate groups; a Googlebot or Bingbot group that skips the * rulesWarningOlder crawlers may read only the first group; a crawler with its own group ignores Shopify's defaults.
        Rules before any User-agent, paths without a leading /, typos, invisible characters, curly quotes, BOMWarningSkipped by crawlers, or accepted by Google and nobody else. Pasted-in characters are invisible in the editor.
        A template at all; objects the template doesn't support; outdated tokens; crawl-delay for GoogleInfo / warningA template switches the store to the older defaults. Only robots, group, rule, user_agent, sitemap and request work here1.

        How Google reads the rules

        The path tester follows Google's published rules7. A crawler obeys one group: the one naming its user-agent token, or the * group if none does. Groups that name the same crawler are merged. User-agent matching ignores case; paths don't. Inside the group, the longest matching rule path wins, and when an Allow and a Disallow tie, the Allow wins. Only two characters are special: * matches any run of characters, and $ at the end anchors the match to the end of the URL. A ? is just a character.

        URLRulesResult
        /pageallow: /p, disallow: /Allowed. The Allow is longer.
        /folder/pageallow: /folder, disallow: /folderAllowed. A tie goes to the Allow.
        /page.htmallow: /page, disallow: /*.htmBlocked. The Disallow path is longer.
        /allow: /$, disallow: /Allowed. /$ matches only the root.
        /account/loginManaged file: allow: /account/login, disallow: /accountAllowed. The login page stays crawlable.
        /collections/x?filter.v.option.color=redEither default fileAllowed. Only two or more filters together are blocked6.
        /search?q=shirtManaged file / the loopAllowed (and noindexed by its header) / blocked.

        The tool's automated tests run every example in Google's wildcard and precedence tables. Two limits: it reads rules the way Google does, and other crawlers can differ at the edges. For a pasted template it tests the rendered approximation, not your store's exact defaults.

        Install and test

        1. From your Shopify admin, go to Online Store. For the theme, click the … menu, then Edit code4. Work on an unpublished copy of the theme first.
        2. Click Add a new template, select robots, then Create template. The developer docs describe the same step as creating robots.txt.liquid in the Templates folder2.
        3. Replace the contents with your generated file and save. Changes are instant.
        4. Open your-store.com/robots.txt, paste it into the checker as live output, and make sure every directive is on its own line.
        5. Test the URLs that matter with the path tester, then with Search Console's URL Inspection tool. Search Console's robots.txt report shows the file Google fetched and lets you request a recrawl12. Google caches robots.txt for up to 24 hours7.

        FAQ

        Can I edit robots.txt on Shopify?

        Yes, through the robots.txt.liquid theme template. Shopify Support won't help with edits, so keep Shopify's loop, change as little as possible, and test before you publish4.

        Why can't I find robots.txt.liquid in my theme?

        It isn't included in themes by default. Until you add it, Shopify serves its managed robots.txt2.

        Why does my robots.txt still block /search and /policies/ when other stores' don't?

        Your theme almost certainly has a robots.txt.liquid. On 16 September 2026, stores with a template printed Shopify's older default rules through the loop, and stores without one got the newer managed file. Delete the template if it adds nothing you need, or use the generator's "match Shopify's managed file" option.

        Should I block /search?

        On most stores, no. Shopify's search result pages send X-Robots-Tag: noindex, nofollow, and Google can only read that if the page isn't blocked8. The managed file leaves /search crawlable. A very large store worried about crawl budget can keep the block, which is what Google's crawl-budget guide suggests for sites of that size9.

        Will blocking GPTBot take my store out of ChatGPT?

        No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI says each setting is independent; blocking OAI-SearchBot is what removes you, and changes take about 24 hours13.

        Does blocking Google-Extended remove me from AI Overviews?

        No. Google-Extended covers Gemini training and grounding, not Search10. AI Overviews follow Googlebot, and Google points to nosnippet, data-nosnippet, max-snippet or noindex to limit what they show11.

        Can I noindex pages with robots.txt?

        No. Google doesn't support noindex in robots.txt, and a Disallow stops Google from seeing a noindex tag on the page8.

        Does this tool send my file anywhere?

        No. The generator, checker and tester run in your browser. Nothing is uploaded or stored.

        More free tools

        All free Shopify tools →

        Guides