Published2026-05-22

robots.txt.liquid

Shopify robots.txt: default rules, safe edits, AI bots

robots.txt.liquid is the editable Liquid template that renders Shopify's /robots.txt file. It lives in your theme at templates/robots.txt.liquid². It is not in the theme by default — you create it when you need to customise behaviour. Shopify's verbatim warnings: "Shopify Support can't help with edits to the robots.txt.liquid file" and "Incorrect use of the feature can result in loss of all traffic"¹.

This cluster is the deepest in the entire SEO pillar because robots.txt is where the AI shopping wedge meets classical SEO. The five leaves below cover the default-rule list, AI bot configuration, app-leftover route disallow, the sitemap directive, and the ThemeKit-vs-admin-upload preservation gotcha.

§ 01 What

What robots.txt.liquid actually is

Every Shopify store serves a /robots.txt whether you touch it or not; robots.txt.liquid is the theme template that lets you override what it says. You create it at theme/templates/robots.txt.liquid when you need to add or change rules — this page is the editing guide.

The file has three components per Shopify's docs²: User agent, Rules (Allow / Disallow, plus optional crawl-delay), and an optional Sitemap URL. The Liquid objects you write against are robots.default_groups, group.user_agent, group.rules, group.sitemap, and sitemap.directive / sitemap.value. For the definition of the file itself and where it came from, see the robots.txt.liquid entry in the glossary.

§ 02 Warnings

Shopify's verbatim warnings — read these first

Two verbatim warnings from Shopify's Editing robots.txt.liquid help page are worth quoting in full: 'Shopify Support can't help with edits to the robots.txt.liquid file' and 'Incorrect use of the feature can result in loss of all traffic.' Both are literal. A misplaced Disallow rule on the wrong user agent block can de-index your entire store. This is the most powerful and most dangerous file Shopify exposes to merchants.

WATCH OUT

The robots.txt mistakes that kill stores

The three most common fatal edits, in order of frequency:

A Disallow: / rule on User-agent: * — blocks all crawlers from the entire store. Usually pasted from a staging environment robots.txt and forgotten.
Replacing the default-groups loop with a plain-text robots.txt — permanently disconnects the store from Shopify's automatic default-rule updates.
A Disallow: /products/ rule — blocks all product pages while looking like it blocks something else. Common copy-paste error from generic SEO guides written for non-Shopify platforms.

Check every edit in the rendered /robots.txt and with Search Console's URL Inspection tool before pushing to live. Edit in a development theme first.

§ 03 Default template

The verbatim default robots.txt.liquid template

Shopify's Dev Docs publish a default Liquid template for robots.txt.liquid. The block iterates over robots.default_groups and renders each group's user agent, rules, and optional sitemap. This is the starting point Shopify recommends — layer your customisations on top of this loop, not in place of it. Replacing the loop with plain text permanently disconnects the store from Shopify's automatic default-rule updates.

liquid Default robots.txt.liquid template — verbatim from Shopify Dev Docs

{% for group in robots.default_groups %}
  {{- group.user_agent -}}
  {% for rule in group.rules %}
    {{- rule -}}
  {% endfor %}
  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Per Shopify Dev Docs²: it is "strongly recommended to use the provided Liquid objects whenever possible" because "The default rules are updated regularly to ensure that SEO best practices are always applied."

Two things the docs don't tell you. First, the loop no longer prints the same file a store without a template gets. On 16 September 2026, stores with a robots.txt.liquid printed Shopify's older default set through this loop, with /search and /policies/ still blocked, while stores without one served Shopify's newer managed file⁷. Creating the template is what switches a store from one to the other (the research note shows both files and how we checked). Second, Shopify doesn't document where its objects put their line breaks, and live files show custom rules and Sitemap lines fused onto the rule before them (…-remoteSitemap: https://…). Give every line you add a real line break of its own.

§ 04 Defaults

What Shopify's default robots.txt blocks (and why)

Shopify now has two defaults. A store with no robots.txt.liquid gets Shopify's managed file: it blocks /admin, the cart (/cart/ and /cart.js), /checkout and /checkouts/, /orders, /account (keeping /account/login open), sorted collection URLs, tag and multi-filter combinations like /collections/*+*, and theme preview links, and it no longer blocks /search or /policies/. A store with a robots.txt.liquid prints the older default set through the loop: /admin, /cart, /checkout, /collections/*+*, /search and /policies/ among others, plus separate groups for Nutch, AhrefsBot, AhrefsSiteAudit, MJ12bot and Pinterest. Each block exists for a reason — never unblock without understanding the consequence.

Shopify's help page still lists /search and /policies/ among the key default entries¹; the managed file served on 16 September 2026 dropped both⁷. Internal search pages now send an X-Robots-Tag: noindex, nofollow header instead, and Google can only read a noindex on a page it's allowed to crawl⁸. The leaf at /shopify-seo/robots-txt-default-rules/ covers each rule individually with the reasoning. The most-misunderstood block is /collections/*+*: the *+* pattern matches the filter URL signature Shopify generates when a customer combines two filters (e.g. /collections/dresses/red+xl). Each combination is a unique URL with thin, duplicate content — blocking the pattern saves crawl budget and prevents duplicate-content downgrades.

§ 05 Customizations

The customizations Shopify supports

Per Shopify's help doc, the supported customizations are: allow or disallow specific URLs, add crawl-delay rules for specific crawlers, add additional sitemap URLs, and block specific crawlers entirely. Changes take effect instantly, though crawlers may lag in fetching the updated file. Shopify recommends against replacing the Liquid with plain text — that prevents future automatic default-rule updates from applying.

The most common legitimate customisations: adding a Disallow rule for an uninstalled app that left crawlable URLs (cluster leaf app-leftover routes), adding an image sitemap URL from a third-party app (cluster leaf sitemap directive), and adding AI crawler rules (cluster leaf AI bots).

§ 06 AI bots

AI crawlers and the new robots.txt battlefield

In 2026 the active question on robots.txt.liquid is not classical SEO crawlers — it's AI crawlers. GPTBot (OpenAI's training crawler), ClaudeBot (Anthropic's training crawler), PerplexityBot (Perplexity's search crawler), Google-Extended (a token that controls Gemini training and grounding, not Google Search), and Common Crawl's CCBot all respond to robots.txt user-agent rules. The strategic decision is not 'block all AI bots' — that also shuts out the search and user-triggered fetchers, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User, that put your pages into AI answers. Cluster leaf 1C-2 walks the decision matrix bot by bot.

The wedge for this leaf is at /shopify-ai-search/ai-crawlers/ — the AI-side complement to this cluster. The two clusters together cover the full SEO + AI crawler decision. Shopify confirms in its crawling-your-store doc that "your store can be indexed by search engines and large language models (LLMs) without signatures" — meaning AI bots crawl by default unless you explicitly block them in robots.txt.liquid.

§ 07 ThemeKit

The ThemeKit preservation gotcha

Per Shopify's Editing robots.txt.liquid help: ThemeKit (Shopify's CLI for theme development) preserves robots.txt.liquid across pushes; admin theme uploads do not. This is the single most-missed operational detail in Shopify robots.txt management — an agency pushes a new theme version through the admin, the robots.txt.liquid file is silently dropped, and the store reverts to default rules. The fix is to use ThemeKit for any theme deployment that includes robots.txt.liquid customisation, or to manually re-add the file after each admin upload.

The leaf at /shopify-seo/robots-txt-themekit/ covers the preservation behaviour, the post-upload audit pattern, and the CI/CD setup for teams that need automated theme deploys without losing robots.txt.liquid.