Which AI crawlers should my robots.txt allow, and which can I block without losing visibility?
Search and user crawlers decide whether an assistant can read your product pages. Training crawlers do not, by each vendor's own statement.
- When this applies
- Your storefront robots.txt names AI crawlers, an app or a template offered to block them, or you want to keep product pages readable by assistants while staying out of model training. [8] [1]
- When this does not apply
- Product data that reaches ChatGPT, Copilot, or another activated channel through Shopify Catalog does not pass through robots.txt. A robots.txt decision changes only what crawlers read on the open web. [8] [10]
- Why this matters
- OpenAI states that a site opted out of OAI-SearchBot is not shown in ChatGPT search answers, and Google states that the Googlebot rule is the control for AI Overviews and AI Mode. One wrong line removes a route. [1] [5]
- What to do, or not do
- Read your live robots.txt. Keep every search and user crawler allowed. Decide the training crawlers on consent, since OpenAI, Google, and Anthropic each state that their training token does not decide search inclusion. [1] [4] [2]
- Sources checked
- September 5, 2026
When a third party is worth paying for, and what you do not have to buy
- When paid help is worth it
- Paid help is worth it only when a headless or custom storefront sits behind a firewall or CDN that challenges crawlers at the network layer, which robots.txt cannot see or fix. [8]
An app, a template, or an agency may offer to block AI crawlers on your store. The offer bundles two different crawlers under one word. One kind reads a page so an assistant can answer a shopper. The other kind collects pages to train a model. Each vendor documents which of its crawlers does which, and each states that blocking the training crawler has no effect on search inclusion. That is the whole decision.
What robots.txt reaches on a Shopify store
Shopify’s robots.txt guide separates two routes. Product data for an activated agentic storefront such as ChatGPT or Microsoft Copilot travels through Shopify Catalog, independently of robots.txt. AI crawlers may also read the store directly on the open web, and that is the route robots.txt governs.
The same page states three limits. Bot management at the network layer is handled for stores on Shopify, and no action is needed there. Rules in robots.txt are directional and advisory, and not every crawler follows them. Blocking AI crawlers at either layer affects only open-web discoverability, and does not stop Shopify Catalog from sending product data to the channels you activated.
Shopify’s data sharing page adds the other half: even where a channel receives no Catalog data, it can still reach product information through web crawling and indexing, where the data might not always be as complete, accurate, or up to date. The open-web route is the one that stays available to every assistant, so its crawler rules matter even for a store that sells through Shopify Catalog.
Two kinds of crawler, by each vendor’s own words
| Vendor | Reads pages for answers or search | Collects pages for training | What the vendor states about blocking |
|---|---|---|---|
| OpenAI | OAI-SearchBot, ChatGPT-User | GPTBot | Sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. ChatGPT-User is not used to decide whether content appears in search, and robots.txt rules may not apply to it. |
| Anthropic | Claude-SearchBot, Claude-User | ClaudeBot | Disabling Claude-SearchBot prevents indexing for search. Disabling Claude-User prevents retrieval when a person asks. Disabling ClaudeBot signals that future material is excluded from training datasets. |
| Perplexity | PerplexityBot, Perplexity-User | none documented | PerplexityBot surfaces and links sites in Perplexity search and is not used to crawl for foundation models. Perplexity-User generally ignores robots.txt because a person requested the fetch. |
Googlebot | Google-Extended token | AI is built into Search, so the robots.txt rule for Googlebot is the control for AI Overviews and AI Mode. Google-Extended does not affect inclusion in Google Search and is not a ranking signal. | |
| Meta | Meta-ExternalFetcher | Meta-ExternalAgent | Meta-ExternalAgent crawls for training foundation models or for improving products by indexing content directly. Meta-ExternalFetcher fetches individual links at a user’s request and may bypass robots.txt. |
| Amazon | Amzn-SearchBot | Amazonbot | Permitting Amzn-SearchBot makes content eligible for search experiences such as Alexa. Amazonbot improves Amazon products and services and may be used to train Amazon AI models. |
Sources for each row: OpenAI’s crawler documentation, Anthropic’s crawler article, Perplexity’s crawler guide, Google’s common crawlers list and AI features page, Meta’s web crawlers page, and Amazon’s Amazonbot page.
Two rows deserve a note. Google-Extended has no user-agent string of its own; crawling happens under Google’s existing agents, and the token only governs whether the content may train or ground Gemini. Meta’s training crawler and its indexing crawler are the same user agent, so a store that blocks Meta-ExternalAgent gives up whatever Meta indexes directly, and Meta’s page does not separate the two uses.
The decision
Keep every crawler in the first column allowed. Each vendor states that this crawler is the one that decides whether your pages can appear in its answers or search results. A Disallow line for OAI-SearchBot or Googlebot removes a route with no offsetting gain.
Decide the second column on consent. OpenAI, Google, and Anthropic each state that their training crawler or token has no bearing on search inclusion, so blocking GPTBot, Google-Extended, or ClaudeBot costs no visibility by the vendor’s own account. The choice is about whether you want your product copy in a training set, and either answer is defensible.
Treat the user-initiated fetchers as outside the file. ChatGPT-User, Claude-User, Perplexity-User, and Meta-ExternalFetcher act when a person asks for a specific page. Their vendors state that robots.txt may not apply to them, so a line for them documents a preference rather than a control.
What a Shopify default looks like
Shopify serves its own robots.txt for a hosted storefront. The theme template reference explains that a robots.txt.liquid file in the theme replaces that output, and that a template is expected to keep Shopify’s default groups. The default file blocks admin, cart, checkout, filtered collections, search, and policy paths so that indexing focuses on unique content. It carries no rule for any AI crawler.
So if your live robots.txt lists AI user agents you did not choose, an app or template wrote them. If those user agents include search crawlers such as OAI-SearchBot, or if the default groups appear with no rules under them, the file has replaced Shopify’s rules rather than extended them. The Answer “Remove a robots.txt.liquid template and restore Shopify’s rules” covers that removal; this Answer covers what the file should say afterward.
How to check
- Open
https://your-store-domain/robots.txtin a browser and read it as a shopper’s assistant would. - Look for a
Disallowline underOAI-SearchBot,Claude-SearchBot,PerplexityBot, orGooglebot. Any such line removes a route. - For every training crawler listed, confirm that the line reflects a choice you made and can name.
- If the store runs a headless or custom storefront behind a firewall or CDN, check that layer separately. Shopify’s page states that it does not support a proxy in front of a hosted store, and a challenge page at the edge stops a crawler before robots.txt is read.
The result is a file you can defend line by line. It states what you allow. It does not prove that any crawler fetched a page, that the page was indexed, or that a product was shown to a shopper.