Which AI crawlers should I allow, and how do I control training separately?
Keep wanted search routes open and decide training separately. Check live robots.txt, Google’s AI setting, and Cloudflare where you manage it.
· Updated September 21, 2026
- When this applies
- You want assistants to reach public product pages, an app proposes blocking AI bots, or Cloudflare prompts you to review crawler preferences. [9] [10]
- When this does not apply
- Take no crawler action to change an activated Shopify Catalog channel. Its product-data sharing is independent of robots.txt. [9]
- Why this matters
- A broad bot block can close a search route. Cloudflare’s current Training Block also stops mixed-use search crawlers; its Disallow AI Training setting has different effects. [1] [11]
- What to do
- Check public-product rules. Keep wanted search and user-request routes open, decide training separately, and review applicable Google and Cloudflare controls. [1] [6] [11]
- Sources checked
- September 21, 2026
Paid help and free options
- When paid help makes sense
- Paid help can be useful for a specific theme-rule conflict or a firewall or CDN access problem, with affected URLs and a verification step named in advance. [9] [11]
- What’s already free
- Shopify supplies robots.txt and handles network bot management for its hosted storefronts. Reviewing the default does not require an app. [9]
Keep wanted search routes open and decide training separately. A blanket AI-bot block can close routes you wanted assistants to use.
Check the right hostname
Read live /robots.txt for each relevant host. Shopify’s robots.txt guidance separates crawling from Catalog sharing. Shopify manages hosted-store network bots and does not recommend a proxy in front.
Review robots.txt.liquid with the theme maintainer. A separately hosted front end or blog may use its own Cloudflare zone. Cloudflare DNS alone does not prove request filtering.
Separate purposes
| Provider and source | Search route to keep if wanted | Separate training or other-use control |
|---|---|---|
| OpenAI | OAI-SearchBot supports ChatGPT search | GPTBot controls content that may be used for model training |
| Anthropic | Claude-SearchBot supports search indexing | ClaudeBot collects potential training content |
| Perplexity | PerplexityBot supports search | Perplexity says this bot does not crawl for foundation-model training |
Googlebot supports Search | Google-Extended controls documented Gemini training and grounding uses; it has no separate HTTP user agent and does not affect Search inclusion or ranking | |
| Meta | Meta-WebIndexer supports search retrieval | Meta-ExternalAgent covers training and other documented uses |
| Amazon | Amzn-SearchBot supports search experiences such as Alexa | Amazonbot may collect content for AI training |
Check effective public-product rules, including wildcard groups and paths. Cart/admin blocks do not justify removing Shopify defaults. Blocking OAI-SearchBot prevents ChatGPT search-answer inclusion, though navigational links can remain.
User-request fetchers differ: Claude-User honors robots.txt; blocking it prevents query retrieval. ChatGPT-User, Perplexity-User, and Amzn-User may not honor it for user requests. Meta-ExternalFetcher also handles user retrieval. robots.txt is not universal enforced access control.
Check Google’s AI setting
Google’s AI guidance treats Googlebot access as one eligibility condition. The separate Search generative AI control rolled out worldwide August 31, 2026.
In Search Console > Settings > Search generative AI, check the value and inheritance. Exclusion removes covered AI content and links without changing other Search inclusion or ranking. It does not override Merchant Center or control training.
Check Cloudflare only where you manage it
Bot Preference Sync adds generated rules above existing robots.txt. Read the served output and Security settings together.
The September 15, 2026 update distinguishes:
- Disallow AI Training: publishes applicable preferences, keeps Accountable mixed-use Googlebot/Applebot/Bingbot for search, and blocks other training crawlers.
- Block: also blocks mixed-use crawlers. Block on pages with ads blocks them on detected ad pages, even if Search says Allow.
Prior Training Block settings migrate to Disallow AI Training; read the current value before diagnosing a new block. Bing’s no-training preference is not yet conveyed automatically through robots.txt; use its documented operator controls.
Finish with verified rules for affected product URLs and recorded training, Google, and Cloudflare choices. Leave correct settings unchanged. Permission proves no fetch, indexing, quotation, product selection, or sale.