# Which AI crawlers should I allow, and how do I control training separately?

> Keep wanted search routes open and decide training separately. Check live robots.txt, Google’s AI setting, and Cloudflare where you manage it.

September 5, 2026 · Updated September 21, 2026 · Sources checked September 21, 2026

https://app.ai-commerce-ready.com/blog/which-ai-crawlers-to-allow-in-robots-txt/

## When this applies

You want assistants to reach public product pages, an app proposes blocking AI bots, or Cloudflare prompts you to review crawler preferences. [9][10]

## When this does not apply

Take no crawler action to change an activated Shopify Catalog channel. Its product-data sharing is independent of robots.txt. [9]

## Why this matters

A broad bot block can close a search route. Cloudflare’s current Training Block also stops mixed-use search crawlers; its Disallow AI Training setting has different effects. [1][11]

## What to do

Check public-product rules. Keep wanted search and user-request routes open, decide training separately, and review applicable Google and Cloudflare controls. [1][6][11]

## When you’re done

You have checked public product access for the search crawlers you want, recorded the training choices, and verified the applicable Google and Cloudflare settings. [9][6][11]

## What this cannot prove

An allowed robots.txt rule removes one access restriction. It does not prove successful fetching, indexing, quotation, product selection, or sales. [9][1]

## Paid help and free options

- When paid help makes sense: Paid help can be useful for a specific theme-rule conflict or a firewall or CDN access problem, with affected URLs and a verification step named in advance. [9][11]
- What’s already free: Shopify supplies robots.txt and handles network bot management for its hosted storefronts. Reviewing the default does not require an app. [9]

**Keep wanted search routes open and decide training separately.** A blanket AI-bot block can close routes you wanted assistants to use.

## Check the right hostname

Read live `/robots.txt` for each relevant host. Shopify's [robots.txt guidance](https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt) separates crawling from Catalog sharing. Shopify manages hosted-store network bots and does not recommend a proxy in front.

Review `robots.txt.liquid` with the theme maintainer. A separately hosted front end or blog may use its own Cloudflare zone. Cloudflare DNS alone does not prove request filtering.

## Separate purposes

| Provider and source | Search route to keep if wanted | Separate training or other-use control |
|---|---|---|
| [OpenAI](https://developers.openai.com/api/docs/bots) | `OAI-SearchBot` supports ChatGPT search | `GPTBot` controls content that may be used for model training |
| [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | `Claude-SearchBot` supports search indexing | `ClaudeBot` collects potential training content |
| [Perplexity](https://docs.perplexity.ai/guides/bots) | `PerplexityBot` supports search | Perplexity says this bot does not crawl for foundation-model training |
| [Google](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) | `Googlebot` supports Search | `Google-Extended` controls documented Gemini training and grounding uses; it has no separate HTTP user agent and does not affect Search inclusion or ranking |
| [Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) | `Meta-WebIndexer` supports search retrieval | `Meta-ExternalAgent` covers training and other documented uses |
| [Amazon](https://developer.amazon.com/amazonbot) | `Amzn-SearchBot` supports search experiences such as Alexa | `Amazonbot` may collect content for AI training |

Check effective public-product rules, including wildcard groups and paths. Cart/admin blocks do not justify removing Shopify defaults. Blocking OAI-SearchBot prevents ChatGPT search-answer inclusion, though navigational links can remain.

User-request fetchers differ: `Claude-User` honors robots.txt; blocking it prevents query retrieval. `ChatGPT-User`, `Perplexity-User`, and `Amzn-User` may not honor it for user requests. `Meta-ExternalFetcher` also handles user retrieval. robots.txt is not universal enforced access control.

## Check Google's AI setting

[Google's AI guidance](https://developers.google.com/search/docs/appearance/ai-features) treats Googlebot access as one eligibility condition. The separate [Search generative AI control](https://support.google.com/webmasters/answer/16908024) rolled out worldwide August 31, 2026.

In Search Console > Settings > Search generative AI, check the value and inheritance. Exclusion removes covered AI content and links without changing other Search inclusion or ranking. It does not override Merchant Center or control training.

## Check Cloudflare only where you manage it

[Bot Preference Sync](https://blog.cloudflare.com/bot-preference-sync/) adds generated rules above existing robots.txt. Read the served output and Security settings together.

The [September 15, 2026 update](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) distinguishes:

- **Disallow AI Training:** publishes applicable preferences, keeps Accountable mixed-use Googlebot/Applebot/Bingbot for search, and blocks other training crawlers.
- **Block:** also blocks mixed-use crawlers. **Block on pages with ads** blocks them on detected ad pages, even if Search says Allow.

Prior Training Block settings migrate to Disallow AI Training; read the current value before diagnosing a new block. Bing's no-training preference is not yet conveyed automatically through robots.txt; use its documented operator controls.

Finish with verified rules for affected product URLs and recorded training, Google, and Cloudflare choices. Leave correct settings unchanged. Permission proves no fetch, indexing, quotation, product selection, or sale.

## Sources

1. [OpenAI crawlers and user agents](https://developers.openai.com/api/docs/bots)
2. [Anthropic crawler article](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
3. [Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
4. [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
5. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
6. [Search generative AI control](https://support.google.com/webmasters/answer/16908024)
7. [Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
8. [Amazonbot and Amzn-SearchBot](https://developer.amazon.com/amazonbot)
9. [Editing robots.txt.liquid](https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt)
10. [Bot Preference Sync](https://blog.cloudflare.com/bot-preference-sync/)
11. [Search discovery and AI training controls](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/)