Cloudflare says it now manages my robots.txt. Does that touch my Shopify store?
Cloudflare now writes an AI-bot block into a zone's robots.txt. Shopify serves the storefront file, so this reaches only a zone you own.
- When this applies
- Cloudflare prompted you to confirm AI bot preferences, or a block of crawler rules you did not write sits above your own rules on a domain you route through Cloudflare. [1]
- When this does not apply
- Take no Cloudflare action for a storefront Shopify hosts. Shopify serves that robots.txt from the store's primary domain, so a Cloudflare zone setting is not in the path. [3] [1]
- Why this matters
- A dashboard setting can now write crawler rules into robots.txt with no edit to the file, and Cloudflare changes its defaults for Agent and Training crawlers on September 15, 2026. [1] [2]
- What to do, or not do
- Name the system that serves robots.txt for every store address. Where you own the Cloudflare zone, open its Security settings, read the generated block, and set Search, Agent, and Training before September 15, 2026. [1] [2]
- Sources checked
- September 4, 2026
- Changed
- August 21, 2026
When a third party is worth paying for, and what you do not have to buy
- When paid help is worth it
- Paid help is worth it only when somebody else runs the Cloudflare zone in front of a headless or custom store surface, and you need the three category settings stated in writing. [1]
You may have received a Cloudflare prompt about AI bot preferences, or found a block of crawler rules at the top of a robots.txt nobody on your team edited. Before you change anything, establish which system serves the file you are looking at. Cloudflare and Shopify serve different addresses, and a store often has both.
For a storefront Shopify hosts, Shopify serves robots.txt and a Cloudflare setting cannot reach it. Where an address a shopper or an agent can reach runs through a Cloudflare zone you own, the setting is yours to read and decide.
Name the writer for each address
Shopify serves robots.txt at the root of the store’s primary domain. The one merchant-side override is a robots.txt.liquid template in the theme, and deleting that template restores Shopify’s default file.
That leaves three possible writers, on different addresses:
| Address | Who writes robots.txt |
|---|---|
| The storefront Shopify hosts | Shopify, unless a robots.txt.liquid template in the theme replaced it |
| A headless storefront, a landing-page domain, or a blog subdomain on your own Cloudflare zone | Cloudflare’s generated block above whatever the origin serves |
If the file belongs to the Shopify storefront and does not match Shopify’s rules, the writer is a theme template rather than a CDN. The Answer “Remove a robots.txt.liquid template and restore Shopify’s rules” covers that case and this one does not.
What Cloudflare changed on August 21, 2026
Cloudflare announced Bot Preference Sync, which keeps a zone’s robots.txt reflecting the Search, Agent, and Training preferences already set on the zone dashboard. It is available on every plan from Free to Enterprise and can be turned on or off at any time.
Four details decide whether it changes anything for a store:
- The generated block is prepended to an existing robots.txt between
BEGINandENDmarkers, so anyDisallowdirectives already in the file are kept below it. - The user-agent list inside the block is periodically refreshed from the bots Cloudflare tracks in its directory, so the file can change without anyone editing it.
- The sync is on by default for new customers. Existing customers using the older managed robots.txt file are prompted to review and confirm their preferences.
- The sync reflects category-wide policy and does not read individual custom rules. Cloudflare’s stated path for a finer policy is to turn the sync off and maintain the file yourself.
Cloudflare’s own framing separates the two business cases: an ad-supported publisher may want to stay in search while keeping articles out of model training, while an online store may want everything crawled so its products surface when a shopper asks an assistant for a recommendation.
The defaults change on September 15, 2026
Cloudflare’s AI traffic options post sets new defaults on that date. For all new domains onboarding to Cloudflare, the Training and Agent categories are blocked by default on pages that display ads, and Search stays allowed.
The ad-page default is unlikely to select a product page. The part that reaches an existing zone is the second change: from the same date, a crawler that combines Search with Training is judged on all of its behaviours under the most restrictive applicable rule. Cloudflare names Googlebot, Applebot, and BingBot, and states that those crawlers are blocked for customers who chose to block Training, through either the current options or the older Block AI bots service. An owner who wants no change can mark that in Security settings at any point before September 15, 2026.
Agent is the category that carries a shopping agent acting for a person in real time. On a zone where Agent is blocked everywhere, that traffic stops at the edge, and complete product data behind it changes nothing.
What to do if you own a Cloudflare zone
- List every address a shopper or an agent can reach: the storefront, a headless front end, landing-page domains, and any blog or help subdomain.
- For each address, name the system that serves robots.txt. The addresses Shopify serves need no Cloudflare work.
- For each Cloudflare zone you own, open its Security settings, read the generated block, and confirm the Search, Agent, and Training choices against how you want the store found.
- Where a case-by-case rule matters more than a category policy, turn the sync off and maintain the file directly.
The observable result is a written note of who serves robots.txt for each address, plus a confirmed setting for every Cloudflare zone you own. That record states what you allow. It does not prove that an agent reached your products, that a product was selected, or that anything sold.