Cloudflare Blocks AI Crawlers by Default on 15 September 2026: What Changes

In short — On 1 July 2026, Cloudflare made a setting available to every customer, free tier included, that sorts crawlers into three categories: Search, Agent and Training. On 15 September 2026, the defaults change — on pages that display ads, Training and Agent will be blocked without any action on your part. The genuinely critical point lies elsewhere: Cloudflare states that a multi-purpose crawler is handled under the most restrictive rule that applies to it, and names Googlebot, Applebot and Bingbot as crawlers that will be blocked by customers who chose to block Training. None of this will show up in your robots.txt: the block is applied at the network layer, upstream of your origin server. Your server logs are the only place where it becomes visible.

On 1 July 2026, Cloudflare published an announcement that changes how crawlers reach a substantial share of the web. The company sits in front of more than a fifth of the world's websites: a change to its defaults is not a publisher-level setting, it moves the baseline for everyone.

This article covers what was announced, what precisely changes on 15 September 2026, and above all what an SEO lead can do to check whether their site is affected. We consistently separate what is documented by Cloudflare from what is interpretation — the distinction matters here, because the heaviest side effect of this announcement involves a crawler nobody wants to lose: Googlebot.

What Cloudflare announced on 1 July 2026

The announcement, published under the name “Content Independence Day”, introduces control by purpose rather than by crawler. Until now, allowing or blocking a crawler meant naming it: one rule for GPTBot, another for ClaudeBot, a third for PerplexityBot. That approach forces you to track every new agent as it appears — and new ones appear regularly.

The new model groups crawlers by what they do with your content. Cloudflare justifies the change by the imbalance between crawling and referred traffic, citing its own 2025 measurements: roughly 14 crawl requests per referred visit for Google, 1,700 for OpenAI, and 73,000 for Anthropic. These are Cloudflare's own figures, measured on its network; we report them as such and have no way to verify them independently.

The underlying reasoning is explicit: the historic bargain between a publisher and a search engine — you crawl me, you send me readers — no longer holds when the ratio reaches several thousand to one. Cloudflare's answer is to make refusal simpler, and to progressively make it the default position.

Search, Agent and Training: what each category covers

Search, in Cloudflare's definition, covers any behavior that collects or indexes your content so it can answer questions about it later. The category includes classic search engines and answer engines that cite their sources. It is the category that can send you readers, and it remains allowed by default.

Agent covers automated behavior acting in real time on a person's behalf to get something done right now. This is the family of ChatGPT-User and of browser-use agents: a request triggered by a human waiting for an answer. These agents build no index; they read a page at the moment someone needs it.

Training covers a crawler taking your content to train or fine-tune a model. That is the documented role of GPTBot and ClaudeBot. The collected content is absorbed into the model's weights: no citation or referral mechanism is attached to it.

This three-way split largely mirrors the distinction we document crawler by crawler in our crawler reference: training on one side, citation on the other. The novelty is not conceptual but operational — it is now a three-position switch rather than a list of names to maintain.

What exactly changes on 15 September 2026

Since 1 July 2026, all three settings have been available to every Cloudflare customer, including on the free tier. Nothing changed automatically on that date: the controls were offered, not applied.

On 15 September 2026, the defaults flip. On pages that display advertising, Training and Agent will be blocked by default, while Search remains allowed by default. Cloudflare also announced a managed robots.txt and an option to prevent crawlers from reaching the portions of a site monetised through ads.

The operative phrase is “by default”. This is not an imposed block: a site owner can change these settings in their security configuration before 15 September, or after. But inaction is no longer neutral — until now, doing nothing meant allowing; from 15 September, for affected sites, doing nothing means blocking.

The multi-purpose crawler trap: Googlebot, Applebot and Bingbot

This is the part of the announcement that deserves the most attention, and the part that has been least reported. A single crawler can belong to several categories: it crawls to index *and* it collects data that may serve training. Cloudflare states that such a multi-purpose crawler is handled under the most restrictive rule that applies to it.

The consequence is spelled out in the announcement: multi-purpose crawlers such as Googlebot, Applebot and Bingbot will be blocked by customers who have selected to block Training. In other words, a publisher who ticks “block training” intending to refuse GPTBot and ClaudeBot may, through that same action, stop serving Googlebot, Applebot and Bingbot.

It is worth being precise about what that implies. Blocking Googlebot is not a loss of AI visibility: it is a gradual exit from Google's index. Pages stop being recrawled, updates go unseen, and ranking decay follows. A site can therefore deindex its own content while believing it merely refused language-model training — a perfectly legitimate intention with a disproportionate side effect.

Cloudflare frames this mechanism as deliberate pressure on AI providers: separate your training crawlers from your search crawlers, or accept that a block covers both. It is a negotiating lever in which publishers are the intermediaries — and, if the setting is misunderstood, the casualties.

Who is affected by the default change

The 15 September flip does not affect everyone equally. According to the announcement, the new defaults apply to new Cloudflare customers, to new sites created by existing customers, and to all existing free-tier customers.

That last point widens the affected population considerably. Cloudflare's free tier is heavily used by brochure sites, blogs, non-profit projects and small organisations — precisely the profile of owners who do not open their security dashboard and will not read this announcement. They are also the least equipped to diagnose a traffic drop six weeks after the fact.

Conversely, an existing paying customer on an already-configured site keeps their current settings until they change them. If you manage several properties, the inventory to run is specific: which ones are on the free tier, which were created recently, and which display advertising.

If you are unsure whether a site sits behind Cloudflare at all, the quickest check is a response header: a request to the domain returns a server: cloudflare header and a cf-ray identifier when it does. Agencies and freelancers inheriting client sites are the most exposed here, because the hosting decision often predates them and nothing in a CMS admin surfaces it. A property migrated to Cloudflare by a previous provider, on the free tier, is exactly the configuration that flips on 15 September without anyone being told.

Why your robots.txt will not tell you that you are blocked

A block applied by Cloudflare happens at the network layer, before the request reaches your origin server. It does not alter the contents of your robots.txt — that file can keep explicitly allowing Googlebot while the crawler receives a refusal at the proxy.

This is a distinction many technical audits miss, because they check the *declaration* rather than the *behavior*. A robots.txt tester reads a file and simulates a rule; it does not replay the request from the crawler's real IP address, through the network layer that will filter it. The two can diverge completely.

The same limitation applies to JavaScript-tag analytics. A crawler blocked at the network layer executes nothing, loads no script, and appears in no analytics report. It leaves a trace in exactly one place: the record of requests actually received, that is, your server logs.

How to spot a blocked crawler in your server logs

Two signals reveal a block. The first is a refusal status code served to a legitimate crawler: a 403 returned to Googlebot, or a repeated 429, is abnormal and warrants immediate investigation. We cover how to read these codes in our guide to 404 and 5xx errors as crawlers see them — the logic is identical for refusal codes.

The second signal is more insidious: outright disappearance. If the block happens at the network layer, the request never reaches your server, and no line is written to your log. You will not see an error — you will see an absence. This is why a baseline measurement taken *before* 15 September is far more valuable than a diagnosis attempted afterwards: without a point of comparison, an absence is indistinguishable from a lull.

In practice the process has three steps. Export your logs over a representative window — our guide explains how to retrieve them depending on your hosting. Record daily request volumes per crawler, verifying authenticity by IP range rather than by user-agent, the latter being trivially forged. Keep that dated record: it becomes your witness.

→ Build your baseline in the Crawl Lab log analyzer: segmentation by crawler and status code, authenticity checks against official IP ranges, daily volumes. Everything runs in your browser, no file is ever uploaded.

What to check before 15 September 2026

The first check is administrative: identify which of your properties sit behind Cloudflare, on which plan, and which display advertising. It is the intersection of those three criteria that determines whether the flip concerns you.

The second is the setting itself. Open your security configuration and look at the state of the three categories. If you care about classic search visibility, the question to settle is not “do I block AI”, but “does my Training setting take with it crawlers I want to keep”. Given how multi-purpose crawlers are handled, those two questions do not have the same answer.

The third is measurement. Take a dated crawl baseline before the flip, and another in the days that follow. It is the only method that lets you attribute a variation to this change rather than to something else — an algorithm update, an outage, or ordinary fluctuation. Our case study on a new site's crawl budget shows how much crawl volumes swing from one day to the next: without a before/after measurement, a 30% drop is not interpretable.

What this announcement does not yet let us measure

We have observed no effect of this change in our own logs, for a simple reason: the flip has not happened yet. Everything above is documented by Cloudflare, not observed by us. We hold to that distinction — a documented fact and an observed fact do not carry the same weight, and conflating them is the most common way to produce confidently wrong SEO content.

Three unknowns remain in particular. The exact definition of a “page that displays advertising” is not spelled out in detail: it is unclear whether it relies on detecting ad scripts, on a customer declaration, or on another method. The classification of a given crawler into a category rests with Cloudflare and can change without publishers being notified. Finally, how AI providers react — separating their agents, or not — will determine the real scale of the effect on multi-purpose crawlers.

We will publish a follow-up measurement after 15 September 2026, using the same protocol as our previous studies and on the same property, to test these announcements against real logs. Until then, the only defensible advice is this: take your baseline now, while it still has comparison value.

Read next

• Server Log Analysis for SEO: The Complete 2026 Guide
• Crawl budget: what Google's new documentation actually changes
• A new site's crawl budget: what our logs show
• 404 and 5xx errors as crawlers see them: finding them in SEO logs
• Getting your server logs: Apache, Nginx, cPanel, CDN