Cloudflare changes AI crawler rules on September 15 — and Googlebot may be blocked

On September 15, Cloudflare begins applying new controls for AI-related crawlers. In the company announcement, automated traffic is split by purpose: Search for discovery, Agent for actions requested by a user and Training for model development.

The distinction is clear while one crawler performs one job. Googlebot, Applebot and Bingbot complicate it: Cloudflare treats them as multipurpose crawlers and applies the strictest rule selected by the site owner.

A ban on using material for training can therefore deny the same crawler access for ordinary search discovery. This is not a universal block, but the condition is serious enough to inspect before the change date.

For new Cloudflare domains with advertising on their pages, Training and Agent will be blocked by default while Search remains allowed. Existing site owners can choose a different combination in their security settings.

The broad Block AI bots switch is being retired. Cloudflare documentation directs customers towards separate purpose-based policies.

A site owner may want two different outcomes: allow conventional search indexing while refusing model training. When both jobs arrive under one crawler identity, a network filter cannot always pass only the desired activity.

Cloudflare chose the cautious policy: a prohibition wins when purposes conflict. It protects the owner’s stated restriction, but it also gives the owner the risk of losing useful crawling alongside the unwanted use.

TechRadar Pro described the publisher risk: if Googlebot cannot access pages, it cannot discover or refresh them, which may eventually affect search visibility.

It would still be wrong to claim that every Cloudflare site will lose indexing. The domain onboarding date, ads on the page, the former Block AI bots mode and an explicit owner choice all matter. The effective rule for a specific zone must be inspected rather than inferred from one toggle label.

Before the change, an owner can record current settings, inspect multipurpose crawler categories and confirm that required search access remains permitted. Afterwards, Cloudflare logs and Search Console crawl data can show whether Googlebot still retrieves pages.

The new controls make the owner’s choice more precise but not simpler. While search companies use one crawler for several purposes, “do not train on my material” can have a side effect in conventional search. September 15 will test the transparency of the crawlers as much as Cloudflare’s new panel.