Cloudflare Separates Search Discovery from AI Training Controls

Cloudflare announced on September 15, 2026 that its AI traffic controls now distinguish search, training and user-directed agent behavior more explicitly. The practical change is a new Disallow AI Training option intended to let a mixed-use crawler remain available for traditional search while conveying that the site's content should not be used for model training.

For authorized web scraping, market research, ad verification and proxy operations, the update is not a new way to bypass controls. It is a reason to make crawler intent observable. A residential or rotating proxy changes the network route; it does not convert training into search, create permission, or override a publisher's stated preference.

What changed on September 15

Cloudflare now treats three behaviors as separate policy dimensions:

• Search builds an index and may return discovery traffic to a publisher.
• Training collects material for model training or fine-tuning.
• Agent fetches content on a person's behalf in real time.

Cloudflare says its general Block setting now applies to mixed-use crawlers, including crawlers that also support search. Blocking them can therefore affect discoverability. The new Disallow AI Training choice is the more precise option when a publisher wants search indexing to continue but does not want training use.

The company also introduced an Accountable designation for operators that provide or commit to publisher controls, reporting and separation between search visibility and training preferences. Cloudflare named Apple, Google and Microsoft among mixed-use crawler operators meeting that framework, while noting that implementation details and timelines differ.

Why proxy teams should care

Many collection systems still classify jobs only by hostname, user agent or exit IP. That is no longer enough for a policy-sensitive workflow. One crawler identity may support several purposes, and the same URL may permit search discovery while refusing training or restricting agents.

Before a request enters a proxy pool, attach an internal purpose label such as search_measurement, authorized_research, ad_verification, user_agent_fetch or model_training. Route and log that purpose separately. If a job has more than one purpose, apply the most restrictive applicable rule until the operator has confirmed a compliant path.

Use the proxy intent and target throttling diagnostic to distinguish route failure from an intentional target-side policy response.

Audit robots and enforcement together

A robots file expresses a preference, while an edge rule can enforce access. Treat them as related evidence, not interchangeable mechanisms. An audit should capture:

1. the robots policy returned to the exact user agent;
2. relevant page-level directives and publisher terms;
3. the edge response, challenge or block outcome;
4. the declared purpose of the job;
5. whether the request used a direct, datacenter or residential route;
6. the timestamp, region and policy version observed.

Do not retry a denied training request through a fresh residential IP. That changes only the source address and can turn a clear refusal into attempted evasion. Pause the job, preserve the evidence and obtain authorization or a supported data path.

Keep search measurements reproducible

Teams measuring search availability should use a dedicated cohort that never shares cookies, session identifiers or retry queues with training or bulk collection jobs. Keep user agent, request headers, location and timing stable enough to compare results. If the platform changes a control, run a small synthetic test before restoring production volume.

For mixed IPv4 and IPv6 routes, follow the global IPv6 readiness workflow so an address-family change is not mistaken for a policy change.

Operational checklist

• Every automated job has one documented primary purpose.
• Mixed-purpose jobs inherit the strictest applicable rule.
• Search, training and agent traffic use separate queues and credentials.
• Robots directives, page metadata and edge responses are recorded together.
• A proxy rotation never retries an explicit policy denial.
• Direct fallback is disabled when a proxy route is required.
• Rate, concurrency and retry budgets remain conservative.
• Policy tests use authorized targets and synthetic data.
• Global, North America, Europe and APAC results are reported separately.
• Changes are reviewed before production traffic resumes.

FAQ

Does Disallow AI Training block search crawling?

Cloudflare presents it as the choice for preserving search discovery while conveying a no-training preference. The separate Block option can stop a mixed-use crawler entirely and may affect search visibility.

Can a residential proxy make a training crawler count as search?

No. Network origin and declared purpose are different. A proxy does not change the job's intent, identity obligations or permission.

Should blocked requests be retried through another IP?

No. An explicit refusal should stop the job. Diagnose configuration, verify authorization and use a publisher-supported route instead of rotating around the decision.

What should teams measure after the change?

Track allowed, disallowed, challenged and ambiguous outcomes by purpose, user agent, region, address family and policy version. Report useful authorized results, not raw request volume.

Compliance note

Run crawler and proxy tests only on services and data you are authorized to access. Honor robots controls, contractual terms, rate limits, privacy duties and regional law. Do not use address rotation, header changes or browser automation to evade an explicit restriction.

Internal source record: Cloudflare, “Have it both ways: stay discoverable in search while disallowing AI training,” September 15, 2026.