Cloudflare AI Bot Defaults Change September 15: Audit Proxy Automation by Intent

Cloudflare says that, beginning September 15, 2026, newly onboarded domains will receive different defaults for three AI-traffic behaviors. Search remains allowed by default, while Agent and Training traffic will be blocked by default on pages that display ads. Cloudflare also says multi-purpose crawlers can be governed by the most restrictive applicable behavior.

For teams running authorized web scraping, market research, ad verification or browser automation, the operational lesson is not to search for a bypass. It is to stop treating every automated request as the same workload. A proxy route changes where traffic exits; it does not change the purpose of a request, create permission or override a site's policy.

What the announcement does and does not say

The new default applies to domains newly onboarding to Cloudflare from the effective date. It does not mean every existing Cloudflare zone will immediately block every automated request. Site owners can choose their settings, and enforcement depends on the zone, page and traffic classification.

The three behaviors answer different questions:

• Search indexes content with an expectation of discovery or referral value.
• Agent performs real-time activity on a person's behalf.
• Training collects content for model training or fine-tuning.

One crawler may serve more than one purpose. Cloudflare states that the most restrictive applicable setting can control multi-purpose crawlers, so a request that also has a Training role may not receive the same treatment as a Search-only request.

Do not infer policy from one successful fetch. Access can differ by hostname, page type, user agent, session state, market, timing and the site's own configuration.

Build an intent register before testing

Create one row for each automation job and assign an accountable owner. At minimum record:

job_alias
business_purpose
declared_behavior
data_owner
authorization_reference
target_scope
allowed_markets
request_method
identity_or_user_agent
proxy_route_class
retention_period
stop_condition

“Data collection” is not a sufficient purpose. A price-monitoring job, a search-indexing crawler, an ad-rendering check and a user-directed browser assistant have different permissions, evidence needs and failure consequences.

If a job mixes purposes, split it where practical. Separate collection used for search discovery from collection used for training. Separate an automated agent action from passive measurement. This makes policy, rate limits and audit trails easier to apply honestly.

Create a paired direct-and-proxy baseline

Use only destinations and accounts you own or are explicitly authorized to test. For each approved fixture, run a small direct control where policy permits, then repeat through the intended proxy route without changing the browser build, request headers or test data.

Capture:

• final status and redirect chain;
• response-body digest and semantic completion marker;
• challenge or interstitial detection;
• declared and observed market;
• browser and automation build;
• route class and sanitized session alias;
• time to first byte and time to useful result;
• first-attempt result and any bounded retry;
• screenshot or network trace when permitted.

A direct success plus proxy failure is evidence of a route-dependent difference, not proof of deliberate blocking. A direct and proxy failure at the same step points first to page policy, application state or shared client behavior.

Test page classes separately

Because the announced defaults distinguish pages that display ads, do not test only a homepage. Build a small authorized matrix that includes:

1. a non-ad informational page;
2. an ad-supported page;
3. a login-free public fixture;
4. an authenticated fixture if the workflow is authorized;
5. a page whose content is loaded after browser execution;
6. a deterministic endpoint used only for network controls.

Record the page class as metadata. Do not store or reproduce ad identifiers, cookies or personal data unless necessary and approved. If the page owner provides a preferred API, feed or test environment, use it instead of browser collection.

Validate four layers, not just HTTP 200

An allowed response can still be useless, and a denied response can still return a normal status. Validate four layers:

1. Network path

Confirm DNS mode, proxy authentication, TLS validation, address family and destination reachability. Keep SOCKS5 local-DNS and remote-DNS tests separate. Never weaken certificate checks to convert a transport failure into a pass.

2. Policy outcome

Classify allow, explicit deny, rate limit, challenge, consent gate and unknown interstitial. Store a sanitized fingerprint of the result rather than a full page when that is enough.

3. Application completion

Check the specific element, structured value, render state or workflow completion that the job needs. A branded challenge page with status 200 is not useful success.

4. Data integrity

Validate schema, language, currency, freshness, pagination and response digest. If a regional proxy changes localization, that should be an expected assertion, not an unexplained difference.

Use regional proxies as observability controls

For Global, North America, Europe and APAC coverage, sample only markets that the business case and authorization require. Keep concurrency low and use the same test sequence for each route.

Compare outcome distributions rather than hunting for one passing exit. Useful signals include:

• policy outcome by declared behavior and page class;
• useful-result rate by market and route class;
• challenge and rate-limit rate;
• localization correctness;
• retries per useful result;
• p50 and p95 completion time;
• unexpected exit rotation;
• bytes and requests per validated record.

If a market-specific path fails, pause that route and investigate. Do not rotate aggressively until an address happens to pass; that turns diagnosis into evasion and destroys the evidence.

Prepare a safe response to policy denials

When a job encounters a policy boundary:

1. stop the affected job or page class;
2. preserve a minimal sanitized result and configuration manifest;
3. verify the declared purpose, authorization and site terms;
4. check whether an official API, feed or owner-provided method exists;
5. contact the data owner through an approved channel when clarification is necessary;
6. resume only after policy and technical owners approve the path.

Do not hide automation identity, replay challenges, alter fingerprints or cycle residential exits to force access. A proxy service is infrastructure, not consent.

Rollout checklist for September 15

• [ ] Every automation job has one declared primary purpose.
• [ ] Mixed Search, Agent and Training jobs are split or explicitly reviewed.
• [ ] Authorization, owner, market scope and retention are documented.
• [ ] Direct and proxy controls use identical client settings.
• [ ] Ad-supported and non-ad page classes are tested separately.
• [ ] HTTP status, policy outcome, application completion and integrity are distinct fields.
• [ ] Browser, proxy, DNS and address-family versions are captured.
• [ ] Regional tests use bounded concurrency and approved markets.
• [ ] Challenges and denials stop the job instead of triggering route rotation.
• [ ] Secrets, cookies and personal data are excluded from evidence.
• [ ] An API or owner-approved fallback is documented.
• [ ] Alert thresholds are based on useful results, not raw request success.

FAQ

Will every Cloudflare-protected site block proxy traffic on September 15?

No. The announcement concerns new defaults for newly onboarded domains and specific AI-traffic behaviors. Individual zone settings and page conditions still matter.

Does changing proxy location change a job from Training to Search?

No. Purpose is determined by what the automation does and how the data is used. Network location does not change intent or authorization.

Is HTTP 403 enough to identify the policy?

No. It is a denial signal, but it does not by itself identify the governing rule. Use owner-provided information and approved observability; do not reverse-engineer controls to bypass them.

Should a team retry challenges through a new residential IP?

No. Stop, classify the result and review permission and the approved access method. Address cycling can violate policy and makes incident evidence unreliable.

What is the most useful success metric?

Validated useful results per approved request budget. It combines access, completion, integrity, latency and cost without rewarding noisy retries.

Compliance and safe operation

Use only authorized destinations, identities, data and markets. Respect robots directives, access controls, privacy obligations, intellectual-property rights, platform terms and rate limits. Disclose automation where required, minimize retained evidence and never treat a proxy as a way to conceal prohibited behavior.

Continue with the proxy response-integrity test, idle-timeout and keep-alive test, and Node.js proxy-client canary plan.

Source note: Cloudflare, “Your site, your rules: new AI traffic options for all customers,” July 1, 2026; Cloudflare product changelog, reviewed September 13, 2026.