Cloudflare BotBase Adds Traceable Verification for Bot Operators

Cloudflare released BotBase for Operators on August 28, 2026, giving bot operators a clearer way to submit, track, correct, and maintain entries in its directory of bots and agents. The update matters beyond AI crawlers. It reflects a broader change in how website defenses distinguish accountable automation from traffic that merely claims a recognizable user agent.

For authorized data collection, search indexing, monitoring, ad verification, and market research, proxy capacity is only one part of the operating model. A crawler also needs a consistent explanation of who runs it, what it does, how it uses content, and how a destination can verify its identity. If those claims drift away from the network path actually used, a larger proxy pool will not restore trust.

What Cloudflare announced

Cloudflare says the new BotBase area is available to all customers. It now separates three operator tasks:

• browsing the directory of bots already tracked;
• submitting a new bot;
• viewing submission history.

Submission history exposes three useful states: waiting for review, accepted, and rejected. A rejected submission includes a reason that the operator can act on. Operators can also edit an existing submission when identification details change, or cancel one that is still waiting for review.

The review path now performs automated checks before routing ambiguous cases for manual review. Cloudflare describes checks for duplicate entries, sufficiently specific user-agent patterns, and whether the claimed verification method works. Depending on the method, the service may fetch a published IP list, confirm reverse DNS, or validate a Web Bot Auth signature.

Public source note: Cloudflare, “BotBase for Operators: A clearer path to joining Cloudflare's directory of bots and agents,” August 28, 2026.

This is not a universal permit to crawl. Individual website owners still decide what traffic they allow. Directory acceptance should therefore be treated as identity evidence, not as consent, contractual authorization, or a promise that every request will pass.

Why proxy-backed crawlers need a separate identity plane

A proxy route changes the network origin visible to a destination. A bot identity describes the accountable operator and intended behavior. Combining the two into one undocumented assumption creates several failure modes:

1. the directory entry references an IP list that no longer contains the active exits;
2. a rotating pool introduces addresses that cannot be attributed to the declared bot;
3. reverse DNS is valid for one region but not for another;
4. a broad user-agent string overlaps unrelated clients;
5. a signature is valid, but a retry path or alternate client stops signing requests;
6. a vendor silently changes upstream supply while the operator's public identity record remains unchanged.

The practical response is to manage identity and routing as two linked but independently testable systems.

Do not infer one plane from another. A successful response does not prove identity, and verified identity does not grant permission.

Build a crawler identity manifest

Create one versioned internal manifest for every production crawler. Keep secrets out of it. The manifest should contain:

• accountable team and operational contact;
• crawler purpose and supported use cases;
• exact user-agent pattern;
• content-use declaration;
• direct or intermediary operating model;
• selected verification method;
• authoritative IP-list or reverse-DNS ownership process;
• signing coverage when signed requests are used;
• approved regions, proxy products, and protocols;
• change owner, review date, and rollback rule.

The manifest is a control document, not marketing copy. It should be precise enough that an engineer can compare it with a live route without guessing.

Audit rotating exits against the declared identity

1. Freeze the expected exit cohorts

For each approved market, record a provider-neutral cohort identifier rather than copying raw credentials or session tokens. Separate residential, ISP, mobile, and data-center products. Record IPv4 and IPv6 cohorts independently because their ownership and reverse-DNS behavior may differ.

2. Sample the path with bounded tests

Run a small authorized test at concurrency one. Capture the exit address only for the minimum time required to compare it with the declared verification source. Hash or tokenize the address in routine logs when the raw value is not operationally necessary.

3. Compare three sets

For each window, calculate:

• declared_but_unseen: addresses or prefixes in the verification source that did not appear;
• seen_and_declared: active exits that match the verification source;
• seen_but_undeclared: active exits that cannot be verified by the claimed method.

The third set is the release blocker. Do not solve it by widening an IP list without validating ownership. Quarantine the cohort, investigate the supply change, and update the manifest only after the evidence is correct.

4. Test fallback paths

Force each approved retry, regional failover, protocol fallback, and client implementation through a non-destructive test. A primary client may sign requests correctly while a legacy worker, browser task, or emergency route sends unsigned traffic.

Use the proxy route leak detection guide to ensure that a failed proxy does not silently switch to a direct connection. Then apply the residential proxy session stickiness test when identity checks depend on a stable session window.

Treat verification changes as releases

Moving an IP-list endpoint, changing reverse-DNS ownership, adopting signed requests, or adding a new proxy supplier can affect how the bot is recognized. Manage each change with a small release process:

1. open a change record that identifies the affected crawler and route cohorts;
2. test the new verification method against a staging or authorized target;
3. confirm every production client follows the same method;
4. update the directory submission and internal manifest together;
5. monitor accepted, rejected, challenge, and block outcomes by cohort;
6. keep the old method only for a defined overlap window;
7. roll back if undeclared exits or unsigned requests appear.

Avoid changing the user agent, proxy supply, verification endpoint, retry policy, and signing code in one release. When several variables move together, a rejection cannot be attributed reliably.

Measure recognition without evading controls

Useful metrics focus on consistency and permission, not on defeating defenses:

• verification coverage by route cohort;
• share of requests using the exact declared user agent;
• signing success rate by client and retry path;
• reverse-DNS forward-confirmation rate where applicable;
• rate of undeclared exits;
• accepted, challenged, blocked, and policy-denied outcomes;
• time from infrastructure change to directory record update;
• response validity and latency for allowed requests.

Do not use rapid IP rotation to search for a route that avoids a block. If a destination rejects the crawler or signals a restriction, stop the automated path and resolve identity, permission, or policy with the site owner or an authorized interface.

Operational checklist

• [ ] Bot purpose, operator, content use, and operating model are documented.
• [ ] The user-agent pattern is specific and consistent across clients.
• [ ] Verification evidence matches the active production routes.
• [ ] IPv4 and IPv6 cohorts are tested separately.
• [ ] Retry and failover paths preserve verification behavior.
• [ ] Undeclared exits are quarantined instead of silently allowlisted.
• [ ] Directory and internal records are updated in the same change.
• [ ] Permission evidence is stored separately from identity evidence.
• [ ] Logs exclude credentials, cookies, signing keys, and unnecessary raw IP history.
• [ ] Blocks and challenges trigger a stop-and-review path.

FAQ

Does a BotBase accepted status authorize crawling?

No. It indicates that the bot is tracked in the directory after review. Website owners still decide what traffic they allow, and the operator must still respect contracts, terms, robots guidance, consent requirements, and rate limits.

Is a recognizable user agent enough?

No. A user-agent string is a claim. Verification may depend on a published IP list, reverse DNS, or a signed-request mechanism, and the live traffic must continue to match that evidence.

Can a residential proxy pool be used for a verified crawler?

Only when the operating model, provider terms, destination rules, and verification method are compatible. Highly dynamic shared exits may be unsuitable for identity schemes based on a stable operator-controlled IP inventory.

Should raw exit IPs be stored indefinitely?

Usually not. Retain the minimum data needed for verification and incident response. Prefer short-lived evidence, hashed cohort tokens, and documented deletion windows.

What should happen after an unexplained rejection?

Compare the declared identity with the exact live route, check user-agent specificity, verification coverage, reverse-DNS or signature behavior, and recent infrastructure changes. Do not respond by increasing request volume or rotating faster.

Compliance note

Use automated collection only for lawful, authorized purposes. Respect destination terms, robots guidance where applicable, content-use preferences, consent requirements, privacy obligations, and reasonable rate limits. Verified identity does not override access controls. Never use proxy infrastructure to impersonate another operator, conceal prohibited activity, or bypass a site's decision to block or challenge a crawler.