Cloudflare Gives Bot Operators Status Tracking and Editable BotBase Submissions

Cloudflare introduced BotBase for Operators on August 28, 2026, giving bot and agent operators a dashboard for submitting, tracking and updating their identity records. The change matters to teams that run authorized web crawlers, search tools, monitoring services and user-directed agents because it turns bot registration from a one-time form into an ongoing operational record.

The new workflow does not create a universal permission to crawl. Website owners still decide which automated traffic they allow. It does, however, raise the standard for operators who want their traffic to be identifiable, accurately described and easier for site owners to evaluate.

What changed

The operator experience now sits in the BotBase area of the Cloudflare dashboard and includes three practical capabilities.

Submission history and visible status

Operators can see whether a submission is waiting for review, accepted or rejected. Rejected records include a reason and a path to correct the problem. Accepted records can show classification adjustments made during review.

This is more than a convenience. A status history creates evidence for release reviews and incident response. A crawler team can show when an identity was submitted, what verification method it used and whether a later infrastructure change made that information stale.

Editable and cancellable submissions

Operators can update an existing submission when identification details change. A waiting submission can also be cancelled. This addresses a common lifecycle problem: user-agent patterns, published IP ranges, reverse-DNS names and authentication methods do not remain static forever.

Treat editing as controlled change, not routine housekeeping. A modified identity record should be linked to the deployment that caused the change, reviewed before traffic is shifted and verified after release.

A behavior and content-use declaration

The updated intake asks operators to describe three dimensions separately:

1. What the bot does — such as search indexing, user-directed action, data collection, training or SEO support.
2. How it uses content — for example search, reference or model training.
3. Who directs the request — the direct operator or an intermediary carrying traffic for another product.

That separation reduces the ambiguity of a single label such as “crawler.” A service that fetches a page for a user request is operationally different from a system that stores large corpora for training, even when both use HTTP automation.

Verification is becoming an engineering dependency

Cloudflare says its rebuilt review path can automatically check whether a submission duplicates an existing bot, whether the user-agent pattern is specific enough, and whether the claimed verification method works. Depending on the method, that can include fetching a published IP list, confirming reverse DNS or validating a Web Bot Auth signature.

Crawler identity therefore needs an owner, tests and monitoring just like an API integration.

Create an identity inventory containing:

• stable bot name and accountable operator;
• narrowly scoped user-agent pattern;
• declared behaviors and content uses;
• direct-versus-intermediary role;
• verification method and its public evidence;
• production egress ranges or signing-key references;
• last verification date and next review date;
• change owner, rollback plan and incident contact.

Do not put proxy credentials, private keys, cookies or access tokens in that inventory. Store references to protected secrets, not the secrets themselves.

What this means when a crawler uses proxies

Proxies can provide regional routing, capacity separation and session control for legitimate collection. They must not make the crawler's declared identity misleading.

If verification depends on published IP ranges, compare the declared range with the egress actually used by each production route. A rotating residential pool may not be compatible with an identity model that promises a small static allowlist. If verification uses reverse DNS, monitor forward and reverse consistency. If the crawler uses signed requests, protect signing keys and confirm that every egress path preserves the required headers.

Keep identity and transport as separate control planes:

Rotating an exit after a policy denial does not repair identity or authorization. It can turn a diagnosable mismatch into behavior that appears evasive.

A release checklist for identifiable crawlers

Before changing crawler infrastructure, run this sequence.

1. Freeze the declared identity

Record the current user agent, behaviors, content uses, operator role and verification method. Export only non-secret configuration suitable for audit.

2. Map every production egress path

List direct connections, datacenter proxies, residential proxies, IPv4 and IPv6 paths, regional gateways and failover routes. Identify which paths can satisfy the chosen verification method.

3. Test verification independently

Confirm the published IP document is reachable, reverse DNS is correct or signed requests validate. Run this test before crawling any third-party site.

4. Reconcile site permissions

Read the target's terms, robots directives, content-use signals and rate limits. A verified identity is evidence about who is calling; it is not consent for every purpose.

5. Run a bounded canary

Use low concurrency, a small approved URL set and explicit stop conditions. Record request time, route, identity version, policy decision and application result.

6. Monitor for identity drift

Alert when production emits an undeclared user agent, uses an unverified egress path, loses a signature header or changes content purpose without an approved declaration update.

7. Keep the directory record current

Update the submission when verification evidence or declared behavior changes. Preserve the reason, reviewer and deployment reference.

Metrics that reveal operational quality

Do not reduce the program to acceptance rate. Track:

• percentage of production requests carrying the approved identity version;
• percentage using a currently verifiable route;
• mismatches between declared and observed content purpose;
• policy denials by site and reason;
• rate-limit events and retry-budget exhaustion;
• time from identity change to directory update;
• useful-result rate and cost per useful result;
• incidents caused by stale IP, DNS or signing evidence.

Segment results by region and route. A global success rate can hide one proxy gateway that is outside the declared identity boundary.

What teams should not conclude

Accepted in a directory does not mean globally allowed. Each website retains control over automated access.

A verified user agent does not prove ethical data sourcing. Purpose, consent, minimization, retention and downstream use still require governance.

An IP allowlist is not a permanent identity. Egress supply changes, and shared addresses can create ambiguity.

A proxy rejection is not automatically an IP-quality failure. Identity, permission, rate and application policy must be diagnosed separately.

Changing user agents or exits to defeat a site's decision is not a remediation plan. Correct the declaration, obtain permission or stop the collection.

FAQ

Should every internal scraper be submitted to BotBase?

No. A crawler that stays inside systems you own has a different exposure. Submission is relevant when an operator wants public sites and Cloudflare customers to identify and evaluate its automated traffic. Decide based on scope, policy and the verification model.

Is Web Bot Auth required?

Cloudflare describes several possible verification methods, including published IP lists, reverse DNS and Web Bot Auth. Choose a method that matches the crawler's architecture and can be maintained across every production route.

Can a rotating residential proxy crawler be verified by IP range?

Only when the operator can truthfully publish and maintain the relevant egress evidence. Highly dynamic third-party supply may not fit a static IP identity model. Do not publish ranges you do not control or claim exclusivity you cannot prove.

Does verification replace robots.txt or content-use signals?

No. Identity answers who the caller is. Site directives and contractual terms answer what the caller may do. Both must be satisfied.

Pair this update with 98IP's guides to audit proxy-pool network concentration, test residential proxy session stickiness, and run a defensible proxy pilot.

Compliance note

Operate crawlers only on systems and content you are authorized to access. Respect website terms, robots directives, content-use signals, rate limits, privacy obligations and applicable law. Maintain truthful identity declarations, minimize collected data and stop when a site denies the requested use. Proxies are a transport tool, not a way to conceal responsibility or override a publisher's decision.

Source note: Cloudflare, “BotBase for Operators: A clearer path to joining Cloudflare's directory of bots and agents,” published August 28, 2026. The source URL is retained only in 98IP's internal operations record under the website's zero-external-link policy.