How to Test a Proxy Client Circuit Breaker Without Stopping Healthy Markets

A circuit breaker can protect a proxy workload from repeated timeouts and retry storms. A poorly scoped breaker can also stop every market because one city has thin inventory, or keep sending requests after the destination has explicitly asked the client to slow down. The control is useful only when its failure classification, scope and recovery behavior match the workload.

This guide explains how to build an authorized test that proves the breaker opens for persistent faults, rejects work quickly while open, admits only limited probes while half-open and closes only after meaningful recovery. The goal is not to evade destination controls. It is to reduce waste, preserve healthy traffic and make failures visible.

Define what the breaker protects

Do not begin with one global counter. Choose a key that matches the likely failure domain. A practical key may combine:

• proxy provider or gateway;
• country, region or city pool;
• destination hostname or approved endpoint class;
• credential or account scope;
• request operation and idempotency class.

If all markets share one breaker, a failure limited to a small regional pool can halt healthy country-level traffic. If every IP gets its own breaker, rotation can hide a provider-wide outage and create thousands of useless state machines. Start with the narrowest stable operational boundary and aggregate signals upward for visibility.

Classify outcomes before counting failures

Not every unsuccessful response should trip the same circuit.

A transport success is not necessarily a valid result, and a destination rejection is not necessarily a broken proxy. Preserve the proxy status, destination status, timing and content-validation result separately.

Model closed, open and half-open behavior

In the closed state, ordinary authorized requests flow and recent eligible failures are measured over a time or count window. Open the circuit only when the minimum sample and threshold are both met; one isolated timeout should not stop a healthy pool.

In the open state, fail quickly before allocating a connection or consuming a proxy request. Queueing thousands of calls behind an open breaker simply moves the outage into memory and latency. Return a typed error that identifies the breaker key, state and next eligible probe time without exposing credentials.

In the half-open state, admit a small, fixed number of probes. A recovering gateway can fail again if the whole backlog is released at once. Close only after the probe set meets the same business-validity rules used in normal traffic. If a probe fails with an eligible fault, reopen and increase the cool-down within a bounded policy.

Build the test matrix

Use a proxy test endpoint and destination that you own or have explicit permission to exercise. Prepare deterministic fixtures for:

1. healthy response with required content;
2. connection refusal or reset;
3. delayed response beyond the client timeout;
4. proxy authentication failure;
5. destination 429 with a retry instruction;
6. destination 5xx;
7. HTTP 200 with deliberately invalid content;
8. location pool with no available exit;
9. recovery after a controlled fault window.

Run each fixture against two breaker keys: one affected market and one healthy control market. Keep concurrency, timeout, session policy and request volume fixed. Record state transitions, allowed calls, fast rejections, queued work, retries, recovery probes, valid outcomes and total bytes.

Use the proxy timeout budget guide to ensure the attempt deadline fits inside the operation deadline. Pair it with retry amplification controls so retry layers do not multiply each other.

Verify failure isolation

The affected circuit should open without changing unrelated breaker keys. During the open window, confirm that:

• healthy markets continue within their latency and success thresholds;
• no failed request silently goes direct;
• no location selector silently broadens;
• no credential or session crosses into the fallback route;
• the queue remains bounded;
• dashboards distinguish fast breaker rejection from network failure.

If an alternative provider or region is permitted, treat it as a separate route with its own capacity and policy. Fallback is not automatically success: validate location, session, content and cost again. Use the proxy failover recovery drill for the complete route-change test.

Tune thresholds with production-shaped evidence

Three values interact: minimum sample size, failure or slow-call threshold, and open duration. A low threshold reacts quickly but can oscillate on small pools. A large window is stable but may waste requests before opening. A long cool-down protects a struggling dependency but delays recovery; a short one can flood it with probes.

Tune with replayed timing distributions or a controlled load test, not intuition. Measure:

• eligible failure rate and slow-call rate;
• p50, p95 and p99 latency;
• requests prevented while open;
• false-open events on healthy controls;
• time from recovery to stable closed state;
• half-open probe volume and outcome;
• queue depth and dropped work;
• accepted business outcomes per proxy byte and per unit cost.

Use consecutive successes only when they represent independent, meaningful probes. Ten cached health checks through one reused connection may not prove that new sessions, location selection and destination validation have recovered.

Make retries breaker-aware

Retry transient faults only when the request is safe and the total retry budget allows it. Add bounded exponential backoff and jitter. When a breaker is open, retry code must stop immediately instead of repeatedly receiving the same fast rejection.

Honor destination instructions such as Retry-After. Never rotate exits to evade rate limits, access controls or a stop signal. For non-idempotent operations, do not retry unless an idempotency mechanism makes the result unambiguous.

Place the breaker where it can see the full outcome. A low-level connection breaker cannot know that an HTTP 200 page failed content validation; an application-only breaker may miss TCP saturation. Keep the signals separate, then let an explicit policy decide which breaker they affect.

Acceptance checklist

• Breaker keys map to real provider, pool, market and destination failure domains.
• Failure categories are explicit and do not treat all 4xx, 5xx and timeouts alike.
• The minimum sample prevents one error from opening a low-volume circuit.
• Open state rejects quickly before connection and billing work.
• Queues, retries and memory remain bounded.
• Half-open admits a limited number of meaningful probes.
• Healthy markets remain unaffected during a local fault.
• Direct bypass and silent geographic broadening are impossible.
• Fallback routes revalidate location, content, session and cost.
• State transitions, reason codes and next-probe times are observable.
• Manual override is audited and expires automatically.
• Recovery closes only after business-valid results, not merely a TCP connection.

FAQ

Should a 429 open the proxy breaker?

Usually it should activate a destination-specific rate-limit policy and honor Retry-After, not mark the proxy pool unhealthy. Never rotate proxies to bypass the destination's limit.

Should every proxy exit have its own breaker?

Usually no. Per-exit state is unstable in rotating pools and can hide a shared outage. Key breakers by a stable operational failure domain, then retain exit-level evidence for diagnosis.

Can a health endpoint close the circuit?

It can support recovery, but it may not prove that the real workflow, location selector, session and content are valid. Use limited application-representative probes when authorization and cost permit.

What if only one city has no inventory?

Open or throttle the city-pool circuit, not the entire provider. Fail clearly or use only an explicitly approved broader target. Keep other markets independent.

Compliance

Test only infrastructure and destinations you own or are authorized to access. Respect provider contracts, destination terms, rate limits, robots guidance, privacy obligations and regional law. Circuit breakers must reduce load and contain faults; they must not be used to evade controls, disguise prohibited automation or continue after a destination signals that requests should stop.