Proxy Worker Health Checks: Avoid Restart Loops During Remote Outages

A proxy worker cannot reach a target, its health check fails, and the supervisor restarts it. The same remote outage then repeats across every replacement. Restarting may destroy useful state without repairing the dependency. For authorized regional jobs, define what each probe proves and which component owns recovery.

Separate three observations

Local liveness asks whether the process can perform a bounded internal operation. Work readiness asks whether it can safely accept the assigned job under your dependency policy. Target acceptance asks whether the authorized target returns the expected business fixture. These results can disagree legitimately. In orchestration systems such as Kubernetes, liveness failure and readiness failure have different consequences; configure your own supervisor's behavior explicitly.

Do not make a remote website the sole restart condition for every worker. A process can remain healthy while a route or target is unavailable. Conversely, a cached successful remote response does not prove the local task loop is making progress. Record local progress and the age of each external observation separately.

Build a bounded probe contract

• Define a local check with no remote dependency, its timeout and expected result. Avoid an endpoint that returns200 while the worker's processing loop is blocked.
• Use an owned small fixture endpoint for the route probe. Check a current fixture marker and the observed route context, not merely TCP connection success. Keep TLS verification active.
• Check the target only with permission and an agreed rate budget. Distinguish authentication, permission, throttling and content mismatch from connection failure.
• Save probe time, duration, state age and sanitized reason. Do not expose credentials or response payloads containing personal data through public health endpoints.

Map results to actions

Local check fails: investigate the worker and use the approved restart policy after its failure threshold. Local check passes but route fixture fails: pause new route-dependent work and compare an approved control path. Route fixture passes but target check fails: handle that target's rules or outage; do not label every proxy exit broken.

If both control and proxy routes fail against the fixture, check the fixture and shared dependencies first. If observations are too old, mark readiness unknown or degraded according to policy rather than carrying forward green indefinitely. Slow startup needs its own initialization window; repeatedly restarting during normal initialization prevents useful work.

Test the decision table on staging

• Create a worker-stall case and confirm the local check detects it without waiting for an external website.
• Simulate an owned route-fixture outage while local processing remains responsive. Verify job intake pauses without uncontrolled restart churn.
• Return an authorized target denial or throttle fixture and confirm the system respects the restriction rather than rotating routes to evade it.
• Restore the dependency and require the documented sequence of fresh successes before resuming. Verify retained checkpoints and uncertain writes are reconciled.

Set timeout, failure and recovery thresholds from observed startup and job requirements. Use a finite retry budget and stagger external probes so many workers do not overload the fixture simultaneously. There is no universal success count that proves a recovered route is safe for every workload.

Use 98IP within the route check

For residential regional jobs, select available settings from 98IP dynamic residential proxies and connect clients through the operation guides. Use the same intended configuration in the fixture probe and job. A different test route cannot certify the assigned route.

Verify actual account options and observed exit. This design does not promise session duration, supervisor compatibility or uptime. External probes consume requests and potentially billed traffic, so include them in your measured operating budget without inventing savings.

Acceptance checklist and FAQ

• Probe ownership, timeout and remediation are explicit; remote denial does not trigger an indiscriminate restart loop.
• Stale green observations expire, recovery uses fresh evidence and paused jobs keep durable checkpoints.
• Probe load is bounded and reports exclude credentials, cookies and private target data.

Does a200 health endpoint prove job readiness? Only if its documented checks cover the job requirements; status alone is insufficient.

Should a target outage restart all workers? Not automatically. A restart must address a confirmed local condition, not merely the absence of a remote response.

Can a static IP replace these checks? It changes route selection, not local liveness or the target's availability.

Test only owned or authorized systems and respect terms, access limits and data protection. The examples are fault-test designs, not measured 98IP availability results.