Proxy DNS TTL and Failover Validation: Prevent Stale Gateway Outages

Internet routing nodes showing DNS cache expiry and a healthy proxy failover path

A proxy gateway can be healthy while clients still reach its retired address. The usual cause is not the authoritative record itself, but a longer-lived cache, a negative answer, a resolver failure cache, or an existing connection pool. This guide turns those hidden states into measurable cutover gates.

Why the authoritative TTL is only the starting point

Positive answers cache an address for a defined lifetime. Negative answers, such as a missing name, have their own caching behavior. Temporary resolution failures may also be cached. Some recursive resolvers can serve a stale answer during an upstream outage. Finally, applications may reuse established sockets long after DNS would otherwise be refreshed.

RFC 2308 defines negative caching, RFC 8767 describes serving stale DNS data, and RFC 9520 clarifies caching requirements for resolution failures. A reliable proxy migration must test all four behaviors rather than assume the displayed TTL controls the entire path.

Map the complete resolution path

Inventory every layer that can remember an answer:

  • operating-system and runtime caches;
  • local stub and recursive resolvers;
  • browser, JVM, container, or service-mesh behavior;
  • local DNS versus proxy-side hostname resolution;
  • separate A and AAAA records;
  • HTTP keep-alive, HTTP/2, QUIC, and connection pools;
  • primary and fallback proxy gateway names;
  • fail-closed controls for restricted destinations.

Record the effective lifetime at each layer. The longest observed lifetime—not the shortest configured TTL—sets the safe overlap window.

Build a safe test zone

Use a domain and two harmless endpoints you control. Endpoint A represents the current proxy gateway; endpoint B represents the replacement. Return a unique, non-secret marker from each endpoint so the client path is observable. Do not test by disrupting a production hostname or by sending traffic to systems you do not own.

Eight validation steps

1. Establish a clean baseline

Flush only the documented test caches, start a new client process, resolve the name, and make several requests. Capture resolver, address family, selected endpoint, lookup time, connection-reuse state, and timestamp.

2. Measure positive-cache expiry

Point the test record to endpoint A, wait for propagation, then change it to B. Probe at fixed intervals from every representative runtime and region. The pass condition is that new connections move to B within the approved window.

3. Test negative caching

Query a deliberately absent test name, create it afterward, and measure how long each client continues to receive the negative result. Include both local-resolution and proxy-side-resolution modes.

4. Test resolution-failure caching

In an isolated test resolver, simulate a temporary upstream resolution failure. Restore service and measure recovery. Repeated retries must remain bounded and must not amplify load.

5. Detect stale-answer service

Make the authoritative test source temporarily unavailable while a previously valid answer exists. Determine whether the recursive resolver serves stale data, for how long, and with what operational consequence. Treat this as an availability feature with a migration cost.

6. Validate A and AAAA independently

Change IPv4 and IPv6 test records separately. Confirm both families converge, health checks cover both, and a broken family cannot silently absorb a large share of traffic.

7. Drain connection pools

Repeat the cutover with long-lived HTTP/2, QUIC, and keep-alive connections. DNS convergence does not move an established connection. Define maximum connection age and a controlled drain procedure.

8. Prove fail-closed behavior

For workloads that must use an approved proxy, remove all permitted endpoints in the test environment. The client should stop with a clear error, not fall back to a direct route or an unapproved gateway.

A safer migration timeline

Lower the TTL at least one full previous-TTL interval before the planned change. Keep both old and new gateways available during the overlap. Change the record, watch new connections converge, drain old pools, and retire the old gateway only after the longest measured cache and connection lifetime has passed.

Acceptance gates

  • New connections converge within the documented window in every target region.
  • Negative and temporary-failure caches recover within their limits.
  • Stale-answer behavior is known and included in the overlap plan.
  • A and AAAA paths both pass availability and identity checks.
  • Retry volume stays bounded during resolver failure.
  • Existing connections drain without request loss.
  • Restricted workloads fail closed when no approved proxy is available.

Pre-cutover checklist

  • [ ] Resolution path and cache owners documented
  • [ ] Previous TTL preserved in the migration timeline
  • [ ] Positive, negative, failure, and stale-cache tests passed
  • [ ] IPv4 and IPv6 tested separately
  • [ ] Connection-age and drain policy verified
  • [ ] Rollback record and gateway capacity ready
  • [ ] Alerts distinguish DNS, connect, TLS, and application failures
  • [ ] Results linked to the proxy DNS leak and IP parity tests

FAQ

Is lowering the TTL five minutes before a cutover enough?

Only if the previous TTL and every downstream cache have already expired. Otherwise, clients may retain the old answer for the original lifetime.

Does a successful DNS lookup prove proxy failover works?

No. A cached socket can remain attached to the old gateway, and proxy-side resolution can differ from local resolution.

Should stale DNS answers be disabled?

Not automatically. They can preserve availability during resolver trouble, but their duration must be known and included in gateway retirement planning.

Related 98IP guides

Compliance note

Run these checks only against domains, resolvers, proxy gateways, and endpoints you own or are authorized to test. Keep request rates bounded, preserve audit logs, and follow the destination's terms and applicable data-protection rules.

Source note: Internet Engineering Task Force, RFC 2308, March 1998; RFC 8767, March 2020; RFC 9520, December 2023.