How to Test HTTP/2 GOAWAY Draining Through a Proxy

An HTTP/2 connection can carry many concurrent requests. When an endpoint sends GOAWAY, it is not simply saying “the socket failed.” It is announcing that the connection is being retired and identifying which streams might have been processed. A proxy client that ignores this boundary can open work on a dying connection, lose requests or replay an unsafe operation.

This guide defines a controlled test for HTTP/2 proxy paths. The goal is to prove that the client stops assigning new streams to the draining connection, preserves eligible in-flight streams, opens a replacement connection and retries only when the request semantics and stream evidence allow it.

What GOAWAY evidence means

Record the error code, last stream identifier, receipt time, connection identity and every stream state. Streams above the reported identifier can be treated as not processed by that peer. Lower or equal identifiers are ambiguous if they did not complete: some data may already have reached application logic.

Do not reduce this to “retry every failed request.” GET-like reads may be retryable under your policy. A POST that creates an order, starts a job or charges an account is not safe to replay merely because its response was missing. Use an idempotency key only when the destination actually enforces it.

Build an isolated harness

Use a destination and proxy route you own or are authorized to test. The origin should expose:

• a fast read endpoint;
• a delayed read endpoint;
• a state-changing endpoint with a server-enforced idempotency key;
• a state-changing endpoint without replay protection;
• request and connection identifiers in server-side evidence.

The client must reveal connection IDs, stream IDs, assignment time, headers-sent time, response status, completion state and retry decision. Capture proxy logs without credentials or response bodies containing personal data.

Run the baseline

Open one HTTP/2 connection through the proxy and send a mix of fast, delayed and state-changing requests. Confirm that requests are genuinely multiplexed on one connection and that the origin sees each operation exactly once.

Measure normal p50 and p95 completion time, new-connection setup time and steady-state concurrency. These values become the control; a drain test without a baseline can mistake ordinary latency for shutdown behavior.

Trigger a graceful drain

Start several delayed streams, then make the controlled endpoint emit GOAWAY with NO_ERROR. Keep some streams below the last stream identifier in progress and attempt to schedule fresh work immediately after the frame arrives.

The expected behavior is:

1. the old connection is marked draining at once;
2. no new stream is assigned to it;
3. eligible in-flight streams are allowed to finish within a bounded drain window;
4. a replacement connection is opened for new work;
5. streams known not to have been processed are classified for retry;
6. ambiguous non-idempotent operations are surfaced for reconciliation, not blindly replayed.

Repeat with a last stream identifier that excludes several recently opened streams. Then repeat with an error GOAWAY and with a connection close after the frame. The client must not assume every drain is clean.

Test the dangerous timing windows

Run each case many times with controlled jitter:

• GOAWAY arrives before request headers are sent;
• it arrives after headers but before the request body completes;
• it arrives while a response body is streaming;
• two GOAWAY frames arrive with a lower final last stream identifier;
• the socket closes before the drain deadline;
• the replacement connection fails authentication or TLS verification;
• many workers observe GOAWAY at the same time.

Verify that concurrency does not create a retry storm. Only one bounded replacement effort should be needed per affected route or pool. Apply the backoff principles in the proxy retry budget guide, and compare the result with the idle timeout and keep-alive test.

Define retry classes before testing

Create an explicit decision table:

request_class | stream_evidence | automatic_action
idempotent read | not processed | retry within budget
idempotent read | ambiguous | retry only under documented policy
protected write | not processed | retry with the same enforced key
protected write | ambiguous | reconcile by key before retry
unprotected write | any ambiguity | do not auto-replay

Keep transport retry, application retry and business reconciliation separate. A library default is not a business-safety policy.

Measure the outcome

Useful metrics include:

• new streams assigned after GOAWAY: must be zero;
• eligible in-flight completion rate;
• replacement-connection establishment time;
• requests retried by class and reason;
• duplicate side effects: must be zero;
• abandoned or orphaned operations;
• retry amplification per original request;
• drain duration and forced-close count;
• unaffected connection and route success rate.

Segment results by proxy protocol, exit market, client runtime, library version, IPv4 or IPv6, and sticky or rotating session mode. A rotating route may legitimately change exit identity on the replacement connection; a sticky contract may require a new session token or explicit continuity check.

Production rollout checklist

• GOAWAY is visible as a distinct event, not a generic socket error.
• Connection and stream identifiers are present in sanitized logs.
• The draining connection accepts no new streams.
• In-flight work has a bounded completion window.
• Replacement creation is coalesced and rate-limited.
• Retry rules are based on method semantics, stream evidence and idempotency controls.
• Unsafe or ambiguous writes enter reconciliation.
• Duplicate side effects remain zero under repeated fault injection.
• Retry amplification stays inside budget.
• IPv4, IPv6 and required markets are tested separately.
• Credentials never appear in traces or fixtures.
• A rollback can disable automatic replay without disabling all traffic.

FAQ

Is GOAWAY always an error?

No. NO_ERROR commonly signals graceful retirement. Other codes can indicate a protocol or internal failure. In both cases, the connection must stop receiving new streams after the applicable boundary.

Can every stream above the last stream identifier be retried?

The transport evidence says those streams were not processed by the peer that sent GOAWAY, but application policy still matters. Preserve authentication, deadlines, idempotency keys and retry budgets; do not replay work after its business deadline.

Should the client wait for the old connection before opening a new one?

Usually no. New work should move to a replacement while eligible old streams drain. Bound replacement concurrency so a fleet-wide drain does not create a connection storm.

What if the proxy translates HTTP/2 to HTTP/1.1 upstream?

Test the behavior you can observe at both ends. The client-facing GOAWAY boundary remains important, while origin evidence is needed to decide whether an ambiguous write was processed.

Compliance note

Run drain and retry tests only on systems, accounts, proxy routes and data you are authorized to use. Respect destination terms, rate limits and privacy requirements. Use synthetic records, redact credentials and retain only the evidence needed for reliability and audit purposes.

Source note: IETF, RFC 9113 “HTTP/2,” June 2022. The external research location is retained only in the internal operations record.