How to Test gzip and Brotli Integrity Through a Proxy

An HTTP response can return status 200 while the client, proxy or runtime disagrees about compression. A stale Content-Encoding header, double decompression, truncated stream or incorrect byte counter can corrupt collected data and distort proxy cost estimates. These faults are easy to miss when a health check only searches for one text marker.

This guide creates a deterministic gzip and Brotli test for proxy trials. Run it only against an endpoint you own or are authorized to test.

Define two truths: wire bytes and decoded content

Do not compare only the final string. For every response, capture:

test_id
route_alias
accept_encoding
content_encoding
content_length_header
wire_bytes
decoded_bytes
wire_digest
decoded_digest
stream_complete
failure_phase

The wire digest proves what arrived before decompression. The decoded digest proves whether the original fixture was reconstructed. Many high-level clients automatically decode content and remove or preserve headers differently, so record runtime behavior instead of assuming it.

Never log proxy credentials, cookies, authorization headers or personal response data. Use synthetic payloads with known digests.

Build an owned fixture set

Create several deterministic responses:

• compressible UTF-8 text with repeated patterns;
• already-compressed binary bytes that should gain little from compression;
• multilingual text and emoji split near chunk boundaries;
• a large streamed response;
• empty, 204 and HEAD responses;
• intentionally truncated gzip and Brotli streams for negative controls.

Return a random challenge outside any shared cache key and expose the expected decoded digest through a separate authorized control channel.

Test a direct-and-proxy matrix

Keep target, payload seed, headers, timeout and client version fixed. Repeat each cell enough to distinguish a deterministic mismatch from a transient transport failure.

Step 1: establish identity and direct controls

Request identity first and verify the decoded digest and byte length. Then request gzip and Brotli directly. Confirm that each compressed stream decodes to exactly the identity fixture.

This establishes whether the origin and client are correct before adding a proxy. If the direct path fails, a proxy comparison cannot identify the cause.

Step 2: repeat through each proxy route

Run the same requests through every purchased gateway and market selector. Compare status, Content-Encoding, wire length, decoded length and both digests.

Classify each result:

• pass: header, stream and decoded digest agree;
• transparent transformation: wire representation changes but decoded content remains correct and headers match the transformation;
• header mismatch: the declared encoding does not match the received body;
• double decode: a layer decodes content and another layer tries again;
• truncation: stream ends before the decoder reaches a valid finish;
• content corruption: decoding completes but the digest differs.

Do not call every transparent transformation a defect. It becomes operationally important when it violates product promises, breaks signatures, changes billing or varies by route.

Step 3: exercise chunk and stream boundaries

Compression bugs often appear only when headers or multibyte characters cross chunks. Vary server chunk sizes and introduce controlled pauses. Read incrementally rather than buffering every body, and ensure the decoder reports a clean end.

For long-running collection, apply bounded idle and total timeouts separately. A partial decoded document must never be committed as a complete record. The response integrity test provides a wider semantic check after transport validation.

Step 4: prevent cache confusion

If a cache is present, verify that responses vary correctly by Accept-Encoding. A gzip object must not be served to an identity-only client with incompatible headers. Use unique challenge values and compare cache-hit and cache-miss paths separately.

Do not bypass third-party cache controls. Test only your fixture or an explicitly authorized environment.

Measure bandwidth and cost fairly

Providers and clients may report different byte layers. Label every metric as wire, decoded, request, response or total. Calculate:

• compression ratio = wire response bytes / decoded response bytes;
• transport success rate = complete valid streams / attempts;
• useful-byte rate = valid decoded bytes / billed bytes;
• cost per valid document = total route cost / valid documents.

Compare with the bandwidth cost estimation guide and cost per successful request. Never compare one provider's decoded bytes with another provider's wire bytes.

Failure and retry policy

Retry truncated or reset reads only when the operation is safe. A GET can still trigger side effects on poorly designed systems, so verify the owned endpoint. Do not blindly replay POST, registration, order or message actions. Preserve whether failure occurred before headers, during compressed body read, during decode or after digest verification.

Release checklist

• Identity, gzip and Brotli direct controls pass.
• Wire and decoded digests are stored separately.
• Headers match the actual body representation.
• Multilingual and chunk-boundary fixtures pass.
• Truncated negative controls fail closed.
• Partial documents are never marked complete.
• Cache variants respect the encoding offer.
• IPv4, IPv6 and gateway results are separated.
• Billing metrics name the byte layer.
• Logs contain no credentials or user data.

FAQ

Why can Content-Length differ from the decoded body length?

It commonly describes the encoded representation on the wire. Automatic decompression can produce a larger decoded body. Record both layers explicitly.

Is Brotli always smaller than gzip?

No. Results depend on payload, settings and existing compression. Measure representative fixtures instead of assuming.

Can I validate only with a JSON parser?

No. Some corrupt payloads still parse, and some valid streamed payloads fail before completion. Verify stream completion and a known decoded digest.

Should compression be disabled to avoid problems?

Identity mode is a useful fallback and diagnostic, but may raise bandwidth and latency. Fix the mismatched layer and measure tradeoffs before disabling compression broadly.

Compliance note

Use synthetic or authorized content, follow destination policies and rate limits, and minimize retained payloads. Do not inspect third-party private traffic or expose credentials. Compression validation should improve data correctness, not bypass access controls.