Proxy Response Integrity Test: Detect Truncation, Decompression Errors, and False Success

A proxy request can return HTTP 200 and still fail the business task. The body may end early, decompression may fail, a streaming parser may accept only the first records, a range response may be reassembled incorrectly, or an application may cache a partial object. Counting such attempts as success makes an exit pool look healthy while downstream datasets become incomplete.

Response integrity is therefore a separate quality dimension from reachability, latency, geography, and status code. Test it explicitly with authorized, deterministic fixtures before trusting production results.

What “complete” means at each layer

Do not use one checksum as a substitute for layer diagnosis. A useful test distinguishes:

1. Transport completion: the connection and protocol stream ended normally.
2. HTTP message completion: the received body satisfied the applicable framing rules.
3. Representation completion: content encoding was decoded successfully and yielded the expected bytes.
4. Application completion: required records, fields, pagination markers or closing delimiters are present.
5. Business completion: the result is fresh, belongs to the requested market and is usable for the approved purpose.

The RFC Editor’s HTTP specifications provide the protocol boundary. For HTTP/1.1, a message with Content-Length is incomplete when fewer octets arrive than declared; chunked transfer is incomplete without its terminal zero-length chunk. HTTP semantics also distinguish the encoded representation from the decoded content and define how byte ranges relate to the encoded sequence.

Build deterministic test fixtures

Use endpoints you own or are explicitly authorized to test. Prepare several stable objects:

• a small uncompressed text file with a known byte length and digest;
• a larger compressible JSON document with a stable record count;
• gzip and Brotli variants when the client supports them;
• a binary object with a known digest;
• a resource that supports single byte-range requests;
• a streamed format with an explicit final record or closing marker;
• a response that intentionally closes early in a test environment.

Version each fixture. Record its encoded length, decoded length, digest, media type, encoding, expected record count and modification identifier. Do not use a public page that can change between direct and proxy requests as the sole reference.

Capture the right evidence

For every trial, retain a sanitized record such as:

trial_id
fixture_version
proxy_route_alias
requested_market
observed_market
address_family
protocol_version
status_code
content_length_header
transfer_encoding
content_encoding
content_range
wire_bytes_received
decoded_bytes
decoded_digest
record_count
completion_marker
attempt_count
failure_layer
duration_ms

Avoid storing proxy credentials, cookies, tokens, personal data or unrestricted response bodies. A digest and carefully selected structural assertions are usually enough for repeatable comparison.

Run a direct-and-proxy matrix

Compare the same fixture under the same client build, timeout, headers and observation window across:

• direct control where policy permits;
• authenticated HTTP or HTTPS proxy;
• SOCKS5 with required local and remote DNS modes;
• rotating residential route;
• sticky residential session;
• static or dedicated route;
• required IPv4 and IPv6 paths;
• Global, North America, Europe and APAC markets used by the workload.

Repeat each cell enough times to observe rare truncation without creating unnecessary load. Randomize route order so a temporary origin issue does not affect only one provider or market.

Validate HTTP framing before parsing content

Content-Length responses

Compare received message-body octets with the declared length before decompression. If the connection closes or times out early, record an incomplete-message failure. Do not pass the partial bytes to a downstream parser as a valid object.

Chunked HTTP/1.1 responses

Require correct chunk syntax and the terminal zero-length chunk. A clean TCP close is not a substitute for valid chunk completion. Treat malformed sizes, missing terminators and premature closure as framing failures.

HTTP/2 and HTTP/3 responses

Use the client library’s stream-completion and reset signals rather than applying HTTP/1.1 chunk rules. Capture stream reset or protocol errors separately from application timeouts. Do not assume that receiving some DATA frames means the response completed.

Close-delimited responses

These are harder to distinguish from interruption. Prefer length- or encoding-delimited fixtures for reliable tests, and retain underlying connection or TLS closure evidence when close delimitation is unavoidable.

Validate content encoding and representation bytes

Send explicit Accept-Encoding values rather than inheriting an unknown client default. Record the returned Content-Encoding order, decode in that order, and fail closed on decompression errors.

Compare both levels where possible:

• wire length and encoded digest for transfer consistency;
• decoded length and digest for representation integrity;
• media-type and character-set expectations;
• structured parse result after decoding.

A proxy or client may transparently decompress content and adjust headers. Define whether your capture point observes encoded or decoded bytes, then keep that point constant across the comparison.

Test byte ranges carefully

For an authorized range-capable fixture:

1. request a known single range;
2. require a valid 206 Partial Content response when the server honors it;
3. validate Content-Range start, end and complete length;
4. confirm the response-body length equals the inclusive range size;
5. compare the range digest with the same slice of the canonical encoded representation;
6. request an invalid range and verify the expected 416 behavior;
7. if resuming, require a stable validator so bytes from different representation versions are not merged.

The server is allowed to ignore a range request and return the complete resource with 200. Your client must distinguish that valid behavior from an incorrectly labelled partial response.

Add application-level completion assertions

Protocol completion alone cannot prove useful data. Choose assertions that match the format:

• JSON parses fully and contains required top-level keys;
• line-delimited JSON ends on a complete record and matches the expected count;
• CSV has a complete header and consistent column count;
• HTML includes a required closing structure plus a stable business marker;
• an archive opens and passes its internal test;
• a media or binary container exposes the expected footer or index;
• paginated data contains the expected terminal cursor or explicit continuation state.

Avoid fragile assertions based on total HTML size or a single phrase. They can produce false failures when legitimate content varies and false passes when an error template contains the phrase.

Classify failures before retrying

Use failure classes that lead to different actions:

• proxy_connect: gateway not reached;
• proxy_auth: credentials rejected;
• dns: gateway or destination resolution failed;
• tls: certificate or handshake failure;
• http_framing: declared message did not complete;
• content_decode: encoding could not be decoded;
• range_integrity: partial response metadata or bytes were inconsistent;
• semantic_integrity: body completed but required structure failed;
• business_validation: structure passed but the result was unusable.

Retry only when the operation is safe and the class is likely transient. Use an attempt budget, exponential backoff and jitter. Never concatenate bytes from separate full-response attempts unless the protocol, validators and application explicitly support safe resumption.

Measure useful delivery, not request success

Track by provider, route class, market, address family and protocol:

• framing-complete rate;
• decode-complete rate;
• semantic-complete rate;
• useful-result rate;
• truncation rate by byte percentile;
• retries per useful result;
• p50 and p95 time to useful result;
• transferred bytes per useful result;
• cost per useful result.

An inexpensive route with more partial responses can cost more once retries, parsing failures and missing records are included.

Acceptance checklist

• [ ] Fixtures are owned or explicitly authorized and versioned.
• [ ] Encoded and decoded lengths and digests are known.
• [ ] Direct and proxy trials use the same client and headers.
• [ ] HTTP framing is validated before application parsing.
• [ ] Decompression errors fail closed.
• [ ] Range responses validate status, metadata, length and digest.
• [ ] HTTP/2 or HTTP/3 resets are recorded separately.
• [ ] Required application records and completion markers are asserted.
• [ ] Direct fallback and unexpected cache hits are detected.
• [ ] Retries have an attempt budget, backoff and jitter.
• [ ] No credentials or personal data appear in evidence.
• [ ] Results are compared by route, market, address family and protocol.

FAQ

Is Content-Length enough to prove integrity?

No. It helps determine message completion in applicable cases, but matching length does not prove that the bytes are correct, fresh or useful. Add decoded digests and semantic checks.

Should the digest cover compressed or decompressed bytes?

Ideally record both at a clearly defined capture point. The encoded digest diagnoses transfer differences; the decoded digest verifies the representation consumed by the application.

Can I compare a live public page with and without a proxy?

Only as a supplementary test. Personalization, edge selection, time, consent state and experiments can legitimately change content. Use deterministic authorized fixtures for integrity gates.

Should every incomplete response be retried?

No. Confirm that the request is safe to repeat, that the failure is likely transient, and that the retry budget remains available. Persistent integrity failures should quarantine the route rather than create a loop.

Compliance and safe operation

Use only authorized endpoints, accounts, data and markets. Respect access controls, platform terms, privacy rules, regional law and rate limits. Do not use proxies to bypass restrictions or conceal prohibited collection. Keep TLS verification enabled, protect credentials, minimize stored bodies and redact artifacts before sharing.

Continue with the proxy request-header integrity test, proxy TLS session-resumption test, and proxy WebSocket long-lived connection test.

Internal research basis: RFC Editor, RFC 9110 HTTP Semantics and RFC 9112 HTTP/1.1, both published June 2022; reviewed September 12, 2026.