Diagnose Truncated Web-Scraping Responses Before Blaming the Proxy

A scraper can receive HTTP 200 and still save an incomplete document. The connection may close early, a client may stop reading, a decompressor may fail, a gateway may apply a size ceiling, or the target may intentionally return a smaller variant. Replacing the proxy pool before identifying the boundary wastes time and can hide the real fault.

This guide builds a reproducible response-integrity test. It is designed for authorized data collection, QA, monitoring, and research workflows.

Define “complete” for the resource

Do not use HTTP status alone. Define at least two completion signals:

• Transport signal: the received byte count agrees with the declared framing when a reliable length is available.
• Application signal: a known closing marker, record count, checksum, schema field, or pagination token is present.

For HTML, a closing tag is only a weak sentinel because valid pages can omit it. Prefer a stable footer identifier, structured-data block, or expected content section. For JSON, parse the full document and validate required keys. For archives or media, use the format’s checksum or end-of-file rules.

Capture the raw facts before parsing

For every attempt, record:

Keep raw headers and a hash of the body, but redact cookies, authorization headers, tokens, and personal data.

Do not compare the wrong byte counts

If Content-Encoding: gzip is present, Content-Length commonly describes the compressed representation on the wire. Many libraries expose the automatically decoded body. Comparing decoded bytes directly with the compressed length creates a false truncation alert.

Track three values separately:

1. declared transfer length, when present;
2. received compressed or wire length, when observable;
3. decoded application-body length.

If the client hides wire bytes, disable automatic decompression in a controlled diagnostic run or capture the transfer with an approved observability layer. Do not change production behavior merely to obtain a convenient metric.

Run a four-route control test

Use the same URL, headers, method, timing window, and parser across four bounded routes:

1. direct control from an authorized network;
2. one known-good proxy route;
3. the suspected route;
4. a second independent exit in the same target region.

Repeat each route a small fixed number of times. Compare status, headers, wire bytes, decoded bytes, body hash, sentinel result, and timing.

Interpret the pattern:

• Only one route truncates: investigate that exit, upstream gateway, or connection reuse.
• All proxy routes truncate at the same byte boundary: inspect shared client, gateway, or provider limits.
• Direct and proxy routes truncate identically: the target, request profile, or client is more likely responsible.
• Lengths vary with content but sentinels pass: the target may be serving legitimate personalized or compressed variants.
• Failure occurs at a fixed elapsed time: investigate read timeout or stalled transfer before assuming a size ceiling.

Add streaming integrity checks

A robust collector should count bytes while reading and preserve the last successful offset. The example below uses generic JavaScript-style pseudocode so the logic can be adapted to the approved HTTP client in use.

async function collectWithIntegrity(response, expected) {
  const chunks = [];
  let decodedBytes = 0;

  for await (const chunk of response.body) {
    decodedBytes += chunk.byteLength;
    chunks.push(chunk);
  }

  const body = concat(chunks);
  const result = {
    status: response.status,
    decodedBytes,
    declaredLength: parseLength(response.headers.get("content-length")),
    contentEncoding: response.headers.get("content-encoding"),
    contentRange: response.headers.get("content-range"),
    bodyHash: sha256(body),
    sentinelPresent: expected.sentinel ? body.includes(expected.sentinel) : null
  };

  result.complete = classifyIntegrity(result, expected);
  return { body, result };
}

The classifier should return complete, truncated, intentional_partial, or inconclusive. Never force every response into pass or fail.

Recognize intentional partial content

HTTP 206 with a valid Content-Range can be correct when the request used a range or when a media client resumes a transfer. Before labeling it truncation, check:

• whether the request included Range;
• whether a redirect or retry added it;
• whether the returned range matches the requested interval;
• whether the total resource size is known;
• whether the collector correctly assembles multiple ranges.

Unexpected 206 responses, overlapping ranges, missing intervals, or a 200 response that ends early deserve investigation.

Separate client ceilings from network failures

Test controlled synthetic payloads on infrastructure you own or are authorized to use: for example 256 KB, 1 MB, 2 MB, 4 MB, and 8 MB. Use incompressible and compressible payloads separately. A fixed decoded-size boundary points toward a client or post-processing ceiling; a fixed wire-size boundary points toward transfer or gateway policy; a time-dependent cutoff suggests timeout behavior.

Google Search Central has publicly documented that its crawler fetches only a bounded number of bytes per resource and treats the fetched prefix as the available document. That example is a useful reminder: an apparently clean connection can still be intentionally bounded by the consumer. Record every component’s documented and observed limits.

Retry without destroying evidence

Blind retries can turn one diagnostic signal into many inconsistent samples. Use a small retry budget and preserve the first failure.

• Retry connection resets and timeouts with exponential backoff and jitter.
• Do not retry deterministic sentinel or schema failures indefinitely.
• Rotate an exit only when the experiment is explicitly testing route dependence.
• Keep request headers, locale, cookies, and timing stable between control routes.
• Store the reason for each retry and the final classification.

Operational checklist

• Define transport and application completion signals.
• Record redirects, status, framing, encoding, ranges, bytes, hashes, and timings.
• Keep compressed, wire, and decoded lengths separate.
• Compare direct, known-good, suspected, and same-region control routes.
• Test fixed synthetic payload sizes on authorized infrastructure.
• Treat valid HTTP 206 responses as intentional until evidence shows otherwise.
• Preserve the first failed body and its last successful byte offset.
• Use bounded retries and stable request profiles.
• Redact credentials, tokens, cookies, and personal data.
• Escalate with a compact evidence table instead of a screenshot alone.

FAQ

Does HTTP 200 prove that the complete body arrived?

No. A connection can terminate after headers are delivered. Validate body framing when possible and use an application-level sentinel or schema check.

Should Content-Length always equal the body buffer size?

Not when automatic decompression is involved. The header may describe compressed transfer bytes while the client exposes a decoded buffer. Compare like with like.

Is a smaller page always blocking or cloaking?

No. Language, device, session, consent, experiments, compression, and legitimate personalization can change size. Compare semantic sentinels and route-controlled samples before drawing a conclusion.

When should the proxy route be replaced?

Replace or quarantine it when controlled repetitions show route-specific resets, truncation, corrupt chunks, or materially worse completion rates while the same request succeeds through independent controls.

Compliance note

Collect only data you are authorized to access. Follow website terms, robots guidance where applicable, privacy and data-protection requirements, contractual limits, and reasonable request rates. Do not use proxies to bypass authentication, access controls, paywalls, or enforcement mechanisms.

For the next diagnostic step, combine this workflow with the proxy-versus-target control-route plan and the browser proxy capacity guide.