How to Detect Stale Cached Responses in Proxy Data Collection

A proxy request can complete quickly, return 200, and still deliver information that is too old for a price check, inventory monitor, market-research snapshot, or localized content audit. Rotating the exit IP does not guarantee a new representation: browser caches, shared intermediaries, CDNs, service workers, application caches, and the origin itself may all reuse data.

The right question is not “Was this cached?” Caching is normal and often correct. The useful question is: “Did this response satisfy the freshness contract for this job, and can we prove which layer supplied it?”

Public source note: RFC Editor, RFC 9111 “HTTP Caching,” June 2022.

Define freshness in business terms

Write a maximum acceptable age before sending requests. It should depend on the decision:

Do not invent one global threshold. “Fresh” for a quarterly directory is not fresh for a short promotion. Store the contract with the result so later reviewers understand why it passed.

Understand the evidence each header provides

Capture response headers without treating any single field as absolute truth:

• Date identifies when the origin says the message was generated.
• Age reports an estimate of time spent in caches when a stored response is reused.
• Cache-Control describes caching and revalidation directives.
• Expires can define an expiration time.
• ETag and Last-Modified are validators that can support conditional requests.
• Vary identifies request fields that affect cache selection.
• Warning or intermediary-specific diagnostic fields may add context, but portable logic should not depend on an undocumented vendor header.

Clock skew can distort wall-time comparisons. Use synchronized UTC for timestamps and a monotonic clock for request duration. The proxy clock-skew guide explains that separation.

An absent Age header does not prove the response came directly from the origin. A present Age value does not prove the body is unacceptable. Compare all evidence against the job's contract.

Build a controlled fixture

Use an authorized endpoint whose representation you can update. Return a harmless version marker inside the body and support ETag, Last-Modified, Cache-Control, Date, and conditional requests.

Prepare at least these phases:

1. Publish version A and retrieve it through the direct control and each proxy route.
2. Repeat within the declared fresh lifetime and confirm legitimate reuse.
3. Publish version B while version A is still fresh; document whether the contract permits A.
4. Wait until validation is required, then repeat.
5. Send a conditional request with the version-A validator.
6. Simulate an authorized origin outage and observe whether stale serving is explicitly permitted.
7. Restore the origin and confirm the route converges on version B.

Never use a random production page as the fixture. If the content changes unpredictably, you cannot distinguish cache behavior from origin variation.

Record a result envelope

For each attempt, store:

test_case_id
observed_at_utc
route_id
exit_region
session_id_hash
request_header_profile
final_status
redirect_chain_digest
date_header
age_seconds
cache_control
expires
etag_hash
last_modified
vary
content_type
body_byte_count
body_digest
semantic_version_marker
duration_ms
freshness_contract_seconds
freshness_result

Hash validators when they may reveal internal identifiers. Never log proxy credentials, authorization headers, cookies, tokens, personal data, or full sensitive response bodies.

Use the request-header integrity test to prove that control and proxy requests use the same relevant headers. If Accept-Language, encoding preferences, cookies, or authorization differ, a different body may be correct rather than stale.

Classify the result before retrying

Use explicit states:

• Fresh and equivalent: within contract, expected version or validator, correct semantics.
• Fresh but variant: within contract, but content differs for a legitimate Vary, region, account, or experiment dimension.
• Stale but permitted: older than normal freshness, yet explicitly allowed by the response policy and acceptable to this job.
• Stale and unacceptable: exceeds the job contract or persists after required validation.
• Origin-updated, cache-not-converged: controlled origin has version B while a route still returns A outside policy.
• Inconclusive: missing clock, header, body, or control evidence.
• Invalid content: block page, login shell, partial body, or wrong schema; freshness cannot rescue it.

Do not label every digest difference “fresh.” A different error page is not a useful new representation. Validate required fields, locale, record counts, and version marker after transport succeeds.

Revalidate without destroying the evidence

Run the original request first and preserve its result. Then, when the origin's contract allows it, issue a controlled revalidation request using the recorded validator. A 304 Not Modified can confirm that the stored representation remains valid; it does not provide a new body.

Use Cache-Control: no-cache when the test requires validation, do not confuse it with no-store, and do not add random query parameters by default. Cache-busting query strings can change the application resource, server behavior, analytics, or signature requirements. They also test a different cache key.

If you must test a unique URL, use an authorized fixture endpoint and label it as a separate experiment. Do not use uncontrolled cache busting against third-party sites.

Compare routes one dimension at a time

Create a matrix across:

• direct control and proxy routes;
• each required region;
• fresh and reused connections;
• clean and documented browser profiles;
• normal retrieval and conditional validation;
• exact URL, method, relevant headers, and account state;
• origin healthy and approved outage simulation.

Changing exit, IP, cookies, locale, browser version, request headers, and URL simultaneously destroys attribution. The proxy connection-pool age test helps isolate connection reuse from response reuse.

Diagnose common failure patterns

Same old body, increasing Age

This can be legitimate until the response or business freshness lifetime expires. If it persists beyond policy, collect the cache directives, validator behavior, route, and body digest before escalating.

Same old body, no Age

Check the browser cache, service worker, application response, and origin. Absence of Age is not proof of a direct origin fetch.

Different body after rotating IP

Confirm locale, account state, experiment assignment, and Vary inputs. The new body may be a regional variant, not a fresher representation.

304 but semantic data appears old

The validator may correctly identify an unchanged representation while the origin application itself has stale data. Cache validation proves representation currency relative to the validator, not business truth.

Fast response treated as a cache hit

Latency is not reliable cache evidence. A nearby origin can be fast and a congested cache can be slow. Use headers, validators, controlled version changes, and body digests.

Operational checklist

• Define the freshness threshold for each job.
• Record exact URL, method, headers, route, region, and browser/client build.
• Preserve Date, Age, Cache-Control, Expires, validators, and Vary.
• Hash the body and validate its business semantics.
• Compare against a controlled direct route.
• Preserve the first attempt before revalidation or retry.
• Change one dimension at a time.
• Distinguish legitimate variants from stale content.
• Avoid uncontrolled query-string cache busting.
• Redact credentials, identifiers, cookies, and sensitive bodies.
• Alert on contract failure, not merely on cache use.

FAQ

Does rotating a residential proxy guarantee fresh content?

No. Rotation changes the network exit, but browser, intermediary, CDN, service-worker, application, or origin caches may still reuse a representation. It can also produce a different regional variant rather than a newer one.

Is Age: 0 proof that the origin generated a new body?

No. It is one cache signal. Use validators, controlled version markers, body digests, and direct comparisons.

Should every data-collection request use no-cache?

No. That can add origin load and remove valid performance benefits. Use it when the job requires validation and the destination permits the request behavior. Match the request to the documented freshness contract.

Can a stale response ever be acceptable?

Yes, if HTTP policy permits it and the business contract accepts it—for example, a noncritical reference during an approved origin outage. Label it explicitly rather than calling it simply fresh.

Compliance note

Test only endpoints, accounts, datasets, and proxy infrastructure you are authorized to use. Respect cache directives, access controls, contracts, privacy, copyright, robots policies where applicable, and rate limits. Keep fixture traffic small; do not use cache testing to bypass restrictions or force excessive origin load.