Proxy Pagination Data Integrity: Find Duplicate and Missing Records

A page returning HTTP 200 proves that a response arrived. It does not prove that every record arrived once. In authorized regional catalog research or testing your own list API, a proxy change, a changing source list and an incorrectly saved cursor can produce similar symptoms. The useful question is which boundary changed before the first duplicated or missing record.

Set the acceptance rule before collecting pages

Choose a stable record key, such as the source's documented item ID. Keep the regional context alongside it: the same item in two markets may legitimately be two observations. Define required fields and whether the job expects a frozen snapshot or a best-effort view of a live list. If the source provides no snapshot, stable ordering or authoritative total, label completeness unproven instead of declaring success from a page count.

Illustrative case: three pages contain IDs A–D, D–G and I–L. Counting twelve rows hides the repeated D. H looks absent only if an authorized reference establishes that H belongs in this result. Deduplication detects repetition; it cannot prove what is missing.

Use a controlled run to locate the fault

• On an endpoint you own or are allowed to test, create a small fixed fixture with twelve known IDs and a page size of four. Keep credentials, filters, language, sort order and page size unchanged throughout the run.
• Collect a permitted baseline route and a residential-proxy route against the same fixture. Record request time, response status, request context, returned IDs and the next-page indicator. Never put passwords or full bearer tokens into logs.
• Repeat the proxy run with one consistent regional/session configuration. Separately test a deliberate route change at a page boundary if the target permits it. Change only this variable; a country change mixed with a filter change provides no clear diagnosis.
• Compare ordered IDs, duplicates, required-field coverage and end-of-list behavior. If both routes fail on the fixed fixture, inspect application pagination first. If only the changed-context run fails, examine whether the origin binds cursors or results to that context. This result does not by itself prove a proxy service defect.

Follow the origin's pagination contract

For offset pagination, inserting or deleting items before the current offset can shift later pages. A documented immutable sort key plus a unique tie-breaker helps, but only the origin can define snapshot semantics. For cursor pagination, treat the returned cursor as opaque: preserve it exactly and do not construct it from the last visible timestamp unless the API explicitly requires that.

Keep filters and sort settings identical when continuing a cursor. An expired or rejected cursor is a controlled failure, not permission to silently switch to offset zero and append the result. A repeated next cursor, an unexpectedly empty page with a continuation flag, or an unbounded chain should stop the run for investigation. Use the documented termination rule and a finite page/request budget.

Commit records and checkpoints together

Save each accepted batch with a run ID, context fingerprint, page index, stable record keys, returned continuation value and validation result. Persist the batch and its checkpoint in one transaction where possible. With file storage, write a complete batch durably before marking its checkpoint committed; recovery must be able to replay that batch idempotently.

Advance only after storage succeeds. A crash after saving records but before saving the cursor can replay a page, so use a unique key scoped to the run and observation context. A crash after advancing the cursor but before saving records can lose data. Deduplicating output does not repair that second case. On resumption, verify the saved context and cursor validity; otherwise start a separately identified run rather than merging incompatible snapshots.

Choose a 98IP route for the task, then verify it

Use 98IP dynamic residential proxies when the authorized test needs residential regional viewpoints. Before a multi-page job, choose the region and extraction mode in your account and check the available session settings through the operation guides. Do not assume a fixed-duration setting guarantees an unchanged exit IP or an origin-valid cursor. Measure the whole traversal on your endpoint first.

For a single regional dataset, maintain one observation context rather than rotating countries between pages. Separate country runs when you need a comparison. A static route may be appropriate for a longer stable-route requirement, but it still cannot freeze the origin's data, repair broken pagination or override access limits. Review the account's actual product terms before choosing a route; no specific session duration or throughput is promised here.

Release checklist and failure handling

• Known fixture IDs match exactly; no unexpected duplicates or omissions, and required fields pass validation.
• All pages share the intended context; the cursor chain terminates under the documented rule and stays inside the request budget.
• An interruption/resumption test neither loses a committed batch nor duplicates its logical records.
• Live-data limitations are stated, discrepancies are quarantined, and monitoring includes unique-record count, duplicate count, rejected cursor count and traffic used.

If authentication, permission or rate-limit errors occur, pause and follow the origin's rules. Do not rotate IPs to evade a restriction. If the fixed fixture passes but a live list differs, compare source update times and ordering before replacing the proxy. Keep failed batches separate from the accepted output so a clean-looking report cannot hide incomplete data.

FAQ

Does a sticky proxy session guarantee complete pagination? No. Route stability controls one variable; ordering, source mutations, cursor validity and storage correctness remain separate requirements.

Is removing duplicate IDs enough? No. It makes repeated records easier to manage, but cannot establish absent records without a reliable reference or snapshot contract.

Can I resume with another IP? Only if the origin's documented behavior and an authorized test support it. When uncertain, stop the run, retain its checkpoint and create a new observation instead of claiming continuous completeness.

Authorization and evidence

Apply this checklist only to your own endpoints or sources you are authorized to access. Respect terms, request limits and data-protection requirements; minimize retained personal data. The example is a test design, not a measured 98IP benchmark. A proxy changes the network route and does not grant collection permission.