Proxy NDJSON Streams: Test Read Boundaries Before Blaming Missing Records

An authorized NDJSON export works on one route but fails when network reads arrive in different sizes. A client that treats each read as one JSON record can appear reliable until a record spans two reads, several records arrive together, or a UTF-8 character is split between byte buffers.

Test one simple invariant: the same valid byte sequence must produce the same accepted records regardless of how those bytes are divided into reads. This guide proposes controlled parser and proxy-path tests; it does not report executed results or guarantee 98IP streaming behavior.

1. Specify the record and completion contract

Confirm that the destination actually returns NDJSON, not one JSON array or server-sent events. The NDJSON specification uses UTF-8, with each JSON text followed by a newline; parsers accept LF and CRLF delimiters. Decide and document the handling of empty lines, malformed JSON and size limits.

For this test, use a strict newline-terminated fixture. A complete JSON value lacking its required final delimiter is a framing failure under that contract, but does not by itself prove a network truncation. A partial JSON fragment is a different failure. Keep both distinct from a successful end-of-stream.

Define completeness separately from syntax. For a small finite fixture, require the expected record IDs and values. If the real API supplies a documented manifest or completion record, verify it under that API's rules; do not invent one or assume every stream has a known count.

2. Prepare one byte fixture and several partitions

1. Create three harmless synthetic records with unique known IDs. Include one non-ASCII value and one string containing an escaped newline, rather than an actual line break inside a JSON text.
2. Serialize once using the declared UTF-8 and delimiter policy. Retain the exact bytes and expected parsed values as the local test reference.
3. Feed those same bytes to the consumer as one read, many small reads, and several deliberately uneven reads.
4. Place boundaries inside a multibyte character, inside a JSON token, between CR and LF for the CRLF variant, and after more than one complete record.

These partitions are controlled client fixtures, not a promise about TCP packets or the chunks a browser will expose. Do not add delays and assume they force a particular read boundary. Verify the consumer first with deterministic partitions, then observe the actual network path separately.

3. Preserve decoder and line-buffer state

Use an incremental UTF-8 decoder that retains state between reads, or an appropriate text-decoding stream supported by the runtime. Independently decoding each byte buffer can damage a character spanning buffers. Flush the decoder at end-of-stream and treat a decoding error according to the explicit strictness policy, not as successful content.

Append decoded text to a pending line buffer. Extract only complete delimited lines, parse each according to the record contract, and retain any unfinished tail for the next read. One read may yield zero, one or many records. Do not split on every carriage return without accounting for a CRLF pair divided across reads.

Limit buffered bytes, line length, total accepted records and the overall test duration. A stream that never emits a delimiter must not grow memory indefinitely. Keep byte limits and character limits clearly named, since they are not interchangeable for non-ASCII data.

4. Compare the proxy path without confusing representations

Serve the owned fixture from a small authorized endpoint, then use the same client and contract on an approved baseline route and the proxy route. Check status and actual representation before parsing: an HTML login response behind HTTP 200 is not NDJSON.

The 98IP dynamic residential IP product page helps identify the product used for authorized data-route testing; verify current account options before the run. Check the documented connection method through the API configuration page. Neither page guarantees particular read sizes or defines the destination's record schema.

Record which layer's bytes are hashed or counted. If the HTTP client decompresses a response, compare the fixture against the decoded representation delivered to the consumer, not compressed transfer bytes. A changed read partition with identical representation bytes should not change accepted IDs or values.

5. Keep partial output separate from complete acceptance

At EOF, flush decoding, process only valid complete lines under the declared rules, and inspect the remaining buffer. Mark a malformed tail, missing delimiter, read error, size-limit stop or missing expected record explicitly. Do not discard a bad final fragment merely because earlier records parsed.

Use four evidence columns: representation received, decoding/framing outcome, parsed-record validation and final fixture completeness. In a synthetic case with two valid records and a partial third, report two provisional records and an incomplete stream. Do not count the export as complete.

Choose whether your application stages records until completion or commits incrementally with a documented partial-state marker. If retrying a finite export, use the destination's approved recovery and duplicate-handling rules. Do not blindly append a replay to earlier partial output and then report the larger row count as success.

6. Rehearse failures that an HTTP status cannot detect

Propose local tests that cut a multibyte character at EOF, remove the final delimiter, truncate the last JSON value, insert malformed JSON in a middle line, and exceed the configured line bound. Each should produce its declared error or incomplete verdict, without silently accepting the whole fixture. These exercises have not been executed for this article.

Also vary only the partitions of the valid fixture and require identical parsed results. If this fails locally, fix the decoder or framing consumer before blaming the proxy. If local partition tests pass but representation bytes differ on the network path, investigate response generation, transfer and client transformations with redacted evidence.

Record case ID, fixture version, delimiter policy, partition plan or observed read sizes, decoder policy, first failing boundary, accepted IDs and completion verdict. Keep actual data private; a small synthetic reproducer is usually easier to review than an entire customer export.

FAQ and acceptance checklist

Can I parse every read independently with JSON.parse? No. A read boundary is not a record boundary. Frame complete lines after incremental decoding.

Does a stream that opens cleanly in a viewer prove completeness? No. A viewer can display valid earlier records while the finite export is incomplete.

• The representation, delimiter and EOF policies are explicit.
• The same bytes yield the same records across controlled partitions.
• UTF-8 and CRLF state survive read boundaries.
• Buffers, duration and record counts are bounded.
• Partial records and complete-export verdicts remain separate.
• Retry and commit behavior cannot hide a missing tail or create untracked duplicates.

Use only owned or explicitly authorized endpoints, synthetic data and a small request budget. Redact proxy credentials, access tokens and signed URLs, and retain only necessary diagnostic artifacts. Do not claim measured throughput, loss rates or business outcomes from this proposed acceptance plan.