Residential Proxy Crawl Scope: Check URLs Before They Enter the Request Queue

A valid seed does not constrain the whole crawl

A monitoring job may start with an approved catalog page yet later discover account links, arbitrary search parameters, or a different host. Buying more residential routes does not resolve that expansion. Define the permitted request set before scaling the worker, and apply the decision to every candidate, not just the initial seed.

RFC 9309 specifies robots exclusion rules for automated clients; those rules are not access authorization. Use a tested robots parser for that layer. This guide proposes a separate application gate combining explicit task permission, URL scope and operational limits. It is not a replacement parser implementation or a claim of legal permission.

1. Turn the monitoring brief into a request contract

For an owned test site or a site you are explicitly authorized to monitor, record approved scheme, exact host and permitted ports; path families; allowed query parameters and values; purpose; expiry; and the person responsible for scope decisions. Define a page and request budget before the first run. A discovered URL and a successful 200 response are not additions to this contract.

For example, an owned fixture may permit /catalog/ and a fixed set of market and page values while excluding /account/ and /search/. This is a sample application contract, not a rule attributed to 98IP or to a target website. Store a version identifier so a later decision can be traced to the scope in force at that time.

2. Normalize for comparison without changing the request's meaning

Resolve relative links against the actual document URL and parse them with a maintained URL library. Compare hostnames as hostnames, not by checking whether a string contains an approved name. An exact host allowlist must not accept an unrelated hostname merely because the approved text appears in its label. Treat subdomains and non-default ports according to the contract rather than including them implicitly.

Remove fragments for queue deduplication, but handle path case, percent encoding, duplicate query keys and parameter order according to the target's semantics. Do not blindly decode reserved delimiters or drop every parameter. Hold an ambiguous URL for review; sending a changed URL to simplify matching can test a different resource.

3. Decide before enqueueing and again before sending

Evaluate the resolved candidate against explicit authorization and the application's host, path and query contract, then apply the applicable robots decision using your real crawler identity. Check the remaining budget and policy validity. A useful result model has three states: send, reject, and hold. Record a reason code such as host_out_of_scope, query_not_approved, policy_expired or policy_unavailable.

Apply the same gate at discovery, queue admission and dispatch. A URL admitted yesterday must not remain approved after the task contract changes today. Persist the policy version with the item; compare it with the active version at dispatch, and recompute the decision when necessary. Do not put rejected items into an ordinary retry queue.

4. Recheck each redirect destination before following it

Automatic redirect following can send a request before application code sees the next destination. For this acceptance test, configure the client to expose redirects, resolve the Location value, and run the destination through the gate before following it. Bound the chain length and detect repeated destinations. A permitted first host does not authorize the final one.

Use a controlled endpoint that redirects from an approved catalog path to an excluded account path, then to a host outside the contract in another fixture. The expected result is a stopped chain with a reason, not a fetch followed by disposal of the response. Also confirm that authentication material is not forwarded across an unapproved boundary.

5. Test queue behavior with a small owned fixture set

Create six candidates: an approved catalog URL; an excluded account path; a forbidden query value; a redirect outside scope; a queued item carrying an expired policy version; and an approved URL discovered repeatedly. Before sending, write down the expected decision for each. The duplicate should consume one admitted task according to the chosen canonicalization, while the expired item must be evaluated under the current policy.

Use origin access logs or an instrumented local request sender to verify that rejected destinations received no request. Queue counters alone cannot establish that. If robots retrieval or its evaluation is unavailable or ambiguous, our conservative application policy holds affected tasks for review; this is a chosen operational safeguard, not a description of every protocol error rule.

Accept the run only when the expected approved sample was retrieved, denied samples were never sent, holds remained separate from transport retries, and the budget stopped new requests at the configured threshold. A timeout or 429 is a network or target outcome for an admitted URL; it must not widen scope, reset the budget, or justify switching identity to defeat a refusal.

Where residential proxy routing helps

Use 98IP dynamic residential IP when an approved monitoring contract needs observations through selected residential markets. Keep the same scope gate regardless of the exit used, confirm the observed route, and configure only account-supported options. A proxy selects a route; it does not grant access rights, interpret your contract, or guarantee that a target accepts the request.

Before increasing scale, review the 98IP operation guides for the account's connection workflow and limits. Compare admitted requests and actual account usage over matched runs; the gate may reduce unwanted work, but no particular speed, cost saving or target compatibility is established by this design.

FAQ and operational checklist

Is an allowed robots result enough? No. The task still needs explicit permission and must fit the application contract. Conversely, do not use a proxy to bypass a robots restriction or an access denial.

Can rejected candidates be retried after IP rotation? A scope rejection is a policy decision, not an exit failure. Reconsider it only after an authorized scope change; an uncertain decision remains on hold until resolved.

Release checklist: exact host/port handling, meaningful path/query comparisons, real crawler identity, versioned contracts, dispatch rechecks, redirect interception, separate reject/hold/retry queues, finite budgets, and origin-log evidence. Keep diagnostic URLs free of secrets or redact sensitive query values. Apply privacy and data-retention rules to the collected content as well as to request logs.