Service 12 · Long-run extraction
Extraction runs that survive a failure
The hard part of extraction is not the first hundred records. It is hour three, when a session expires and the run quietly starts returning nothing.
Fig. 12 — Long-run extraction
What this usually looks like
- A run that dies halfway and has to start again from the beginning.
- A login that expires and produces empty rows instead of an error.
- Duplicate records, because the same page was read twice.
- No way to tell whether the run covered everything it should have.
What you get
- A resumable run, so a failure costs minutes rather than the whole batch.
- A session that reports when it has expired instead of returning nothing.
- Deduplication on a key you choose.
- A coverage check, so you know what was and was not reached.
Built an extraction tool that retrieved 3,500 records behind a login, with session recovery and a coverage report. It runs 127 automated tests.
Questions
Is this allowed?+
It depends entirely on the site and its terms. We will tell you plainly if the answer is no, and often it is.
How large a run?+
Tens of thousands of records is routine. Beyond that the constraint is usually the source, not the tool.
What if the site changes?+
The run reports a shape change rather than silently returning wrong data. That is the difference between a fixable problem and a silent one.
Next step
Tell us what is not working
Describe the problem in a few lines. You get a plain answer about whether it is worth fixing and what it would take, before any money changes hands.
Start a conversation →