One Bad Row Killed the Whole Batch: Partial-Failure Handling for n8n Loops
This is part of my build-in-public series on running LLM work...
For further actions, you may consider blocking this person and/or reporting abuse
one bad row shouldn't eat 453 good ones.
That is the whole point in one line. The part that stings is the silence: a batch that stops at row 47 usually still reports "failed" for the one row, but nothing reports the 453 that never ran. So the fix that matters most is arguably not the isolation itself but the report — processed, failed, skipped, retried — because "failed: 1" and "failed: 1, never attempted: 453" are two very different mornings.
yeah, the silence is the worst part. no error, just stops.
Exactly — and the fix for the silence is to make "no error" never count as a result. Two rules I use now:
The error was never the missing exception. It was a loop that could finish without ever saying what it did.
zero rows processed counting as a failure, that's the one I'd keep.
That's the one I'd keep too — but with one refinement: zero processed is only a failure when you expected input. A run that legitimately had nothing to do (no new orders overnight, an empty source) and pages on it will train people to ignore the page, which is how the next real silence gets missed.
So I record three numbers, not one: rows fetched, rows processed, rows skipped. The rule becomes:
The implementation detail that makes this work: the fetch step must never return an empty list on error. An error returns an error, not zero rows. If the only thing downstream ever sees is a list, "failed to fetch" and "fetched nothing" are indistinguishable, and no summary format can save you.
Two drills worth running: one with a genuinely empty source, one with the fetch broken mid-pull. They must produce distinguishable reports — not the same "done, 0 rows". If they don't, the zero-row rule is decoration, not a check.
The four counters are valuable because they stop “complete” from hiding a partial result. I would add one invariant to the run summary:
processed + failed + skipped + pending-review = input, with a stable item key behind each count. That makes a missing row detectable immediately and gives the dead-letter retry flow a bounded, auditable input instead of relying on operators to reconstruct what happened from logs.Yes — and I would harden your invariant in one way: a sum is necessary but not sufficient. Processed + failed + skipped + pending-review = input can still hold while the wrong row is missing, if one row is dropped and another is counted twice. So the check I actually enforce is a key-set partition, not just an arithmetic sum: every input key appears in exactly one bucket, no key in two buckets, and no input key unaccounted for. The sum is the cheap alert; the set difference is the proof.
Two details that make that workable in practice. First, 'input' has to mean rows admitted to the loop, not rows the source claims to have. If the fetch itself fails partway, input is unknown, and the invariant cannot rescue it — that is a separate fetch-outcome line (source, status, rows admitted), not a fifth counter to fudge the sum. Second, pending-review must be its own bucket, not folded into failed: a failed row is terminal and retryable from the dead-letter list, while a pending-review row is unresolved — usually an ambiguous send where re-running it could duplicate an effect. Mixing them lets an ambiguous row look resolved.
On the stable key: it has to be a business key, not a row index or position, or a retry that reorders the batch silently re-keys everything. And the dead-letter retry must re-enter through the same dedup check, so a retried row that already succeeded can never be processed twice just because it is back in the input.
Exactly—conservation counts tell you volume; key-set partition tells you identity. I’d also persist the input fingerprint and run ID alongside each bucket, so retries and replays can be tested without confusing duplicate execution for a dropped row.
Exactly — and the split you are drawing is the one that makes replay tests meaningful: fingerprint over the key set for identity, and a separate content hash if you also want to detect mutation. If you fold them into one hash, a legitimately corrected field looks like a dropped row, and your replay test fails for the wrong reason.
Two details I would add. First, the fingerprint has to be computed over admitted keys only, not over what the source claims. If the fetch fails partway, there is no trustworthy fingerprint yet — that run gets a fetch-outcome record, not a fingerprint over a partial set, or a later complete run will never match it and you will chase a phantom difference.
Second, keep origin_run_id and run_id as separate fields on every bucket row. A retry is a new run_id carrying the old origin_run_id and the same item keys. Then a replay test can assert exactly what you want: same origin fingerprint plus same keys gives the same partition, a retried key lands in the same bucket it would have landed in originally, and a duplicate execution shows up as two run_ids sharing one key — not as a missing row. Without that pair, retry and duplicate are indistinguishable in the report, which is the confusion the fingerprint was meant to remove.
That distinction also suggests a useful invariant for the report: only compare partition fingerprints for runs marked complete, while incomplete fetches are summarized by the admitted-key count and failure boundary. That keeps an interrupted acquisition from masquerading as a deterministic-routing regression.
Exactly — and the way I would enforce that invariant is to make completeness a field on the fingerprint record, not a property the comparison step has to infer. A fingerprint comparison only runs when both runs are marked complete; an interrupted run never produces a comparable fingerprint in the first place. Its summary is a different shape: admitted-key count, failure boundary — where acquisition stopped, the last key admitted, the error — and the fetch-outcome record. Comparing that against a complete run's fingerprint would not find a regression, it would manufacture one.
One boundary case worth pinning down: a run that finished fetching but was interrupted mid-processing must not be compared either. Its fingerprint covers admitted keys only, and treating it as complete hides the unprocessed tail behind a matching prefix. So "complete" has to mean both fetch complete and partition complete, not just "the run ended."
The interrupted run still has to surface — just under a different alert class. "Acquisition incomplete" is an operations alert, "routing regression" is a code alert, and if the two share a channel, someone will eventually tune the router to fix a fetch outage. Keeping them apart is what makes the fingerprint comparison trustworthy when it does run: a mismatch then means the routing actually changed, not that one side of the comparison never finished collecting.
I hit the other side of this with email. Each send was its own queued job, so one bad address never held up the rest. But an address with a cut-short domain, like
name@hotmail, kept getting retried until the job's window ran out, because the provider answered with a validation error and my job retried it like any other failure.The fix was sorting the errors. An address the provider calls invalid now fails once, gets marked bounced and is skipped after that. Rate limits and outages still retry.
That is the exact classification the batch version of this needs too, and your email case shows why the classes have to be three, not two. A provider validation error is permanent: fail once, mark bounced, never retry. A 429 or 5xx is transient: retry with backoff. But the third class is the one that burns people — accepted-then-dropped-response, where the send may already have happened and the response never arrived. That one must never auto-retry; it goes to review, because retrying it is how you send the same email twice.
Two practical notes from doing this in batch loops. First, the retry budget should be per class, not just per job: your cut-short domain kept retrying until the window ran out because the budget was time-based, and a permanent error spent the whole budget. Permanent errors should spend zero retries. Second, mark the bounce per address plus reason, not as a hard delete — if the user corrects the address later, you want a path back in, and the bounce record is what tells you the correction is new rather than another retry of the same bad input.
A bounce doesn't delete anything in mine. The user just gets a bounced flag and a timestamp, and since the email jobs already filter on it, that one write stops all of them.
The path back is where your note caught me. The reason only goes into a log line, and changing the email doesn't clear the flag. So a corrected address stays muted until I clear it by hand in my admin.
Your third class isn't handled either... a timeout still retries like an outage.
You found the two holes in my earlier note, and both have the same root cause: the bounce state is stored in the wrong shape.
First, the path back. A bare flag plus a reason buried in a log line cannot recover, because the flag does not know what it is about. Store the bounce as a record keyed by (user, address) with reason code, provider code, and timestamp in the same row — not a flag on the user and a log line somewhere else. Then a corrected address is simply a different key with no record, so it starts clean, while the old address record stays for audit. If you must keep a user-level flag, the rule should be: an email-change event clears the flag only for the new address, never by hand in admin.
Second, the timeout class. A timeout is not an outage and must not retry like one — the send may already have happened. Give it zero automatic resends. Park it in review with attempt count and last observed response (or the lack of one), and let the recovery step be a status check against the provider, not another send. Outage retries are for cases where you know nothing was accepted; a timeout is the one case where you know the least, so it should get the most caution, not the same retry loop.
The shape of the record decides whether recovery is possible. A flag can only say "stop"; a keyed record can say "stop this address, for this reason, until this changes."
Where do you draw the line between skipping a bad row and pausing the batch? A malformed phone number should only block that lead, but an expired CRM credential could send all 500 into the failure list. I'd want to pause on that shared failure and leave the remaining rows marked as not attempted.
Good line to draw, and I would draw it by error source, not by count alone. A malformed phone number is tied to one item's payload, so it skips just that row into the dead-letter list even if it happens fifty times — fifty bad imports are a data problem, not a reason to stop the good rows behind them. An expired credential, a 401, a provider outage is shared: it says nothing about the next row except that it will fail the same way.
My rule in practice is a breaker on shared error class: a small run of consecutive failures with the same provider/auth/transport error — say three to five — pauses admission. The rows already attempted keep their verdicts, and every row not yet attempted gets marked not-attempted, which is its own bucket: not failed, because failed is a terminal per-item verdict, and not skipped-as-duplicate. Pausing must also emit a report immediately — paused at row k, attempted x, not-attempted y — or you have rebuilt the silence this whole pattern is trying to kill.
Resume is explicit, not scheduled: only after the shared cause is fixed, and only the not-attempted rows plus the dead-letter retries re-enter, through the same dedup check. If a scheduled run auto-resumes into the same expired credential, it just converts 500 not-attempted rows into 500 failed rows, and the distinction you wanted to preserve is gone.
That answers it, thanks. "Not attempted" tells the person recovering the batch something useful: these rows don't need fixing, the connection does. I'd add this distinction to the article—the instruction to always finish the batch reads differently with that exception spelled out.
You are right, and that wording is worth fixing in the article itself: "finish the batch" should mean every admitted row ends in exactly one bucket — processed, failed, skipped, or not-attempted — not that every row gets attempted. A paused run is finished in the reporting sense: the attempted rows have verdicts, the rest are explicitly not-attempted with the shared cause attached, and nothing is silently unaccounted for.
The exception to spell out is that the pause verdict replaces the per-row verdict, it does not suspend it: the run's report is complete, and resume only re-enters the not-attempted rows plus dead-letter retries, never re-runs the rows that already have verdicts. Without that sentence, "always finish the batch" does read like "never pause," which is the opposite of what the breaker is for.
catching the error as data inside the loop instead of letting it throw is the only way that survives untrusted input. returning a structured result object with an explicit error field for that index lets the downstream aggregator write out the 453 good rows and dump the bad one to a dead letter table with the payload attached, so you can inspect row 47 without the batch grinding to a halt.
Exactly — and the detail that makes error-as-data work is that the result object has to carry the item's business key, not just its position in the batch. An index alone is useless once the aggregator reorders or retries: when row 47 fails, the dead-letter row needs the lead's own id plus the raw payload, so the retry workflow can reprocess that one lead without re-running the batch or re-fetching from the source.
One second-order trap worth naming: the aggregator itself must not throw on a malformed failure record. If the error object for one row is missing a field and the aggregator crashes on it, you have rebuilt the same bug one layer up — one bad failure record kills the 453 good results. I validate the failure record's shape separately from the item's, so a failure can never fail the aggregation.
Thank you for sharing such an insightful piece, cheers!