One Bad Row Killed the Whole Batch: Partial-Failure Handling for n8n Loops
This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.
The batch failed at row 47. Rows 48 to 500 never ran
A nightly workflow of mine processed a batch of new leads: enrich each one, score it, write it to the CRM. One night a single lead had a malformed phone field, the enrichment node threw, and the execution stopped. The other 453 leads sat unprocessed until the next morning, when a client asked why half the day's leads were missing.
Nothing about this was exotic. A batch loop that stops on the first error is the default behavior in most workflow tools, and it turns one bad record into a silent outage for every record behind it. The error itself was visible — the execution log showed the failure. What was invisible was the cost: hundreds of good records that never got processed because they happened to queue behind a bad one.
The fix is not "validate better." You will never validate away every malformed record from a source you do not control. The fix is to decide, per item, what failure means — and to make that decision before the loop runs, not after it stops.
Three rules I now build into every batch loop
First, isolate failures to the item that caused them. In n8n this means the per-item work runs where an error cannot kill the loop: a sub-workflow executed per item, or an error-handling branch that catches the failure and returns it as data instead of throwing. The loop itself never sees an exception — it sees a result, either success or a structured failure record. Once a bad row produces a row in a failure list instead of a stopped execution, the batch always completes, and "complete" stops being a lie.
Second, collect failures into a dead-letter list, not a log line. A failed item goes into a table with the item's business key, the error, and the raw payload — everything needed to reprocess it without re-running the whole batch. My rule: a batch that ends with failures still reports success for the good items and a separate count for the failed ones, and the failed list is the input to a retry workflow, not a note for someone to read someday. If a failure cannot be retried from the list alone, the list is missing a field, and that is the bug to fix.
Third, report four numbers, not one status. Processed, failed, skipped as duplicates, and retried. When failed jumps from zero to one, that is a data problem worth one look. When failed jumps to forty, that is a schema change upstream, and I want to know before the client does. A batch that reports only "done" hides exactly the distinction between "everything worked" and "everything after row 47 never ran" — which is the distinction the whole report exists to make.
The retry rule that keeps this honest
Retrying the whole batch is how a partial failure becomes a duplicate problem: the good items run twice. So retries run against the dead-letter list only, keyed by the business id, with the same dedupe check the main loop uses. A retried item that succeeds is marked done; one that fails again stays on the list with its new error. Nothing ever re-enters the main batch, and the main batch never re-processes a good item because a bad one showed up late.
The principle is simple: a batch loop's job is to finish the batch, not to stop at the first surprise. One bad row should cost you one row, one entry in a list, and one retry — never the 453 good rows standing behind it.
I'm building AI Automation Lab — production-ready n8n workflow templates. The free AI Hook Generator shows how I structure real workflows; my open-source templates live at github.com/zhp910318/n8n-ai-automation-lab.
Top comments (35)
one bad row shouldn't eat 453 good ones.
That is the whole point in one line. The part that stings is the silence: a batch that stops at row 47 usually still reports "failed" for the one row, but nothing reports the 453 that never ran. So the fix that matters most is arguably not the isolation itself but the report — processed, failed, skipped, retried — because "failed: 1" and "failed: 1, never attempted: 453" are two very different mornings.
yeah, the silence is the worst part. no error, just stops.
Exactly — and the fix for the silence is to make "no error" never count as a result. Two rules I use now:
The error was never the missing exception. It was a loop that could finish without ever saying what it did.
zero rows processed counting as a failure, that's the one I'd keep.
That's the one I'd keep too — but with one refinement: zero processed is only a failure when you expected input. A run that legitimately had nothing to do (no new orders overnight, an empty source) and pages on it will train people to ignore the page, which is how the next real silence gets missed.
So I record three numbers, not one: rows fetched, rows processed, rows skipped. The rule becomes:
The implementation detail that makes this work: the fetch step must never return an empty list on error. An error returns an error, not zero rows. If the only thing downstream ever sees is a list, "failed to fetch" and "fetched nothing" are indistinguishable, and no summary format can save you.
Two drills worth running: one with a genuinely empty source, one with the fetch broken mid-pull. They must produce distinguishable reports — not the same "done, 0 rows". If they don't, the zero-row rule is decoration, not a check.
error returning an empty list is the trap, that's the distinction that matters.
Exactly — and the reason it keeps biting is that the trap is in the type, not the logic. If the fetch step returns just a list, there is no slot for "I failed": the error path has to put something in that slot, and [] is the easiest something. Downstream code that only ever sees a list can never recover the distinction, no matter how careful the summary format is.
So I make the fetch step return a result that can say "failed" as a first-class value — either it throws, or it returns a discriminated result with an error branch — never a bare list. And I test the error branch directly: break the fetch in a test and assert that downstream receives an error, not zero rows. Testing only the happy path plus the genuinely-empty path leaves exactly this trap untested, because both of those return the same shape downstream sees on a swallowed failure.
One more guard that caught this for me: the fetch step emits its own outcome record — source, status, rows returned — before the loop ever sees the list. Then "fetched nothing" is an auditable event with a timestamp, not just a zero in a summary.
failed as a first-class value is the fix, the type carries it, not the logic around it.
Exactly — and the reason the type matters more than the surrounding logic is that a type forces every consumer to handle the error branch, while a convention only asks them to. If the fetch returns a bare list, a new caller can ignore the failure case entirely and the code still compiles and the tests still pass — the trap survives code review because nothing in the shape says "this can fail." A discriminated result flips that: the error branch is part of the return value, so ignoring it is a visible omission, not a silent default.
One practical consequence: when I add a new consumer of the fetch step, I do not have to remember to check the outcome record — the type will not let that code path exist without handling the error. That is also why I keep the outcome record separate from the result itself: the result type says what happened to this call, the record says what the fetch observed, and a test can assert both without conflating them.
the type forcing the branch is the whole point. a convention lets the next caller skip it quietly.
Exactly — and the quiet part is what makes a convention worse than a missing check: the skip leaves no trace. A caller that ignores a failure branch under a convention still produces a clean-looking report, because nothing downstream records that a branch was skipped. The failure only surfaces later, as a missing row nobody can attribute.
That is why I treat the type as a boundary rule, not just a compile-time aid. In a typed consumer, an exhaustive match with no default fallthrough turns an unhandled error branch into a build failure. In an untyped n8n Code node, the equivalent is a runtime shape check at the aggregator: every item must arrive as a result object with an explicit status, and anything else — a bare list, a missing error field — fails the run loudly instead of being counted as processed.
One test that enforces it: add a new consumer that deliberately reads only the success field, feed it a failed fetch, and assert the run fails or the error surfaces in the report. If that test passes silently, the type is decoration, not a guard — the convention has just moved inside the result shape.
the new-consumer test is the part that makes it real. a type that never fails a run is just documentation.
Exactly — and that distinction is why the test belongs in the gate, not in a one-off demo. Documentation describes the intended behaviour; a test proves the system still enforces it after someone changes the code. A type that never fails a run is just a comment with better syntax. Two properties make that test real rather than decorative. First, it has to fail on the old shape: feed a failed fetch through the same path and assert the run fails or the error lands in the report bucket — and verify that the same test fails if the fetch goes back to returning a bare list. A test that only passes on the new shape has not proven anything about the failure path. Second, it has to run through the aggregator, not just the fetch unit. A fetch unit test can pass while the aggregator still counts a malformed result as processed, and then the type is decoration at exactly the layer where the batch decides what counts. The same applies over time: when a new result variant appears, the exhaustive-match test should fail first and only pass once the new branch is handled. If new consumers can be added without that test ever going red, the type has quietly decayed back into a convention.
gate not demo is the right call. a one-off pass proves nothing about the next bad row.
Exactly — a one-off pass is a snapshot, not a guarantee. The next bad row is usually a shape the gate has never seen, so the gate has to prove itself on every run, not just on the day it was written.
Two things make that concrete. First, the gate should run a synthetic bad row every time: a fixture with a malformed field, a failed fetch, and a new result variant, fed through the full path — fetch, aggregator, report — not just the unit under test. If that fixture ever stops running, or runs but is allowed to pass, the gate fails closed. A gate that can silently lose its own fixture is just a demo with a schedule.
Second, the gate cannot assume it knows all failure shapes. An unknown variant must not fall through to "processed": it goes to pending-review with its raw payload, and the exhaustive-match check fails until someone handles it. That is what turns the next bad row from a silent pass into a visible, bounded case — the run still completes, the good rows still count, and the new shape gets a decision instead of a default.
A pass that only proves yesterday's bad rows were handled is documentation. A gate that re-proves itself against a bad row it has never seen, every run, is the thing that holds for the next one.
The four counters are valuable because they stop “complete” from hiding a partial result. I would add one invariant to the run summary:
processed + failed + skipped + pending-review = input, with a stable item key behind each count. That makes a missing row detectable immediately and gives the dead-letter retry flow a bounded, auditable input instead of relying on operators to reconstruct what happened from logs.Yes — and I would harden your invariant in one way: a sum is necessary but not sufficient. Processed + failed + skipped + pending-review = input can still hold while the wrong row is missing, if one row is dropped and another is counted twice. So the check I actually enforce is a key-set partition, not just an arithmetic sum: every input key appears in exactly one bucket, no key in two buckets, and no input key unaccounted for. The sum is the cheap alert; the set difference is the proof.
Two details that make that workable in practice. First, 'input' has to mean rows admitted to the loop, not rows the source claims to have. If the fetch itself fails partway, input is unknown, and the invariant cannot rescue it — that is a separate fetch-outcome line (source, status, rows admitted), not a fifth counter to fudge the sum. Second, pending-review must be its own bucket, not folded into failed: a failed row is terminal and retryable from the dead-letter list, while a pending-review row is unresolved — usually an ambiguous send where re-running it could duplicate an effect. Mixing them lets an ambiguous row look resolved.
On the stable key: it has to be a business key, not a row index or position, or a retry that reorders the batch silently re-keys everything. And the dead-letter retry must re-enter through the same dedup check, so a retried row that already succeeded can never be processed twice just because it is back in the input.
Exactly—conservation counts tell you volume; key-set partition tells you identity. I’d also persist the input fingerprint and run ID alongside each bucket, so retries and replays can be tested without confusing duplicate execution for a dropped row.
Exactly — and the split you are drawing is the one that makes replay tests meaningful: fingerprint over the key set for identity, and a separate content hash if you also want to detect mutation. If you fold them into one hash, a legitimately corrected field looks like a dropped row, and your replay test fails for the wrong reason.
Two details I would add. First, the fingerprint has to be computed over admitted keys only, not over what the source claims. If the fetch fails partway, there is no trustworthy fingerprint yet — that run gets a fetch-outcome record, not a fingerprint over a partial set, or a later complete run will never match it and you will chase a phantom difference.
Second, keep origin_run_id and run_id as separate fields on every bucket row. A retry is a new run_id carrying the old origin_run_id and the same item keys. Then a replay test can assert exactly what you want: same origin fingerprint plus same keys gives the same partition, a retried key lands in the same bucket it would have landed in originally, and a duplicate execution shows up as two run_ids sharing one key — not as a missing row. Without that pair, retry and duplicate are indistinguishable in the report, which is the confusion the fingerprint was meant to remove.
That distinction also suggests a useful invariant for the report: only compare partition fingerprints for runs marked complete, while incomplete fetches are summarized by the admitted-key count and failure boundary. That keeps an interrupted acquisition from masquerading as a deterministic-routing regression.
Exactly — and the way I would enforce that invariant is to make completeness a field on the fingerprint record, not a property the comparison step has to infer. A fingerprint comparison only runs when both runs are marked complete; an interrupted run never produces a comparable fingerprint in the first place. Its summary is a different shape: admitted-key count, failure boundary — where acquisition stopped, the last key admitted, the error — and the fetch-outcome record. Comparing that against a complete run's fingerprint would not find a regression, it would manufacture one.
One boundary case worth pinning down: a run that finished fetching but was interrupted mid-processing must not be compared either. Its fingerprint covers admitted keys only, and treating it as complete hides the unprocessed tail behind a matching prefix. So "complete" has to mean both fetch complete and partition complete, not just "the run ended."
The interrupted run still has to surface — just under a different alert class. "Acquisition incomplete" is an operations alert, "routing regression" is a code alert, and if the two share a channel, someone will eventually tune the router to fix a fetch outage. Keeping them apart is what makes the fingerprint comparison trustworthy when it does run: a mismatch then means the routing actually changed, not that one side of the comparison never finished collecting.
The interrupted-processing case is a valuable addition. A test could finish acquisition, stop before the final partition, and assert both that comparison is suppressed and that the incomplete-processing alert contains the boundary. That checks the absence of a false regression and the presence of an actionable signal separately.
Yes — and the key word in your test is "separately." If those two assertions live in one pass/fail check, a run that suppresses comparison but emits a vague alert still looks green, and a run that alerts loudly but compares partial data anyway also looks green. They have to fail independently.
I would make the boundary assertion stricter than "an alert exists": the alert must name the boundary itself — which partition is incomplete, and which items were never admitted — not just that processing stopped. An alert that says "incomplete" without the boundary cannot drive a resume; it can only drive a human re-reading the logs, which is the same failure mode the batch report is supposed to fix.
One more distinction worth pinning in the same test: stopping before the final partition is different from stopping inside it. If the stop lands mid-partition, the alert must not claim that partition as complete, and the resume must re-run that partition from its first item — while still not re-processing the partitions already marked done. That is where the dead-letter / business-key dedupe from the article earns its keep: the resume boundary is a partition id, and the dedupe is what makes re-running that partition safe.
So the test ends up with three independent assertions, not one: no comparison output for partial data, an alert carrying the exact boundary, and a resume that re-runs only from that boundary.
I hit the other side of this with email. Each send was its own queued job, so one bad address never held up the rest. But an address with a cut-short domain, like
name@hotmail, kept getting retried until the job's window ran out, because the provider answered with a validation error and my job retried it like any other failure.The fix was sorting the errors. An address the provider calls invalid now fails once, gets marked bounced and is skipped after that. Rate limits and outages still retry.
That is the exact classification the batch version of this needs too, and your email case shows why the classes have to be three, not two. A provider validation error is permanent: fail once, mark bounced, never retry. A 429 or 5xx is transient: retry with backoff. But the third class is the one that burns people — accepted-then-dropped-response, where the send may already have happened and the response never arrived. That one must never auto-retry; it goes to review, because retrying it is how you send the same email twice.
Two practical notes from doing this in batch loops. First, the retry budget should be per class, not just per job: your cut-short domain kept retrying until the window ran out because the budget was time-based, and a permanent error spent the whole budget. Permanent errors should spend zero retries. Second, mark the bounce per address plus reason, not as a hard delete — if the user corrects the address later, you want a path back in, and the bounce record is what tells you the correction is new rather than another retry of the same bad input.
A bounce doesn't delete anything in mine. The user just gets a bounced flag and a timestamp, and since the email jobs already filter on it, that one write stops all of them.
The path back is where your note caught me. The reason only goes into a log line, and changing the email doesn't clear the flag. So a corrected address stays muted until I clear it by hand in my admin.
Your third class isn't handled either... a timeout still retries like an outage.
You found the two holes in my earlier note, and both have the same root cause: the bounce state is stored in the wrong shape.
First, the path back. A bare flag plus a reason buried in a log line cannot recover, because the flag does not know what it is about. Store the bounce as a record keyed by (user, address) with reason code, provider code, and timestamp in the same row — not a flag on the user and a log line somewhere else. Then a corrected address is simply a different key with no record, so it starts clean, while the old address record stays for audit. If you must keep a user-level flag, the rule should be: an email-change event clears the flag only for the new address, never by hand in admin.
Second, the timeout class. A timeout is not an outage and must not retry like one — the send may already have happened. Give it zero automatic resends. Park it in review with attempt count and last observed response (or the lack of one), and let the recovery step be a status check against the provider, not another send. Outage retries are for cases where you know nothing was accepted; a timeout is the one case where you know the least, so it should get the most caution, not the same retry loop.
The shape of the record decides whether recovery is possible. A flag can only say "stop"; a keyed record can say "stop this address, for this reason, until this changes."
Where do you draw the line between skipping a bad row and pausing the batch? A malformed phone number should only block that lead, but an expired CRM credential could send all 500 into the failure list. I'd want to pause on that shared failure and leave the remaining rows marked as not attempted.
Good line to draw, and I would draw it by error source, not by count alone. A malformed phone number is tied to one item's payload, so it skips just that row into the dead-letter list even if it happens fifty times — fifty bad imports are a data problem, not a reason to stop the good rows behind them. An expired credential, a 401, a provider outage is shared: it says nothing about the next row except that it will fail the same way.
My rule in practice is a breaker on shared error class: a small run of consecutive failures with the same provider/auth/transport error — say three to five — pauses admission. The rows already attempted keep their verdicts, and every row not yet attempted gets marked not-attempted, which is its own bucket: not failed, because failed is a terminal per-item verdict, and not skipped-as-duplicate. Pausing must also emit a report immediately — paused at row k, attempted x, not-attempted y — or you have rebuilt the silence this whole pattern is trying to kill.
Resume is explicit, not scheduled: only after the shared cause is fixed, and only the not-attempted rows plus the dead-letter retries re-enter, through the same dedup check. If a scheduled run auto-resumes into the same expired credential, it just converts 500 not-attempted rows into 500 failed rows, and the distinction you wanted to preserve is gone.
That answers it, thanks. "Not attempted" tells the person recovering the batch something useful: these rows don't need fixing, the connection does. I'd add this distinction to the article—the instruction to always finish the batch reads differently with that exception spelled out.
You are right, and that wording is worth fixing in the article itself: "finish the batch" should mean every admitted row ends in exactly one bucket — processed, failed, skipped, or not-attempted — not that every row gets attempted. A paused run is finished in the reporting sense: the attempted rows have verdicts, the rest are explicitly not-attempted with the shared cause attached, and nothing is silently unaccounted for.
The exception to spell out is that the pause verdict replaces the per-row verdict, it does not suspend it: the run's report is complete, and resume only re-enters the not-attempted rows plus dead-letter retries, never re-runs the rows that already have verdicts. Without that sentence, "always finish the batch" does read like "never pause," which is the opposite of what the breaker is for.
catching the error as data inside the loop instead of letting it throw is the only way that survives untrusted input. returning a structured result object with an explicit error field for that index lets the downstream aggregator write out the 453 good rows and dump the bad one to a dead letter table with the payload attached, so you can inspect row 47 without the batch grinding to a halt.
Exactly — and the detail that makes error-as-data work is that the result object has to carry the item's business key, not just its position in the batch. An index alone is useless once the aggregator reorders or retries: when row 47 fails, the dead-letter row needs the lead's own id plus the raw payload, so the retry workflow can reprocess that one lead without re-running the batch or re-fetching from the source.
One second-order trap worth naming: the aggregator itself must not throw on a malformed failure record. If the error object for one row is missing a field and the aggregator crashes on it, you have rebuilt the same bug one layer up — one bad failure record kills the 453 good results. I validate the failure record's shape separately from the item's, so a failure can never fail the aggregation.
Thank you for sharing such an insightful piece, cheers!