Trust, but Verify: How I Test n8n LLM Workflows Before They Touch Production

Shipping an LLM workflow without tests feels like deploying code without CI — except you can't assert on exact strings, because the model never says the same thing twice. Here's the test harness I run on every n8n LLM workflow before it touches production. It doesn't test vibes. It tests the mechanical guarantees: grounding, fallback, exactly-once, and cost.

This is the testing layer behind the patterns from my last few posts — the evidence-quote gate, the crash boundary, the mini-first cascade. Each of those posts described a mechanism. This post is about how I prove the mechanism works, before a real meeting transcript flows through it.

The fixture pack

Everything starts with three canned transcripts, checked into the repo next to the workflow JSON:

1. Clean — a well-structured standup: named speakers, explicit owners, explicit deadlines. Everything should pass on the first mini attempt.
2. Messy — overlapping speakers, an implied deadline ("let's get it done before the launch"), one action item with no clear owner. The flagship should see some of this; some of it should land in the review queue.
3. Adversarial — a transcript where someone talks about doing something but never commits ("we should probably look into the API rate limits at some point"). The correct output is zero action items. A workflow that extracts an item here has failed the most important test.

Fixed inputs, known-good outputs. Deterministic where it matters.

Test 1: The gate is an assertion

The evidence-quote gate from my exactly-once series is secretly a unit test waiting to happen. I run it standalone against canned LLM outputs, not live model calls:

// n8n Code node test: given this transcript + this canned model output,
// the gate must keep item 1 and drop item 2 (no verbatim quote)
const transcript = $input.first().json.transcript;
const items = $input.first().json.modelOutput; // canned, not live
const kept = items.filter(i => transcript.includes(i.quote));
return kept.map(i => ({ json: i }));

I keep a small table of canned outputs — valid quote, paraphrased quote (must drop), quote from a different meeting (must drop), missing quote (must drop). The gate is pure string matching, so this test is fully deterministic. If someone edits the gate logic and a paraphrase starts passing, the test catches it. The LLM is nondeterministic; the gate must not be.

Test 2: The fallback matrix

A reader asked on the forum what happens when every provider fails — fair question, and it's exactly the kind of thing you test by forcing it. I have a test mode where each LLM node is pointed at a dead endpoint, one at a time, then all at once:

• Kill provider A → the workflow must fall back to provider B and the run completes.
• Kill A and B → the mini-first cascade must still route the hard items to the last-resort provider, and the run completes.
• Kill everything → the run must fail loudly: error branch, alert node fires, nothing silently half-writes.

That last case is the one people skip. A workflow that fails silently when all providers are down will corrupt your task manager with partial results at 3 AM. I'd rather get paged than get phantom tasks. The test asserts the alert fired, not that the extraction was good — there is no extraction when everything is down.

Test 3: The crash test

Kill the execution mid-run — literally stop the n8n execution while it's halfway through a meeting — then rerun the same transcript. The assertions:

• No duplicate action items (idempotency: upsert by meeting ID + item hash, from part 2 of the series).
• The items that were already extracted before the crash are not re-extracted at cost (rerun budget).
• The run log shows the retry, not a silent overwrite.

I do this by hand once per workflow version, not in CI — it's a five-minute test and it exercises the one failure mode that costs real money twice (re-spend + duplicates). The crash boundary post covers the mechanism; this is the proof.

Test 4: The cost rehearsal

Before any new workflow version goes live, I run the messy fixture through it and check two numbers against the budgets from the cascade post:

• Escalation rate: what fraction of items went to the flagship? If it's past the one-third line, the prompt is wrong, not the models. Fix the prompt, don't widen the budget.
• Per-meeting cost ceiling: total tokens × worst-case price. The number must fit inside the escalation budget. If it doesn't, the workflow doesn't ship.

This is the closest thing I have to a load test. It doesn't predict production cost exactly — real meetings are messier than fixtures — but it catches the catastrophic cases: a prompt change that doubles escalations, a retry loop that slipped its bound.

The pre-publish checklist

Every workflow version gets the same runbook, in order:

1. Run all three fixtures. Clean → everything passes first try. Adversarial → zero items.
2. Force-fail each provider in turn, then all of them. Assert fallback fires, then the loud failure.
3. Crash-and-rerun once. Assert zero duplicates, zero re-spend.
4. Run the messy fixture, check escalation rate and cost ceiling.
5. Ship.

Takes about twenty minutes. The first time I skipped step 2, a provider outage at 2 AM taught me why it's there. The lesson cost more than the test ever will.

What I don't test

Output quality — whether an action item is well-written, whether the summary reads nicely. That's subjective, and the review queue from the cascade post is where it gets handled: ambiguous items go to a human, always. My tests verify the mechanical contract (grounded, deduplicated, bounded, loud on failure). The human verifies the prose. Splitting it this way keeps the tests deterministic and the review queue small — it only sees the genuinely ambiguous cases, not the mechanical ones.

I'm building AI Automation Lab — n8n workflow templates for AI automation — and keeping a free lab on GitHub. I also send occasional notes on n8n reliability patterns: get them here.