This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.
The change was small. The queue was not empty.
I once edited a live workflow in the middle of the day. The change was small: one extra field on an outgoing message, one renamed step. I saved, reactivated, and moved on. An hour later the support thread started: some messages went out in the old format, some in the new one, and one job ran twice — once on each version — because it had been picked up, paused on a wait, and resumed after the deploy.
Nothing had failed. Every execution log said success. That is what makes a bad deploy in a workflow system so hard to spot: the deploy does not crash the running jobs. It changes the ground under them while they are still running.
A workflow deploy is not like deploying a stateless web page. A web request finishes in milliseconds, so a new version only affects new requests. A workflow job can live for minutes, hours, or days — waiting on a webhook, a human approval, a retry timer. When you change the workflow, those in-flight jobs are still out there, and they will finish under rules that no longer exist in your editor.
Rule 1: snapshot before you touch anything
Before any change to a live workflow, export the current version and keep it somewhere you can find in a hurry. Not "I can rebuild it" — an actual exported file, with a date and a one-line note about what it does: inbox-triage-2026-10-31-before-notify-field.json.
This sounds like overhead until the first time you need to roll back at midnight. Rebuilding from memory under pressure produces a third version, not the old one, and the third version has its own surprises. A snapshot makes rollback a single action — reactivate the old export — instead of a reconstruction project.
I keep one snapshot per change, not per day. If a change is too big to describe in one line, it is too big to deploy in one step, which brings me to the next rule.
Rule 2: drain before you deploy
The deploy sequence that finally stopped biting me has a boring middle step: stop new work from entering before changing anything.
- Pause the trigger — deactivate the workflow, or pause the schedule, webhook, or queue consumer feeding it.
- Wait for in-flight executions to finish. Check the execution list, not your memory of how long jobs "usually" take. A job on a long wait or a slow retry timer is still in flight.
- Check the stuck places: the dead-letter queue, the review queue, anything holding jobs that did not finish cleanly. A job sitting in review is a job that will resume under the new version if you deploy now.
Only then change the workflow. This is the step people skip because pausing feels like downtime. It is not downtime; it is the deploy. The alternative — mixed old and new runs, a job resuming into renamed steps — costs far more time to untangle than the pause ever did.
How long to wait is a real question, and the honest answer is: until the in-flight count is zero, or until you have explicitly decided what happens to the jobs that are still running. "Deploy and hope the long jobs are fine" is not a decision, it is the absence of one. If a job genuinely cannot finish before your window closes, that job gets a note in the change record: which job, which version it started on, and what you will check when it completes.
Rule 3: a canary job before the flood
After deploying, do not re-enable the trigger and walk away. Run one real job through the new version first — not a unit test of one node, a full end-to-end job with a harmless payload. A test record, a message to yourself, a dry-run flag on the real path.
The canary catches the class of mistake that tests miss: the renamed field that the downstream system actually reads, the step order that only matters when the full payload is present, the credential that works in the editor and fails on the live trigger. I have caught all three with a single canary run, each time before real traffic hit the new version.
Only when the canary's output matches what the old version would have produced — or matches the intended change, if the change was deliberate — do I re-enable the trigger. If the canary fails, the rollback is the snapshot from Rule 1, and the failed canary is evidence, not embarrassment: it did its job.
Rule 4: know what rollback cannot undo
Rollback restores the workflow definition. It does not un-send the messages, un-charge the charges, or un-create the records that the new version already produced. Those are side effects, and side effects do not roll back — which is exactly why the idempotency piece in this series matters: a job that ran on the new version and then gets replayed on the old version must not double-apply its side effects.
So the rollback plan has two halves: reactivate the old version, and list the jobs that ran on the new version so their outputs can be checked by hand. If that list is empty because the canary caught the problem, good. If it is not empty, those jobs get reviewed before anything else resumes. A deploy checklist that only says "revert if broken" is half a plan.
The checklist I actually use
- Export and label the current version.
- Pause the trigger; confirm in-flight executions are at zero or explicitly accounted for.
- Check the dead-letter and review queues.
- Deploy the change.
- Run one full canary job; compare its output to the expected result.
- Re-enable the trigger.
- Watch the first real jobs, not just the first execution log.
It takes maybe ten minutes on a small workflow, longer on one with long waits. It has saved me from the mixed-version hour more than once, and it is the reason I no longer edit a live workflow "quickly" in the middle of the day.
The principle
A workflow is not just code; it is code plus every job currently running through it. A deploy that ignores the running jobs is not a small change, no matter how small the diff looks. Snapshot first, drain, canary, and plan a rollback that accounts for side effects already out the door. The running jobs are part of the system you are changing — deploy like it.
*I'm building AI Automation Lab — production-ready n8n workflow templates. The free AI Hook Generator shows how I structure real workflows; my open-source templates live at github.com/zhp910318/n8n-ai-automation-lab.
Top comments (0)