Idempotency Should Resume Partial Work, Not Hide It

Idempotency is usually introduced with a simple promise: run the operation again and it will not create a duplicate.

That promise is useful, but it can become dangerous when we apply it to an entire workflow instead of the specific durable step it protects.

I recently reviewed a recurring integration job with two independent writes. First, it appended a status note to an external record. Second, it moved that record into a terminal state. The note contained a unique marker, so retries could see that the note had already been written.

The marker worked. The workflow did not.

When the state transition failed once, the note remained. On the next sweep, the job saw the marker and skipped the record completely. A local sign of success for step one had accidentally become a global completion flag for steps one and two.

That is a subtle reliability bug because the automation appears idempotent while the real process is only half finished.

A checkpoint is not the whole journey

Consider a generalized background job:

1. Write a durable note with a unique operation marker.
2. Move the remote record into a terminal workflow state.
3. Repeat the scan for a bounded period so temporary failures can heal.

The marker proves exactly one fact: the note exists.

It does not prove that the state move succeeded. It does not prove that a notification was delivered, a file was copied, or a later API call completed. Yet it is tempting to write an early return such as “marker found, therefore this item is done.”

The safer model gives each durable step its own completion evidence. In this case, the marker suppresses only another note. Later sweeps still inspect the remote record and decide whether the second step is pending.

This sounds like a small distinction. In production, it is the difference between duplicate prevention and recovery prevention.

Reconciliation makes partial completion explicit

A resumable job expects to encounter intermediate states:

• no note and no terminal state;
• note present but transition still pending;
• note present and record already terminal;
• note present but a person changed the record afterwards;
• remote workflow metadata that no longer matches our assumptions;
• a temporary failure that should be tried again;
• a permanent incompatibility that needs attention.

Those are not exotic edge cases. They are the normal state space of a multi-step integration.

The repaired design runs the state decision on every bounded reconciliation sweep. If the note is missing, it writes the note once. Whether the note was just written or already existed, it then evaluates the transition separately.

That separation makes the control flow honest. We can retry the unfinished part without repeating the finished part.

Ask the external system what its workflow means

The original failure was also a reminder that external state names are rarely universal. One workflow may call its terminal state “Done”; another may use a different label. Hard-coding one friendly name works until a customized process rejects it.

The stronger approach reads the record's type and the available states for that type. It prefers an explicitly configured state when that state exists. Otherwise, it can select a state by semantic category, such as completed or removed.

This adds metadata reads and some branching. It also moves compatibility knowledge to the boundary where it belongs. The integration stops pretending that its vocabulary is the external system's vocabulary.

That fallback must remain bounded. If no suitable category exists, the job should surface the incompatibility rather than inventing a transition.

Human intent outranks automation

Retrying forever is not the same as reconciling safely.

Suppose the automation wrote its note yesterday, then a person deliberately reopened or reclassified the record today. A later sweep should not force the record back into the automation's preferred state.

The reconciliation rule therefore needs ordering evidence. If the external state changed after our durable note, leave it alone. If ordering is unknown, prefer the conservative choice and do not overwrite. If the record is already in any terminal category, there is nothing to repair.

This is an important boundary: the job owns completion of its own intent, not the external workflow itself.

Retry by failure class

Not every failure deserves the same response.

A timeout or temporary service error can be retried by the next sweep. A missing permission or incompatible workflow definition will not improve through repetition. That condition should be reported, with noise controlled, while unrelated records continue.

This classification keeps the job useful during partial outages. One bad record does not starve the batch, and operators are not flooded with alerts for failures that the normal reconciliation loop can heal.

Caching workflow metadata for the duration of one sweep can also limit API traffic without letting old assumptions live indefinitely.

Test the half-finished states

Happy-path tests are not enough for resumable automation. The valuable tests begin after something durable has already happened.

I would want focused coverage proving that:

• an existing marker prevents a duplicate note but not a pending transition;
• a previously failed transition is attempted on a later sweep;
• different state names can resolve through semantic metadata;
• terminal records are left untouched;
• a newer human change is preserved;
• unknown ordering is handled conservatively;
• one record's failure does not block the others;
• temporary and permanent failures take different paths.

These tests describe the recovery contract more clearly than a mock verifying one ideal call sequence.

The practical rule

For every multi-step job, list the durable evidence produced by each step. Then ask what the next run should do when execution stops after any one of them.

Idempotency should let a workflow resume safely. If an “already processed” check prevents the system from finishing later steps, it is not protecting reliability. It is hiding incomplete work.