Scenario S7 of my v0.2.0 field test had a pipeline that "ran end to end." The unit suite was green. The container scenario timed out at 180 seconds.
The pipeline had created every child run, returned each id, and never started a single one. The children sat queued forever while the parent reported success. Everything I asserted — the status code, the persisted record — was true. The behavior I cared about never happened.
That was the moment I stopped trusting the 200.
I found this defect class four times in one field test of HivePlane v0.2.0. Every instance was the same shape: a test that proved the plumbing and not the outcome. Every one was invisible to the unit suite because the unit suite stopped at the seam.
The shape
Four findings. One shape. A test that proves the plumbing and not the outcome.
| Finding | What the green test proved | What was actually true |
|---|---|---|
| D-3, pipeline never completes | The child run existed (201, persisted) |
It never transitioned queued → running; nothing started it |
| D-4, fan-out invisible | The delivery hit the webhook sink |
GET /delivery/audit read a different store; the delivery was unauditable |
| #558, approval resolve | The decision record was persisted | The paused run never resumed — resolve() was a no-op lambda |
| #575, lease reclaim | A reassignment was recorded | No lock, no leader gate — concurrent reclaims double-assigned the same run |
The unit tests stopped at the seam. The bug lived one boundary further.
The pipeline that never started
RunNodeExecutor.submit did this:
run = self._runs.submit(workload=..., pipeline_origin=origin, ctx=ctx)
return run.id
That's it. It created the child and returned. A run is dispatched to its adapter only on the queued → running transition. In the Docker profile there's no worker daemon to lease queued runs, and the pipeline driver only reconciled nodes that were already running. So the parent waited on a child that nothing would launch — a deadlock dressed as a success.
The fix was one line, and it's the line that should have been there:
run = self._runs.submit(...)
if run.state is RunState.QUEUED:
self._runs.start(run.id, actor="pipeline", ctx=ctx)
return run.id
The regression test — test_pipeline_starts_its_child_runs — asserts the child transitions to running. It failed before, passes after. Why didn't I have it? Because my unit test used a synchronous executor that hid the missing start. The synchronous path never needs a worker, so the absence of one was invisible. Only the live stack, with no worker, exposed it. The Docker report's §3.3 traces the exact transition, and the orchestration design documents the pipeline execution model.
tip: Any component that produces child runs — pipelines, triggers, certification — must either start them or rely on a worker that leases them. If your test executor is synchronous, it's hiding the exact bug you're shipping.
The delivery that succeeded in the dark
D-4 was sneakier. The fan-out delivered. The webhook sink logged the run.completed payload. The attempts were written to fan_out_deliveries with status delivered.
But GET /delivery/audit — the surface my scenario polled — reads the M51 delivery_attempts store. Two subsystems, one concept, two stores. A successful delivery was unauditable through the control plane. My test asserted the 200 and the empty list and moved on, because the empty list was "no deliveries," which was plausible.
The unused _WEBHOOK_SINK constant in the test was the tell. It was checking the wrong surface. The fix wasn't to the delivery — it was to add GET /runs/{run_id}/deliveries so the audit surface references the store that actually receives the data.
tip: When two subsystems share a concept, your audit endpoint must name which store it reads. "Delivery" meant two things, and the test picked the one that was empty.
The approval that recorded a decision and resumed nothing
This one scared me the most. Interactive and mobile approvals — Slack buttons, email links — had a resolve() wired to a no-op lambda:
# the wire that shipped
approval.resolve = lambda decision: None # persisted the decision, touched nothing
The decision was persisted to approval_decisions. The test asserted the row existed. The test passed. The paused run never resumed. An operator clicked Approve, saw the confirmation, and the run stayed paused forever.
The green test proved a row was written. It did not prove the run moved. The fix: resolve() calls RunService.resume(run_id) — the side effect the operator actually cares about. The regression test asserts the run transitions to running, not that a decision record exists.
tip: The most dangerous "green" test is the one that asserts the record and not the effect. An approval that persists without resuming is a no-op dressed as a workflow. Ask your test: "what did the operator see?" Then assert that.
What I learned
The unit tests stopped at the seam. The bug lived one boundary further — and the synchronous test executor was the reason the seam was invisible. A live stack with no worker, no shortcut, is the only honest witness.
The v0.2.0 field test came back 36/36 scenarios, 67/67 container tests, 50/50 load test — but only because I re-ran it end to end against the live stack and stopped believing my green unit suite. The full receipts are in the field-test report and the Docker test report.
References
-
HivePlane —
pip install hiveplane==0.2.0 - Field test report (36/36) · Docker report (67/67)
- Orchestration design
- v0.2.0 release notes
What's the most expensive "green" test in your suite — the one that asserts the 200 and the row but not the side effect? Go look. I'll wait.
Top comments (7)
The D-3 case is the one I recognise. In my ticket tracker I stopped storing an agent's status as working for a similar reason: a process that dies writes nothing, so the stored row keeps looking true. Now the orchestrator beats every 60 seconds and every screen derives health from the age of the last beat, so silence turns red on its own. Did the field test change what HivePlane treats as proof that a run started, or only which tests you trust?
Thanks, Panth — both, and the first matters more. It changed the definition of "started": the old proof was "a run row exists (201)"; the new proof is "the child reached
runningviaRunService.start," which is why the regression asserts the state transition instead of the 200 + row. Your heartbeat idea is the same principle — derive health from a fresh signal, so silence can turn red on its own.Asserting the transition instead of the 201 is the better contract. The row only proves the request landed. It leaves one gap I'd still watch: a child that reaches running and then stalls there. Does the regression stop at running, or does it also wait for a terminal state with a timeout?
D-3 is also a liveness bug, and a test can't prove liveness by asserting one snapshot. Two things that would have caught it without a live stack: derive the parent's "success" from its children reaching a terminal state (so a queued child can never read as done), and give the pipeline test a deadline instead of a status assert: every child leaves "queued" within N seconds, using an executor that does not run inline.
To check the regression test is worth keeping, delete the start() line and confirm test_pipeline_starts_its_child_runs fails, then do the same for the resolve() no-op. A test that stays green with the fix removed was never testing the fix, and that is the shape you found four times.
In production the same invariant is a cheap alarm: count runs older than N minutes still queued and page on non-zero. It would have turned the 180 second timeout in the container scenario into a metric.
Thanks — the deletion test is exactly what the v0.2.0 regression does: remove the
start()line andtest_pipeline_starts_its_child_runsgoes red, and the same forresolve()on the approval path. That "a test that stays green with the fix removed was never testing the fix" line is the one I'd frame. The production alarm — count queued runs older than N minutes and page on non-zero — is a great addition; the container scenario only surfaced it as a 180-second timeout, and it should have been a metric.Synchronous test executors hide the exact race where state transitions stall. In-memory runners execute the task inline on the calling thread, so the test checks the submission receipt and never touches the lease worker or the queue poller.
The approval case is especially familiar. We had a webhook handler that saved signed approval tokens into Postgres and returned a clean 200. The regression test asserted the token row was active and passed every run. In staging, the background worker picked up the token but died silently because the task payload had an outdated schema from a previous migration. The user saw 'Approved' on their dashboard, the database had the record, and the job never resumed.
Since then, every pipeline test has to wait for a state transition event emitted by the downstream worker rather than asserting against the control plane table.
Thanks, Reid — your webhook-handler story is the same defect wearing a different coat. In HivePlane the sync executor never needed a worker, so
RunNodeExecutor.submitcreating the child and returning its id looked fine — the children satqueuedforever while the parent reported success. The fix was makingsubmitcall_runs.start(...)on thequeued → runningtransition, and the regression test asserts the transition, not the row. "Wait for the downstream worker's event, not the control-plane table" is exactly the fix in one sentence.