Why I'm betting on detection lag over uptime for automated pipeline health

The problem with "zero errors"

I run several automated pipelines: a daily article publisher, a YouTube video generator, an ETL that pulls from HuggingFace and Steam — all of them on GitHub Actions schedule triggers. For a long time I treated green CI as the signal that things were working. A job that completes with exit code 0 is a healthy job, right?

Not quite. Over the past several months I've been building a pdca-baseline — a document tracking every known failure in this project, with the commit that introduced it and the commit that found it. When I totaled up the numbers, five separate failures had run undetected for weeks or months:

The common thread: the job kept producing a well-formed file. An empty array is valid JSON. A 403 that returns null is swallowed by the catch block. A green workflow icon does not mean content shipped.

I was tracking uptime (100%) and error rate (0%) the whole time.

What detection lag actually measures

Detection lag is the number of days between the commit that introduced a fault and the commit that found it. You can calculate it from git log without any monitoring infrastructure — just find the two commits and subtract.

The interesting property of this metric is that it rewards a different kind of engineering caution. Uptime pushes you to keep the job running, even if what it's producing is wrong. Detection lag pushes you to check the output, not just the process.

The fault that went 113 days undetected wasn't a crash. The Steam price scraper kept returning valid JSON with a fresh timestamp and the right field names. The value was wrong — it was silently reading the discounted price instead of the list price — but nothing in the pipeline validated values, only structure. Green exit code, wrong data, for nearly four months.

The 91-day case was similar in shape. Articles were being published, cross-posting was working, the pipeline was "healthy." Zero actual humans were reading the output for three months. That failure doesn't show up in any process metric because the process was working. The process was delivering content to a broken distribution path.

This is the failure mode that three approaches to silent failure detection tries to address — but naming the metric gives you something concrete to track over time, not just fix in one-offs.

The counter-example that sharpened the argument

One failure was caught in approximately two days. A fabricated number made it into an article draft — a claim I had no commit-backed evidence for. The independent review step, reading the staged file, flagged it as an unverified metric. Found and removed in two days.

That's the contrast that matters. A wrong sentence in a markdown draft got caught in two days. A wrong number in a database field ran for 113 days unchallenged.

The difference wasn't monitoring sophistication. The code-review step was reading the output and asking "does this claim have evidence?" — not watching a process and asking "did the job complete?" I've written about this distinction in Why I'm betting output inspection beats process monitoring, but detection lag is the measurable version of that observation. It's what you get when you run the experiment on your own history.

The pattern also connects to what happened with survivorship bias in my analytics: videos I manually deleted stayed out of the sample, making the channel look like it was improving. The pipeline was running. The output was wrong in a subtle way. The full account is in How I fixed survivorship bias in my YouTube analytics — that one took a systematic audit, not an alert, to catch.

The falsifiable bet

Here's the version I can be held to: for solo-developer automation pipelines that produce files rather than expose APIs, detection lag measured from commit history will average more than 30 days across all faults found in a six-month window.

The mechanism behind that prediction: pipelines that write files have no live user traffic to surface wrong values. Nobody sends a 400 error when a JSON file contains the wrong price. Nobody complains when an article is published to an empty audience. The feedback loop is severed at the output boundary.

If that's true, the monitoring question changes. It isn't "did the job fail?" — that question has a real-time answer you can alert on. It's "how long before someone noticed?" — and that question requires reading what the pipeline actually wrote.

One cheap heuristic that doesn't require code: if a file that's supposed to change daily hasn't changed in five days, that's worth a look. That single check would have caught at least two of the five failures above. I've been applying a version of this with git-based diffing of weekly data files.

An edge AI parallel: designing for a known detection lag

The shelf-scanning system I'm building on a Raspberry Pi 3 runs at a median of 8.5 seconds per inference scan (n=19 real scans; range 8.4–11.8 s). On a Pi 3, that's honest — you don't get fast inference without hardware that costs more.

The pipeline design acknowledges it. The system runs hourly, and it doesn't confirm a detection until the same gap appears in at least 2 of the last 3 scans — a temporal majority vote. That means the maximum detection lag for a real gap is three scan cycles, or three hours. False positive rate drops because a camera shake or bad frame doesn't trip the alert.

The tradeoff is explicit: three hours to confirm an empty shelf is the chosen bound. The 8.5 s inference time is acceptable because the cadence is hourly, not real-time. I wrote more about the post-processing design in Three layers that made the shelf detector useful on a Pi 3.

That's a better posture than my publishing pipeline had. The shelf detector has a named detection lag bound. My ETL pipeline had an implicit detection lag of whatever I happened to notice, which turned out to be 113 days.

The detection accuracy on the held-out test set is mAP50 0.844. That sounds solid, but it's the model accuracy — the detection lag is a property of the scheduling and voting logic, not the model. You can have a 0.99 mAP model that still takes three months to report a fault if nobody checks the output.

What would change my mind

I'd update this bet if I found a practical process-monitoring setup that reliably catches silent output failures without reading the output. I haven't found one yet — to know that a JSON file contains wrong values, you have to read and validate the values, which means output checking. But if Datadog or Grafana or a similar platform has a mode that samples output content and checks semantic validity, not just "did the file update?", I'd look at it seriously.

I'd also revise if the cost of output sampling grew high enough to avoid. Right now the checks are cheap: was the file updated? Is the array non-empty? Do the values pass basic plausibility tests (price > 0, title length > 0)? Those are cheap enough that skipping them is hard to defend. If the validation needed to catch failures became expensive, that would shift the math.

The third scenario: if pipeline architecture improved to make silent failures structurally impossible — say, by making every write a validated, schema-checked transaction — detection lag becomes less interesting because the failure mode is designed away. I'd take that as a win. The metric matters because the failure mode currently exists, not as an end in itself.

Measuring it going forward

I'm treating detection lag as the primary lagging indicator of pipeline health quality across this project. Each time a new failure is documented in the pdca-baseline, I record fault-introduction commit, fault-detection commit, days elapsed. If that number trends down over time, the output-checking strategy is working. The pdca-prediction patterns analysis has the earlier framing of this — detection lag is the number I'm watching now.

The number I'd most like to report in 30 days: whether the five cases above are outliers or representative for this class of pipeline. I don't know yet. The data will tell.

FAQ

How do you calculate detection lag if git history doesn't perfectly align with deploy timestamps?

I use commit timestamps, not deploy times. Fault-introduction is the commit that introduced the bug; detection is the commit that documents finding it or the first fix commit. For this project those are usually the same event. It's accurate to within a day or two — close enough to be useful as a trend metric.

What about failures caught and fixed immediately?

Those count, and they're the healthy signal. An average detection lag genuinely under 7 days would mean the output-checking strategy is working. I track all detected failures, not just the slow ones.

My pipelines don't have month-long failures. Is the 113-day case an anomaly?

Maybe. My hypothesis is that file-producing pipelines are more susceptible than API-serving ones, because files have no live error feedback from downstream consumers. If your pipeline serves live traffic and users report issues, you probably catch failures in days not months. Mine don't serve live user traffic in that way.

How does this relate to survivorship bias in analytics?

Directly. The deleted-video survivorship bias ran because the analytics pipeline was producing reports that looked plausible. The flaw was only visible when comparing the pre-deletion and post-deletion samples. That's output validation — comparing what the pipeline said to a ground truth the process logs couldn't see. Same mechanism as the detection-lag problem.

Can you prevent this with better schema validation?

Partially. Schema validation catches structural errors — missing fields, wrong types, empty arrays. It doesn't catch semantic errors. The Steam price was the right type (number), within a plausible range, just sourced from the wrong field. Catching that requires domain-specific output validation. Schema is a floor, not a ceiling.

---

Related reading: Why I'm betting output inspection beats process monitoring · Three PDCA patterns that closed a 92-day detection lag · Three approaches to silent failure detection in GitHub Actions

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.