We were proud of the canary. Five percent of traffic for fifteen minutes, automated analysis comparing error rate and p95 latency against the baseline pods, promote or roll back with no human in the loop. It had caught four bad releases in six months. Then it promoted a release that corrupted nine hours of the nightly settlement run, and the canary's verdict had been green with wide margins.
The release changed how a shared serialiser handled a nullable decimal. The synchronous API almost never hits that path. The thing that hits it constantly is the batch consumer that drains the settlement topic at 01:00. Our canary ran at 14:30, got a uniform random sample of HTTP traffic for fifteen minutes, and measured a code path the change did not touch.
Worse, the canary pods were deliberately excluded from the Kafka consumer group. Someone had done that early on to avoid duplicate processing during analysis, which is reasonable and meant the asynchronous half of the system was structurally untestable by our canary. So the release passed a gate that, by construction, could only ever observe a third of what the service does.
Two other gaps came out of the review. The automated analysis looked at HTTP status codes and latency and nothing about whether the work was correct, so a service happily returning 200 with wrong numbers scores perfectly. And one enterprise tenant generates most of our unusual payloads, and uniform sampling gives a fifteen minute window almost none of them.
What we changed: canary pods now join the consumer group with a small, bounded share of partitions, so asynchronous work is included and duplicate risk is handled by idempotent consumers instead of by exclusion. The batch job has a shadow mode that replays the previous night's messages against the canary and diffs outputs against the baseline, which runs in nine minutes and is now a gate. Analysis includes business metrics: settlements produced, totals matched, records rejected. And a handful of synthetic requests built from that tenant's real payload shapes hit the canary on every release.
A canary measures the traffic it receives. If your release changes a path your canary cannot reach, the gate is not passing it. It is abstaining.
– Sergey Shinder
Top comments (0)