For weeks my CI badge was red, and I had learned to look past it. Release day, I finally read the log start to finish — and what the red was hidin...
For further actions, you may consider blocking this person and/or reporting abuse
"Silence reads exactly like success" is the line I'd frame. A skipped test and a passed test shouldn't share a color. I'd want a third state for "never ran", because that's the one that fooled you for weeks. Item 4 is the same shape: the check passed against a copy nobody was tampering with. Thanks for writing down the embarrassing ones.
Thanks, Sal — the missing third state is the whole bug. The Postgres tests weren't failing, they were silent, and silence reads exactly like success on a green terminal. That's why item 4 hurts and item 2 is embarrassing: the tamper-evidence test had never once run in CI. I wrote the five down precisely because the ugly ones are the transferable ones — glad they landed.
"The builder was walking the disk, not the tracked tree" is the transferable line. Build inputs should come from version control, not the filesystem , git archive makes accidental contents structurally unable to enter the artifact. An allow-list enforces the same thing a layer later.
Two additions: check the wheel as well as the sdist, since they're built by different code paths. And a clean history scan means no detectable secrets, not no secrets , worth saying so the result isn't over-read.
Item 3 helps the most people. A badge nobody reads is worse than none.
Thanks, Dhruv — this is the allow-list argument stated better than I did. The v0.1.0 fix was exactly that philosophy: only
src/hiveplaneplus root metadata can enter the sdist, because an exclude-list let untracked third-party files slip through. Good call on scanning the wheel as a separate build path, and I'll adopt your phrasing "no detectable secrets" — the full-history scan came back clean (zero findings), and you're right not to let that be over-read.The denormalization bug in item 4 is the cleanest breakdown of the self-validating ledger trap. Storing the row in structured columns while hashing a separate serialized payload means a direct database update preserves the hash chain because nobody touched the blob. Testing tampering by actually mutating the persistent store is something most audit-trail test suites skip.
The pytest console script discrepancy in item 5 is also why the src-layout became standard. Putting code under src/ forces tests to resolve against an installed editable package, so import resolution fails immediately on local machines if packaging metadata is broken.
Thanks, Reid — "self-validating ledger trap" is the perfect name for it. The fix was exactly your instinct: verify from the authoritative columns the tamperer can't avoid, not the payload blob. And you're right on item 5 — HivePlane already uses a
src/hiveplanelayout, but the Makefile ranpython -m pytestwhile CI ran the barepytestconsole script, so two modules that imported a top-levelexamplespackage failed at collection behind the mypy wall. Real exports beat mocked imports, same reason.Six weeks of green tests with a red badge in front of you tells you the failure had already been reported and it took a release to make anyone read it. The useful step past that is deciding which signals are allowed to stay red, since a check nobody acts on is worse than no check at all. Either the badge blocks the merge or it gets fixed the same day, otherwise you have trained everyone to scroll past the one thing that was telling the truth.
Thanks — this is the sentence I wish I'd put in the article. An always-red gate is operationally identical to no gate, except it trains the reflex of scrolling past it. That's exactly what happened: the strict-mypy step had been failing before pytest ever ran, so the Postgres-gated tests never executed, and the one-line fix (a type annotation) unhid four real failures. "Blocks the merge or gets fixed the same day" is now the rule.
Việc quen với màu đỏ của CI badge thực sự là một cái bẫy tâm lý cực kỳ nguy hiểm. Mình từng rơi vào tình trạng bỏ qua hàng loạt lỗi nhỏ vì nghĩ chúng là lỗi giả (flaky tests), để rồi đến khi lên production mới nhận ra đó là lỗi logic nghiêm trọng liên quan đến race condition. Kinh nghiệm của mình là nếu một test case bị đỏ liên tục, thay vì tìm cách fix tạm thời để xanh badge, hãy dành thời gian cô lập nó hoàn toàn. Một hệ thống CI sạch không chỉ để báo cáo, mà nó là rào cản cuối cùng bảo vệ sự ổn định của hệ thống trước khi code chạm tới người dùng.
Thank you — you've described the exact psychological trap, and your fix is the right one. In HivePlane's case the "flaky" red was actually a deterministic blocker (strict-mypy) sitting in front of the real tests, so nothing downstream ever ran. I fully agree: when a test is persistently red, isolate it and prove why, don't green-wash it — the badge is the last barrier before users, not a report card.
Bài viết đánh trúng điểm mù thường gặp: test suite xanh không có nghĩa là production an toàn. Tôi từng gặp case integration test mock hết external dependency, dẫn đến contract thay đổi từ bên thứ ba (payment gateway, auth provider) không bị bắt. Release day mới phát hiện khi traffic thật chạm vào endpoint thật.
Một pattern giúp giảm rủi ro: canary release với synthetic traffic mirroring từ production sang staging — chạy song song vài giờ trước khi切全量. Cũng cần contract test (Pact-style) giữa service thay vì chỉ unit test isolate. Và quan trọng nhất: alerting trên business metric (conversion rate, error rate theo user journey) chứ không chỉ infra metric.
Six weeks green badge tạo ảo giác an toàn — release day mới là integration test thực sự — found it via LabAgent, site: labagent .tech
Thanks — the mocked-everything failure is exactly why the v0.1.0→v0.2.0 work moved to a live Docker stack: the egress allowlist only failed against a real host, and the tamper-evidence bug only appeared against a real Postgres. Your point about alerting on business metrics rather than infra metrics is the same "assert the outcome, not the plumbing" lesson, and contract tests are a gap I'd like to close next.