A Detector That Never Fires Scores Perfect on False Positives

Originally published on hexisteme notes.

Two rework incidents this week had the identical shape: a label chip rendered on top of the exact thing the episode was about. In one episode, chile, a ruler's label chip covered the ruler's own body and tick marks — the episode's conclusion rested on comparing the length of two rulers, and the horizontal one wasn't visible anywhere in the frame. In another, siliguri, a scene label chip sat above a narrow corridor of Indian territory and severed it, and that corridor was the episode's subject. Both episodes rendered without error. Schema validation passed. Compliance gates passed. Regression tests passed. The defect was visible on the screen and nowhere else.

This is the fourth attempt at an automatic detector for it, and the first three failed the same way. I've written about checks that go quiet before, each a different failure: a probe whose own prep procedure made a pass the only possible outcome, no matter what had happened to the file under test; a threshold written as an absolute constant, tuned to fixtures at one scale, silently disabled at another; a checker whose zero, four separate times in one audit, meant "I didn't look there" rather than "nothing is there"; a check whose "zero overlaps" was right about a narrower question than the one I was asking. This one is none of those four. Every attempt below ran, at full aperture, with a live path to failure — and still couldn't tell a broken frame from a clean one. What was missing wasn't a working check. It was a test for whether a check was working at all.

Three failures, one blind spot

I built and discarded three separate detectors before this one, and all three failed for the same underlying reason.

The first checked a safe area: the four edge extremes of whatever sat at the frame's border. Both defects sit dead center, so the extremes never moved. For siliguri, the broken frame and the repaired frame produced identical edge extremes.

The second counted a "focus-color run" around the label, tried under six different definitions of what a run is. Under every one, the real defect scored no higher than a clean episode: 26 against 27 under one definition, tied at 28 apiece under another, next to lesotho, a normal episode. All six definitions failed the same way.

The third isolated the amber-colored label blob down to just its glyph. It separated seven samples cleanly. Checked retroactively against 27 published episodes, 10 of the 11 frames it flagged were false positives — 91%.

All three inferred geometry from the color of an already-rendered PNG. The renderer, meanwhile, knew exactly what it had drawn and where, and threw that information away the moment the pixels hit disk. All three failures pointed at the same missing thing.

Teaching the renderer to keep a receipt

The fix wasn't a fourth color heuristic. I made the renderer log its own output. src/shorts_factory/render_receipt.py tags each matplotlib artist with set_gid() to record its role, then reads back the measured bounding box with get_window_extent(renderer) right after savefig and writes it to render_geometry.json.

Two properties made this cheap to add. set_gid is read only by the SVG backend — the Agg backend that rasterizes the frame ignores it — so tagging is provably pixel-neutral: a test, test_tagging_is_pixel_neutral_for_map_frames, monkeypatches the tag call to a no-op and confirms the rendered sha256 still matches. And savefig has already done the drawing by the time the receipt gets collected, so reading the extent back costs nothing extra.

The receipt only records the occluding side — chips, lines, cards. Map polygons don't get logged; one country can be tens of multipolygons across roughly a thousand frames, and the receipt would end up bigger than the video it documents. Instead it records camera state — camera_bbox, axes_rect, highlight, geo — precise enough to reproject the occluded side offline, since the projection is equirectangular — one linear formula.

Overlap stopped being a heuristic at that point. It became arithmetic: does one bounding box intersect another.

Two traps that would have been quiet false negatives

Two matplotlib behaviors nearly undid this, both failing the same direction: evidence quietly missing, downstream code reading the absence as "no overlap."

A Collection artist's get_window_extent returns (inf, inf, -inf, -inf) even after drawing, confirmed on matplotlib 3.11.1. Left alone, ruler_ticks (a LineCollection) and density_dots (a scatter) would silently drop out of every receipt — the exact false negative this module exists to prevent. The fix: get_datalim(ax.transData), both corners transformed through transData, padded by marker radius (√s / 2, s in points squared).

Axes built with ax.inset_axes() don't show up in fig.axes — they live in ax.child_axes. A Monte Carlo minimap's tagged elements were structurally uncollectable as a result: only 3 of its 4 sim_particles ever made it into a receipt. The fix makes the axis-walking function, _iter_axes, recurse into child_axes.

I pinned both down with regression tests.

Measuring before wiring anything

The repository runs on a standing rule: measure retroactive false positives against published output before wiring anything into a gate — the rule that caught the third attempt's 91% false-positive rate, and the same discipline applied here.

The instrument, scripts/measure_render_occlusion.py, defaults to exit code 0 — a verdict threshold activates only when explicitly requested — and it ran against all 25 published episodes.

Re-rendering all 25 for that measurement would take 2.7 hours. Instead I built a probe mode, geometry_probe_step: it samples every N frames and writes nothing at all — no frame PNGs, no render_events.json, no receipt. Read-only is structural here, not a promise, so it can't touch published output no matter what it finds. Sampling every 1.0 seconds across 25 episodes took 12 minutes; a directory listing taken before and after confirmed nothing on disk had moved.

Two predicates, one split

Two detectors came out of the receipt data, and I didn't treat them the same way.

instrument_swallowed — the chile-type defect — fires when a tool line (ruler body, tick marks, leader line, path line, arrow) sits entirely inside an opaque chip with higher z-order. I didn't invent a threshold; it only counts a containment ratio of exactly 1.0.

I wired that into compliance_gate.check_instrument_not_swallowed.

subject_severed — the siliguri-type defect — measures the longest contiguous run of pixels in a row where an overlay chip cuts across the highlighted territory.

It fires on 18 of the 25 episodes. The top-level judgment_card is an outro card designed to cover the map, so its hits aren't defects — the count is real, the alarm isn't. Narrow to scene_label, the same role the actual defect used, and the top three episodes by run length are southamerica-east, lesotho, and russia-nk. All three are normal episodes, tied with the corridor defect itself at 56 pixels. There's no threshold that sits between them.

I didn't wire that one anywhere. It stayed an instrument, not a gate.

The only test that tells a detector apart from a stub

Here's the part that generalizes past matplotlib and video frames. instrument_swallowed's "0 false positives across 25 published episodes" is not, on its own, distinguishable from a function that always returns an empty list. Both produce the identical report: zero findings, every time. The only way to tell them apart is to check whether the detector can fire at all — hold out a sample labeled known-bad, not just a pile of presumed known-good ones, and confirm it fires.

That's what the sensitivity row above really is. _ruler_label_offset_pt is the line of code that fixes the chile defect, so turning it off reproduces the defect exactly, on demand: eight firings with the fix removed, zero with it back in place. instrument_swallowed wasn't verified by its clean run against 25 real episodes. It was verified by being made to fail on purpose, and doing so.

Zero false positives is not evidence that a detector works. It's the score a detector that never fires gets for free. The only way to tell the two apart is a positive control: hold out a known-bad sample and confirm the detector actually fires on it.

subject_severed never earned that kind of confirmation. The wall it hit is the same one the second, color-based attempt hit: whether a chip cutting 56 pixels off a territory is a defect depends on what the episode is arguing, not on the pixels. The identical run length is a defect in siliguri and completely normal in three others. Switching coordinate systems from color to exact geometry fixed instrument_swallowed's precision problem — the false-positive rate that sank the third attempt. It did nothing for subject_severed, because what an episode is trying to say was never a precision problem.

An exemption rule worth keeping

One design choice here is worth keeping. The gate guarding instrument_swallowed passes automatically when render_geometry.json doesn't exist — renders made before the receipt system existed are supposed to lack one. That contrasts with a hardcoded list of episode names the same repository uses to exempt the editor_note evidence system, written the same day: a list has to be maintained by hand and can be forgotten after a snapshot, while re-rendering an old episode here turns the check back on with nothing to remember. A list is still the right call sometimes — file presence alone can't always tell missing evidence apart from the defect itself. Either way, the reason has to live in the code, not in memory.

What's still unsolved

siliguri-type defects still aren't caught automatically. The current evidence guarantees a chip cut 56 pixels off a territory; it says nothing about whether those pixels were the point of the episode. That claim exists — in script.yaml's camera_bbox and emphasis fields — but it never makes it into the rendered pixels or the receipt describing them. Closing that gap means putting what the script claims into the evidence itself — schema work, not instrument work, and it's next.

Both predicates came from the same receipt data, the same week, checked against the same 25 episodes. One runs in the gate today. The other is still a script I run by hand, because it never earned the right to run unattended — and the only reason I know the difference is that I asked both to prove it, not just report it.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.