The diagram vs. the Tuesday afternoon
On the architecture diagram, human-in-the-loop looks great. The model makes a decision, a box labe...
For further actions, you may consider blocking this person and/or reporting abuse
Really sharp analysis. The Tuesday-afternoon queue (reading the first line by item 40, approving on rhythm by item 120) is exactly how review dies in practice, and framing it as an attention-allocation problem is the right call.
The line I'd push a bit further is "near zero is suspicious", because once you add the routing from failure mode 1 it becomes ambiguous. The queue now only holds uncertain or high-impact items, so a healthy override rate should be well above zero. When it sits near zero, either the threshold is sending obvious cases to humans, or the humans stopped looking. The rate alone can't tell you which.
Airport security has dealt with this for years: X-ray software projects fake threats onto real bags (threat image projection), and screeners are scored on how many of those they catch. The same trick works in a review queue. Slip in a few items you know are wrong and measure the catch rate. That's the number that tells you the review still happens.
Failure mode 3 has a mirror image: when the action can't be undone, the approve path is the one that should cost something. On an internal tool that writes to one of our ad accounts, archiving is a dry run by default, only accepts items that are already paused, and makes the operator retype how many entities the plan touches. If you expected 3 and the plan says 40, the mismatch does the review for you.
Have you seen teams seed errors on purpose, or does it feel too much like testing the reviewers?
To piggyback, this might even be a QE-type of problem that could benefit from a more systematic, statistics-based test coverage approach, potentially using something like Hexawise. It would be worth exploring how we could use combinatorial test design to identify high-value combinations of test conditions, while also deliberately injecting bad data and introducing controlled false positives to evaluate detection accuracy, robustness, and resilience. That may help improve test coverage more efficiently and identify many of the gaps that might otherwise go unnoticed.
Good angle, and combinatorial design fixes the weak spot of seeded errors: if the fakes all look alike, reviewers learn to spot the fakes, and you end up measuring fake detection instead of review. Varying them across error type, impact, confidence band and document type, pairwise rather than at random, keeps them honest.
I'd add one dimension no test design tool would list by default: when the item arrives. The article's point is that the same reviewer who reads carefully at item 5 is approving on rhythm by item 120. So the seeded items need to be spread across queue position and time of day too, otherwise the catch rate you measure is the Monday-morning one.
Nice, especially around queue position and reviewer fatigue. Varying when the seeded items appear adds another useful layer to help make the testing more representative of real-world conditions. It's also a predictable way to introduce some intentional unpredictability into the negative testing scenario, while keeping the approach structured and measurable.
The queue position angle is really important. If reviewers know the test is happening, or the seeded errors follow a pattern, we're just testing how well they spot the test. Spreading realistic cases across different queue positions, times and levels of difficulty seems much closer to what we actually want to measure. The goal is to understand where the review process breaks down, not catch someone having a bad afternoon.
Good call. I hadn't thought about approaching this as a QE problem, but combinatorial testing makes sense here. The tricky part is making sure we're testing realistic failure scenarios, not just covering combinations for the sake of coverage. Deliberate bad data and false positives could help reveal gaps we'd otherwise miss.
Routing uncertain items to human review creates a second-order baseline shift once the queue is live. If the threshold filters out the 97% of easy invoices, the remaining 3% sent to the human operator should carry a much higher defect rate, yet uncalibrated softmax probabilities often cluster near the boundary regardless of actual error risk. When an uncalibrated confidence score dumps false alarms into the escalation queue, the reviewer still gets conditioned to click approve on rhythm, only now they are rubber-stamping the exact high-impact edge cases the router was built to catch.
That makes the override log a joint test of model calibration and interface friction rather than a clean measure of reviewer diligence. Measuring whether the human loop is still awake requires decoupling the two, either by tracking time-per-decision against source-span complexity or by injecting known synthetic errors into the stream to establish a live catch-rate baseline.
Good point about the override log being a mix of calibration and interface friction. A low override rate doesn't tell us much on its own. I like the idea of tracking review time against source complexity, and seeded errors could help establish whether reviewers are actually catching things. Though we'd need to make sure the seeds reflect realistic mistakes, not just obvious ones.
@pierrelaurentmedori has the seeding argument and it is the right one, so the part I would add is the limit I keep hitting with it.
An obviously bad item that gets caught tells you the obvious case is caught, which nobody doubted. The informative seed is a marginal one, close enough to the real queue that a tired reviewer at item 200 might wave it through. Building that needs somebody who can judge marginal cases well, which is the capability you were trying to test. The seed inherits the problem one hop back.
The other thing routing does not touch is the incentive. What makes a reviewer real is not where they sit in the chain, it is bearing a cost for being wrong that is larger than the cost of asking an awkward question. A bank's check on a payment mandate survives decades because the bank's own money moves. An internal second signature often does not survive a busy quarter, because approving item 200 costs the reviewer nothing today and querying it costs them a conversation. Cutting 340 items to 30 makes the queue readable. It does not change what the reviewer is paid for.
And one thing cuts against the routing fix the same way the override rate cuts against itself. Once the threshold is live, requesters learn what gets routed. The audit sample on the auto-approved side is the only part of that function that keeps measuring after people have worked out where the line is, which makes it the most important line in the code rather than the footnote.
The incentive point is the part that really sticks with me. We can make the queue smaller and the UI better, but if questioning an answer costs more than approving it, the behavior probably won't change much. And yes, the marginal seed problem is real. Random audits on auto-approved items become even more important once people learn the threshold. Routing helps with attention, but it can't fix incentives on its own.
The invoice example matches what I see in agent review queues: the 9 bad ones look exactly like the other 291, so the override rate drifts to zero and everyone reads that as the model getting better. I like that you still sample the auto-approved items. How do you set AUDIT_SAMPLE_RATE, and do you raise it when reviewer overrides start dropping?
I’d start with a small baseline sample, say 5%, and adjust it based on what the audits actually find, not just the override rate. If the audit catches errors slipping through, increase the sample until you understand what’s going wrong. The right number depends on the cost of a missed error and how much review capacity you have.
And yeah, if overrides start dropping, I’d treat that as a reason to investigate, not automatically crank up the sample rate. It could mean the model improved, but it could just as easily mean reviewers are getting complacent. I’d look at audit failures and override trends together before changing anything.