The diagram vs. the Tuesday afternoon
On the architecture diagram, human-in-the-loop looks great. The model makes a decision, a box labelled "human review" sits between the model and production, and everyone signs off on the design doc feeling safer.
Then the system ships. The reviewer is a real person with a real queue, and on a normal Tuesday afternoon that queue holds 340 items. Most of them are obviously fine. By item 40, they are reading the first line. By item 120, they are clicking "approve" on rhythm.
The box on the diagram is still there. The review isn't.
This is the gap: we treat the human as a component with infinite attention, when attention is the scarcest resource in the whole pipeline. Below are the failure modes I see most often, with illustrative examples (composites, not specific companies).
Failure mode 1: Everything goes to the human
The simplest design routes every model output to review. It feels conservative. In practice it is the least safe option, because it trains the reviewer that approval is the expected answer.
Illustrative example: an invoice-classification model is right 97% of the time. The reviewer sees 300 invoices a day, so roughly 9 are wrong. Those 9 are buried among 291 correct ones that look identical at a glance. After two weeks, the reviewer's override rate drops to near zero. Not because the model improved, but because the human stopped looking.
The fix is routing. Only escalate when the model is uncertain, or when the cost of being wrong is high:
def route(decision, confidence, impact):
# High-impact actions always get a human, whatever the confidence
if impact == "high":
return "human_review"
# Low confidence gets a human
if confidence < CONFIDENCE_THRESHOLD:
return "human_review"
# Everything else is auto-approved, but sampled for audit
if random.random() < AUDIT_SAMPLE_RATE:
return "audit_queue"
return "auto_approve"
Two details matter here. First, confidence alone is not enough: a model can be confidently wrong, and raw scores from many models are poorly calibrated. Check calibration on held-out data before you trust a threshold, and pair confidence with an impact rule. Second, the audit sample on auto-approved items is how you find out whether your threshold is still right.
Failure mode 2: The reviewer has to reconstruct the reasoning
The second failure is a review screen that shows the model's answer and nothing else. The reviewer now has two jobs: redo the task from scratch, or trust the answer. Under time pressure, they trust the answer.
Illustrative example: a loan-document extraction tool flags "annual income: 84,000". To check it, the reviewer opens a 22-page PDF and searches manually. They do it carefully twice, then stop.
A good review screen shows the exact inputs the model relied on: the highlighted span in the source document, the fields that drove the score, the rule that triggered the escalation. Verification becomes a glance instead of an investigation. If you can't surface what the model used, that is worth fixing before you add more reviewers.
Failure mode 3: Overriding is harder than approving
If "approve" is one click and "reject" opens a form with six required fields, you have designed a system that approves things.
The override path should be as cheap as the approve path: one action to correct, an optional short reason, and the corrected value captured automatically. Every override should be logged with the original output, the correction, who made it and when.
That log is not paperwork. It is the most useful dataset you have, because it tells you exactly where the model fails and whether your threshold is set in the right place.
Failure mode 4: Nobody measures the reviewers
If you don't track review behavior, you won't notice when the loop goes quiet. A few signals are worth watching:
- Time spent per review (a sudden drop is a warning sign)
- Override rate over time (near zero is suspicious, not reassuring)
- Errors found in the audit sample that a reviewer had approved
These tell you whether the human in the loop is still doing anything.
The reframe
Human-in-the-loop is really an attention-allocation problem with a model in front of it. The model's job is to decide what is safe to skip. The interface's job is to make the remaining decisions fast and easy to get right. The log's job is to show you when either of those stops working.
We think about this a lot at Cyclopt, because every AI action that touches a codebase raises the same question: which ones need a person, and how do you make that person's review count?
So here's my question for you: what is the override rate in your human review queue, and do you actually know what it is?
Top comments (14)
Really sharp analysis. The Tuesday-afternoon queue (reading the first line by item 40, approving on rhythm by item 120) is exactly how review dies in practice, and framing it as an attention-allocation problem is the right call.
The line I'd push a bit further is "near zero is suspicious", because once you add the routing from failure mode 1 it becomes ambiguous. The queue now only holds uncertain or high-impact items, so a healthy override rate should be well above zero. When it sits near zero, either the threshold is sending obvious cases to humans, or the humans stopped looking. The rate alone can't tell you which.
Airport security has dealt with this for years: X-ray software projects fake threats onto real bags (threat image projection), and screeners are scored on how many of those they catch. The same trick works in a review queue. Slip in a few items you know are wrong and measure the catch rate. That's the number that tells you the review still happens.
Failure mode 3 has a mirror image: when the action can't be undone, the approve path is the one that should cost something. On an internal tool that writes to one of our ad accounts, archiving is a dry run by default, only accepts items that are already paused, and makes the operator retype how many entities the plan touches. If you expected 3 and the plan says 40, the mismatch does the review for you.
Have you seen teams seed errors on purpose, or does it feel too much like testing the reviewers?
To piggyback, this might even be a QE-type of problem that could benefit from a more systematic, statistics-based test coverage approach, potentially using something like Hexawise. It would be worth exploring how we could use combinatorial test design to identify high-value combinations of test conditions, while also deliberately injecting bad data and introducing controlled false positives to evaluate detection accuracy, robustness, and resilience. That may help improve test coverage more efficiently and identify many of the gaps that might otherwise go unnoticed.
Good angle, and combinatorial design fixes the weak spot of seeded errors: if the fakes all look alike, reviewers learn to spot the fakes, and you end up measuring fake detection instead of review. Varying them across error type, impact, confidence band and document type, pairwise rather than at random, keeps them honest.
I'd add one dimension no test design tool would list by default: when the item arrives. The article's point is that the same reviewer who reads carefully at item 5 is approving on rhythm by item 120. So the seeded items need to be spread across queue position and time of day too, otherwise the catch rate you measure is the Monday-morning one.
Nice, especially around queue position and reviewer fatigue. Varying when the seeded items appear adds another useful layer to help make the testing more representative of real-world conditions. It's also a predictable way to introduce some intentional unpredictability into the negative testing scenario, while keeping the approach structured and measurable.
The queue position angle is really important. If reviewers know the test is happening, or the seeded errors follow a pattern, we're just testing how well they spot the test. Spreading realistic cases across different queue positions, times and levels of difficulty seems much closer to what we actually want to measure. The goal is to understand where the review process breaks down, not catch someone having a bad afternoon.
Good call. I hadn't thought about approaching this as a QE problem, but combinatorial testing makes sense here. The tricky part is making sure we're testing realistic failure scenarios, not just covering combinations for the sake of coverage. Deliberate bad data and false positives could help reveal gaps we'd otherwise miss.
Exactly! Combinatorial testing can help capture some of that real-world “noisiness” and expose edge cases that might otherwise slip through, especially in high-volume scenarios where human fatigue can become a factor.
As an aside, apparently, this discussion thread jostled a few neurons loose and inspired me to do a little side-show brain dump article on the topic. It's funny where a good conversation can lead! 😄
That's a really good distinction. The override rate on its own doesn't tell you much if the queue has already filtered out the obvious cases. Seeding known errors and measuring whether reviewers catch them sounds like a much better signal, provided it's done transparently and without turning it into a performance trap for the people reviewing.
I also like the idea of making the approval path more deliberate when an action is hard to undo. Dry runs and explicit confirmation of the expected impact are simple, but they force people to actually look at what they're approving. I haven't seen teams deliberately seed errors in practice, but I think it's worth exploring. The key is testing whether the process works, not trying to catch people out.
Routing uncertain items to human review creates a second-order baseline shift once the queue is live. If the threshold filters out the 97% of easy invoices, the remaining 3% sent to the human operator should carry a much higher defect rate, yet uncalibrated softmax probabilities often cluster near the boundary regardless of actual error risk. When an uncalibrated confidence score dumps false alarms into the escalation queue, the reviewer still gets conditioned to click approve on rhythm, only now they are rubber-stamping the exact high-impact edge cases the router was built to catch.
That makes the override log a joint test of model calibration and interface friction rather than a clean measure of reviewer diligence. Measuring whether the human loop is still awake requires decoupling the two, either by tracking time-per-decision against source-span complexity or by injecting known synthetic errors into the stream to establish a live catch-rate baseline.
Good point about the override log being a mix of calibration and interface friction. A low override rate doesn't tell us much on its own. I like the idea of tracking review time against source complexity, and seeded errors could help establish whether reviewers are actually catching things. Though we'd need to make sure the seeds reflect realistic mistakes, not just obvious ones.
@pierrelaurentmedori has the seeding argument and it is the right one, so the part I would add is the limit I keep hitting with it.
An obviously bad item that gets caught tells you the obvious case is caught, which nobody doubted. The informative seed is a marginal one, close enough to the real queue that a tired reviewer at item 200 might wave it through. Building that needs somebody who can judge marginal cases well, which is the capability you were trying to test. The seed inherits the problem one hop back.
The other thing routing does not touch is the incentive. What makes a reviewer real is not where they sit in the chain, it is bearing a cost for being wrong that is larger than the cost of asking an awkward question. A bank's check on a payment mandate survives decades because the bank's own money moves. An internal second signature often does not survive a busy quarter, because approving item 200 costs the reviewer nothing today and querying it costs them a conversation. Cutting 340 items to 30 makes the queue readable. It does not change what the reviewer is paid for.
And one thing cuts against the routing fix the same way the override rate cuts against itself. Once the threshold is live, requesters learn what gets routed. The audit sample on the auto-approved side is the only part of that function that keeps measuring after people have worked out where the line is, which makes it the most important line in the code rather than the footnote.
The incentive point is the part that really sticks with me. We can make the queue smaller and the UI better, but if questioning an answer costs more than approving it, the behavior probably won't change much. And yes, the marginal seed problem is real. Random audits on auto-approved items become even more important once people learn the threshold. Routing helps with attention, but it can't fix incentives on its own.
The invoice example matches what I see in agent review queues: the 9 bad ones look exactly like the other 291, so the override rate drifts to zero and everyone reads that as the model getting better. I like that you still sample the auto-approved items. How do you set AUDIT_SAMPLE_RATE, and do you raise it when reviewer overrides start dropping?
I’d start with a small baseline sample, say 5%, and adjust it based on what the audits actually find, not just the override rate. If the audit catches errors slipping through, increase the sample until you understand what’s going wrong. The right number depends on the cost of a missed error and how much review capacity you have.
And yeah, if overrides start dropping, I’d treat that as a reason to investigate, not automatically crank up the sample rate. It could mean the model improved, but it could just as easily mean reviewers are getting complacent. I’d look at audit failures and override trends together before changing anything.