The gap between the demo and the pipeline
In the demo, you ask the model to return a confidence score alongside its answer. It says 0.93...
For further actions, you may consider blocking this person and/or reporting abuse
One alternative to a verbalized score is asking for something you can check. Have the model list which fields it matched and which it could not confirm, then route on that list with plain rules. A missing tax ID is a fact your code can act on, where 0.93 is a number nobody has calibrated yet.
Exactly, I'd much rather have explicit, verifiable signals than a number the model made up about itself. The catch is that even extracted fields can be wrong, so I'd validate them against the source or a trusted system rather than treating the model's checklist as ground truth. But it's a much better starting point for deterministic rules.
Agreed, the list is a starting point rather than the answer. One cheap check before any trusted system gets involved: have the model return, for each matched field, the exact text it read the value from. Your code can then confirm that text appears word for word in the source document and that the value sits inside it. A field that fails either check moves to could not confirm, and the rules treat it like a missing one.
I like that, especially because it gives you a cheap validation step before involving another system. I'd still be careful with the word-for-word check for OCR noise, formatting differences or values split across lines, but those can be handled explicitly. The important part is that a failed check becomes an unresolved field, not something the model gets to explain away with a higher confidence score.
Agreed, an exact match is too strict for scanned documents. Normalising both sides before comparing, by collapsing whitespace and line breaks and ignoring case, covers most of the formatting and split line cases, and anything that still fails goes to a person rather than back to the model.
Yeah, that seems like a good middle ground. I'd just be careful not to normalise away meaningful differences, especially in IDs, account numbers or tax identifiers where one character matters. Whitespace and line breaks are one thing, but I'd keep field-specific validation rules so the normalisation doesn't accidentally turn a mismatch into a match.
The payment reconciliation case is where this gets interesting. Imagine an LLM extracting beneficiary details from supplier invoices and matching them against ERP records. It reports 93% confidence in a match, so the pipeline automatically approves it. But if that confidence comes from the model's own response rather than a validated probability, you've basically automated a financial decision using a number that hasn't earned any trust.
I'd go a step further than checking calibration across the whole dataset. Separate clean digital invoices from OCR-heavy scans, new suppliers from established ones, and exact matches from cases involving similar company names or changed bank details. Then measure the actual error rate in each group and route uncertain or high-risk matches for review.
One thing worth watching is sample size, too. A bucket with 93% observed accuracy based on 30 cases doesn't give you the same evidence as one based on thousands. And even a well-calibrated score won't protect you from a new input distribution.
The real test isn't whether the model can produce a convincing confidence score. It's whether that score can reliably tell your system when it's safe to act and when it should stop.
Exactly, and changed bank details are a good example of where the cost of a false positive matters just as much as the error rate itself. I'd probably treat those as mandatory human review regardless of the confidence score, at least until there's strong evidence the system handles them reliably. Calibration helps, but it shouldn't be the only thing deciding whether a financial action goes through.
Exactly. I'd make the review trigger depend on the type of change, not just the model's confidence. Imagine an ERP workflow where an invoice matches the vendor's legal name, purchase order, amount and tax ID, and the model assigns a 99% match score. But the supplier's bank account was changed that morning, and the new account hasn't been independently verified. Every invoice-matching signal can be correct while the payment still goes to the wrong destination.
I'd treat that as a separate state transition: freeze payment release, verify the change through a previously trusted contact channel, and require an independent approval before updating the vendor master record. For an even stronger check, flag a first payment to the new account and reconcile the approval trail against the actual bank-detail change. Calibration tells you how much to trust a prediction; it doesn't establish who authorized the change or whether the underlying business event is legitimate.
That distinction between prediction confidence and authorization is key. You can have a perfectly correct match and still make the wrong decision because something outside the model's scope changed. I think that's where explicit business rules earn their place: some conditions shouldn't be overridable by a confidence score, no matter how well calibrated it is.
Exactly, and I think the hard part is deciding when the controller is safe to act again. If a deployment API times out after accepting a rollback, the controller might still see the old rollout as active while the rollback is already progressing. I'd avoid issuing another mutation until it can establish what happened to the first one. If the provider doesn't expose a reliable operation status, the controller needs to compare the observed resource state with the desired state and treat unresolved outcomes as unknown, not failed. Otherwise, the recovery logic can end up creating the very conflict it's supposed to fix.
Exactly, an ambiguous timeout is dangerous because you don't know whether the operation failed or just the response did. I'd make the controller reconcile against the actual resource state before retrying, and use idempotency keys where the API supports them. The tricky part is that observed state might lag too, so one read showing the old state isn't necessarily proof that the rollback never started.
Your coverage helper would also benefit from an explicit invalid-score count. With
answered = conf >= t, NaN silently becomes a non-answer, while an out-of-range high score can pass every threshold. I'd validate finite values in [0, 1] before the curve and report rejected rows separately from deliberate deferrals. I'd also version each threshold with its model and prompt, so rolling a model back doesn't quietly retain a cutoff calibrated for a different system.Good point, especially the distinction between invalid scores and intentional deferrals. If NaNs just disappear into the non-answer bucket, the curve can look healthier than it really is. And versioning the threshold alongside the model and prompt seems essential to me. A rollback shouldn't silently inherit a cutoff that was validated for a different setup.
One bug in the bin helper that bites exactly where verbalized scores live: with
conf >= lo & conf < hiand edges from linspace(0,1,11), a score of exactly 1.0 falls in no bin. I ran it on five rows (1.0, 1.0, 0.95, 0.9, 0.5): only 3 of 5 get counted, and the two dropped rows are the ones models emit most. Make the last bin closed on the right (or clip conf to 1-1e-9) and print n_total vs sum of bin n as a sanity check.Second thing is sample size inside the bucket you gate on. 46 right out of 50 above the cutoff is 92%, but the 95% Wilson interval is roughly 81% to 97%; 184 of 200 is 92% too and tightens to 87% to 95%. So "the 0.9 bucket is no better than the rest" or "is fine" often can't be told apart until you have a few hundred labelled cases in that bucket. Adding an interval column to coverage_accuracy_curve would show when a threshold is being set on noise.
Do you log the ground truth back for the auto-approved cases too, or only for the ones a human reviewed? The second gives selection bias in exactly the bucket you care about.
Good catch on the 1.0 edge case, that’s exactly the kind of small bug that can quietly skew the results. And yeah, the selection bias point is important. If we only get ground truth for human-reviewed cases, we’re basically blind to how the auto-approved bucket actually performs. You need some form of ongoing auditing there, even if it means randomly sampling approved cases.
Calibration tells you whether the model's reported probabilities match empirical frequencies on average, but setting an operational threshold requires knowing the loss matrix. If auto-approving a clean document saves ten cents of human review while auto-approving a defective one costs five hundred dollars in manual remediation, your implied odds ratio is five thousand to one.
At that degree of asymmetry, the threshold is dictated entirely by the cost of the tail event. A system calibrated to ninety-five percent accuracy across standard bins is irrelevant if the five percent failure mode carries a catastrophic downside. When error costs are asymmetric enough, the optimal operating point moves far past the region where empirical binning has enough sample density to guarantee calibration.
In practice, that means teams cannot rely on probability curves alone to manage tail risk. The only way to bound exposure without setting coverage to zero is structural capping, such as hard transaction limits or deterministic policy checks that constrain the maximum loss of any single automated decision regardless of the score.
Absolutely. The loss matrix changes the whole picture, and I like the idea of structural caps because they limit what a bad prediction can actually cost you. I'd add that some decisions probably shouldn't be automated at all based on a model score, especially when the downside is hard to bound. Calibration tells you about reliability, but it doesn't make every risk acceptable.
I hit this in a small way in my own tool. It sorts comments on my posts: feedback I should act on, a discussion worth a reply, thanks, or spam.
This week I tested it against what I actually did. I took 27 comments I had already dealt with myself (replied, skipped or hid) and ran the decision again with two models: a dedicated decision model (TypeSafe's Jev) and a general LLM (Gemini Flash Lite).
The most useful thing was disagreement. The two models disagreed 4 times, and every time the decision model was unsure (0.52 or lower), which is exactly where I'd want to look myself. So I'd add something to your list. The checks usually start from the threshold: above it is trusted, below it is reviewed. But nobody asks whether the cases above it were good. For decisions that matter, I'd run a second check with different reasoning over all the cases, not only the ones under 0.9. Then I'd route on disagreement, not on either score.
27 is far too few for real calibration bins. What makes it workable is that I see every comment anyway, so my reply, skip or hide is ground truth logged for free.
That example where the decision model was most confident on its worst mistake is exactly why I wouldn't trust disagreement alone either. I like using it as a trigger for review, but agreement doesn't automatically mean correctness, especially if both models share the same blind spot. And having your actual actions logged is a huge advantage here. You can keep checking the system against what you really did as more comments come in.
The coverage side of the trade-off deserves as much attention as accuracy. Raising the threshold only helps if the deferred cases actually get timely human review; otherwise deferral becomes a queue that grows silently and the system ends up answering under time pressure anyway. That makes the threshold a capacity decision as well as a calibration one. How should the curve be read when review capacity is fixed — is there a point where a slightly lower selective accuracy is preferable because it keeps the deferral path usable?
Yeah, that's a good way to look at it. I'd treat review capacity as a constraint when choosing the threshold, not something to figure out afterward. If the queue can't keep up, the whole deferral mechanism breaks down. The best operating point is probably the one that balances error cost, review capacity and turnaround time, rather than maximizing selective accuracy in isolation.
One slice missing from the usual list is age. In people and company data a match score is calibrated on the day the record was written, and then the world moves: the person changes jobs, the company renames, the mailbox is closed. On a standard title-and-industry list I measured this week, a third of the records pointed at people who had already left the role. The match was not wrong when it was written; it was never re-run. So for anything that decays, I would log the confirmation date next to the confidence, bin by age as well as by confidence, and let the threshold rise with the age of the record instead of trusting one number for a row from last year and a row from last week.
This is a really good point. A score can be well calibrated when the record is created and still be misleading months later, even if the matching logic hasn't changed at all. I'd track record age as a separate dimension and measure how accuracy degrades over time. One nuance is that I'd probably prefer re-verification or an expiry rule for high-impact records over just raising the threshold, since an old record can be confidently wrong.
The distinction between coverage and selective accuracy is what most teams skip. A model that says 93% is producing text, and until you have bucketed its answers against outcomes you do not know what that text is worth.
Sina's point about changed bank details is the one I keep coming back to. Some cases should go to a person because of what changed, not because of how sure the model sounds. On client builds we write those triggers down as plain rules before any threshold tuning: a new payee, an amount above a limit, a field that disagrees with the source document. The score only decides among the remaining cases.
The comment about deferral queue capacity matters too. If reviewers can handle forty items a day, that number sets your threshold more than any calibration curve does.
Did the 1.0 edge case Arhan found change how you bin the top bucket, or did you simply clip it?
I'd make the last bin closed on the right rather than clip the scores. Clipping changes the observed values, while including 1.0 in the final bin preserves what the model actually returned. And definitely agree on the hard rules coming first. If review capacity is fixed, the threshold has to account for that too, but it shouldn't override rules for cases that must be reviewed regardless of the score.
A rubber stamp with a decimal point made me laugh and wince at the same time. We humans do the same thing in incidents: 'I'm sure it's the deploy' is also a string, and we treat it like a probability. I love the coverage versus selective accuracy framing. Which chart worked best for getting product folks to accept a lower coverage number?
Haha, true, incident debugging is full of verbalized confidence too. I'd use a coverage vs. selective accuracy curve and mark the actual operating points on it, ideally with the estimated cost of errors next to each one. Product folks can see exactly what they gain in reliability and what they give up in automation. It's a much easier conversation when the trade-off is visible instead of arguing over an arbitrary 0.9 cutoff.
The same 0.9 trap sits one layer down in eval judges. Mine returns a confidence with every verdict, and I trusted it like your invoice team trusted 0.9 until I put it through bins like your calibration_bins function. When you re-run after a model swap, do you reuse the old ground-truth labels or relabel a fresh sample, since the input mix has usually moved too?
I'd keep the old labels for a clean apples-to-apples comparison, but also label a fresh sample from the current input mix. Otherwise, you can't really tell whether the model got worse or the data changed underneath it. For eval judges especially, I'd also track agreement with human reviewers separately, since a judge can be consistently confident and still consistently wrong.
The calibration bins are the right instrument, but there is a cheaper failure I'd check first: is the number even stable? I asked Gemma 2 2B IT (Q4_K_M, temp 0.2) the same yes/no question three times in isolated contexts and got NO / NO / YES. Same prompt, same weights. If the self-reported confidence can move that much on a re-ask, calibrating it is fitting noise.
The fix I would try before calibration: stop asking for a number and read the probability mass on the "yes" token instead. That gives you one distribution per prompt instead of one generated digit, and it makes the 0.9 cutoff a threshold on something that at least has a scale.
Are you logging token-level probabilities anywhere, or is the confidence field the only signal you capture?
I'm not logging token-level probabilities in that example, just the reported confidence and outcomes. But I agree that stability should come before calibration. If the same input flips between yes and no, that's a pretty big warning sign. Token probabilities are worth testing, though I'd still validate them against actual outcomes because they're probabilities under the model's token distribution, not automatically probabilities that the answer is correct. A stable signal can still be consistently wrong.
cool!
Thanks! Curious, have you run into this kind of confidence problem in a project yourself?
Thank you so much for sharing this essay my friend!