The gap between the demo and the pipeline
In the demo, you ask the model to return a confidence score alongside its answer. It says 0.93. You write if confidence > 0.9: auto_approve() and move on.
In production, that number is just more generated text. Nobody measured it against outcomes. It looks like a probability, it's formatted like a probability, and your code treats it like a probability. That is how false precision becomes automated risk.
A confidence score earns the word "confidence" only when it is calibrated: of all the cases where the system said ~90%, roughly 90% were actually correct. Until you've checked that on your own data, the number is decoration.
Failure mode 1: the threshold that was never tested
Illustrative scenario: a team routes invoice classification through an LLM and auto-processes anything above 0.9. On their eval set, accuracy is 94%. Looks great.
What they never measured is accuracy within the >0.9 bucket. When someone finally slices it, the high-confidence bucket is barely better than the low-confidence one. The model emits 0.9+ on almost everything. The threshold wasn't filtering anything; it was a rubber stamp with a decimal point.
Failure mode 2: one accuracy number hiding two systems
If your system can defer (hand off to a human, ask for clarification, fall back to a rule), then you're not evaluating one number. You're evaluating a trade-off:
- Coverage: how often does the system answer instead of deferring?
- Selective accuracy: when it answers, how often is it right?
Push the threshold up and coverage drops while selective accuracy (hopefully) rises. Whether it actually rises is the whole question. Here's the minimal version of measuring it:
import numpy as np
def coverage_accuracy_curve(conf, correct, thresholds):
conf, correct = np.asarray(conf), np.asarray(correct)
rows = []
for t in thresholds:
answered = conf >= t
coverage = answered.mean()
sel_acc = correct[answered].mean() if answered.any() else float("nan")
rows.append((t, round(coverage, 3), round(sel_acc, 3)))
return rows
def calibration_bins(conf, correct, n_bins=10):
conf, correct = np.asarray(conf), np.asarray(correct)
edges = np.linspace(0, 1, n_bins + 1)
for lo, hi in zip(edges[:-1], edges[1:]):
m = (conf >= lo) & (conf < hi)
if m.any():
print(f"[{lo:.1f}, {hi:.1f}) n={m.sum():4d} "
f"stated={conf[m].mean():.2f} actual={correct[m].mean():.2f}")
If "stated" and "actual" drift apart in the bins you care about, your threshold is lying to you.
Failure mode 3: the global threshold
Even a calibrated score can be calibrated on average and miscalibrated everywhere that matters. Illustrative scenario: overall, the 0.9 bucket is 91% correct. Split by language and the English tickets sit at 95% while the Greek and German ones sit far lower. Split by input quality and the clean PDFs are fine while the scanned, OCR'd ones fall apart.
One threshold across all of that averages away exactly the failures you built the deferral path to catch. Run the same curve per slice:
for slice_name, idx in slices.items(): # e.g. by task type, language, source quality
print(slice_name, coverage_accuracy_curve(conf[idx], correct[idx], [0.7, 0.8, 0.9]))
If the curves differ meaningfully, you need per-slice thresholds, or you need to stop trusting the score for that slice at all.
Failure mode 4: verbalized vs. measured confidence
Asking the model "how sure are you?" and reading token log-probs are different signals, and neither is calibrated by default. Self-reported numbers in particular tend to cluster on round, reassuring values. Treat any source of confidence (verbal, log-prob, ensemble agreement, a separate verifier) as a hypothesis to test, not a fact to consume.
The reframe
Confidence isn't a model property. It's a system property you have to measure. The real question isn't "does the model feel sure?" It's "is the system's decision to proceed reliable for this task, in this context, on this kind of input?"
Practical checklist
- Log every decision with its confidence, the slice metadata, and the eventual ground truth.
- Plot coverage vs. selective accuracy, globally and per slice.
- Check calibration bins in the range where your threshold actually sits.
- Re-run it when the model, prompt, or input distribution changes. Calibration doesn't survive a model swap.
How are you deciding when your LLM pipeline is allowed to act without a human? Is anyone here actually running per-slice thresholds in production, or is everyone on a single global cutoff?
Top comments (26)
The payment reconciliation case is where this gets interesting. Imagine an LLM extracting beneficiary details from supplier invoices and matching them against ERP records. It reports 93% confidence in a match, so the pipeline automatically approves it. But if that confidence comes from the model's own response rather than a validated probability, you've basically automated a financial decision using a number that hasn't earned any trust.
I'd go a step further than checking calibration across the whole dataset. Separate clean digital invoices from OCR-heavy scans, new suppliers from established ones, and exact matches from cases involving similar company names or changed bank details. Then measure the actual error rate in each group and route uncertain or high-risk matches for review.
One thing worth watching is sample size, too. A bucket with 93% observed accuracy based on 30 cases doesn't give you the same evidence as one based on thousands. And even a well-calibrated score won't protect you from a new input distribution.
The real test isn't whether the model can produce a convincing confidence score. It's whether that score can reliably tell your system when it's safe to act and when it should stop.
Exactly, and changed bank details are a good example of where the cost of a false positive matters just as much as the error rate itself. I'd probably treat those as mandatory human review regardless of the confidence score, at least until there's strong evidence the system handles them reliably. Calibration helps, but it shouldn't be the only thing deciding whether a financial action goes through.
Exactly. I'd make the review trigger depend on the type of change, not just the model's confidence. Imagine an ERP workflow where an invoice matches the vendor's legal name, purchase order, amount and tax ID, and the model assigns a 99% match score. But the supplier's bank account was changed that morning, and the new account hasn't been independently verified. Every invoice-matching signal can be correct while the payment still goes to the wrong destination.
I'd treat that as a separate state transition: freeze payment release, verify the change through a previously trusted contact channel, and require an independent approval before updating the vendor master record. For an even stronger check, flag a first payment to the new account and reconcile the approval trail against the actual bank-detail change. Calibration tells you how much to trust a prediction; it doesn't establish who authorized the change or whether the underlying business event is legitimate.
That distinction between prediction confidence and authorization is key. You can have a perfectly correct match and still make the wrong decision because something outside the model's scope changed. I think that's where explicit business rules earn their place: some conditions shouldn't be overridable by a confidence score, no matter how well calibrated it is.
Exactly, and I think the hard part is deciding when the controller is safe to act again. If a deployment API times out after accepting a rollback, the controller might still see the old rollout as active while the rollback is already progressing. I'd avoid issuing another mutation until it can establish what happened to the first one. If the provider doesn't expose a reliable operation status, the controller needs to compare the observed resource state with the desired state and treat unresolved outcomes as unknown, not failed. Otherwise, the recovery logic can end up creating the very conflict it's supposed to fix.
One alternative to a verbalized score is asking for something you can check. Have the model list which fields it matched and which it could not confirm, then route on that list with plain rules. A missing tax ID is a fact your code can act on, where 0.93 is a number nobody has calibrated yet.
Exactly, I'd much rather have explicit, verifiable signals than a number the model made up about itself. The catch is that even extracted fields can be wrong, so I'd validate them against the source or a trusted system rather than treating the model's checklist as ground truth. But it's a much better starting point for deterministic rules.
Agreed, the list is a starting point rather than the answer. One cheap check before any trusted system gets involved: have the model return, for each matched field, the exact text it read the value from. Your code can then confirm that text appears word for word in the source document and that the value sits inside it. A field that fails either check moves to could not confirm, and the rules treat it like a missing one.
Your coverage helper would also benefit from an explicit invalid-score count. With
answered = conf >= t, NaN silently becomes a non-answer, while an out-of-range high score can pass every threshold. I'd validate finite values in [0, 1] before the curve and report rejected rows separately from deliberate deferrals. I'd also version each threshold with its model and prompt, so rolling a model back doesn't quietly retain a cutoff calibrated for a different system.Good point, especially the distinction between invalid scores and intentional deferrals. If NaNs just disappear into the non-answer bucket, the curve can look healthier than it really is. And versioning the threshold alongside the model and prompt seems essential to me. A rollback shouldn't silently inherit a cutoff that was validated for a different setup.
One bug in the bin helper that bites exactly where verbalized scores live: with
conf >= lo & conf < hiand edges from linspace(0,1,11), a score of exactly 1.0 falls in no bin. I ran it on five rows (1.0, 1.0, 0.95, 0.9, 0.5): only 3 of 5 get counted, and the two dropped rows are the ones models emit most. Make the last bin closed on the right (or clip conf to 1-1e-9) and print n_total vs sum of bin n as a sanity check.Second thing is sample size inside the bucket you gate on. 46 right out of 50 above the cutoff is 92%, but the 95% Wilson interval is roughly 81% to 97%; 184 of 200 is 92% too and tightens to 87% to 95%. So "the 0.9 bucket is no better than the rest" or "is fine" often can't be told apart until you have a few hundred labelled cases in that bucket. Adding an interval column to coverage_accuracy_curve would show when a threshold is being set on noise.
Do you log the ground truth back for the auto-approved cases too, or only for the ones a human reviewed? The second gives selection bias in exactly the bucket you care about.
Good catch on the 1.0 edge case, that’s exactly the kind of small bug that can quietly skew the results. And yeah, the selection bias point is important. If we only get ground truth for human-reviewed cases, we’re basically blind to how the auto-approved bucket actually performs. You need some form of ongoing auditing there, even if it means randomly sampling approved cases.
The distinction between coverage and selective accuracy is what most teams skip. A model that says 93% is producing text, and until you have bucketed its answers against outcomes you do not know what that text is worth.
Sina's point about changed bank details is the one I keep coming back to. Some cases should go to a person because of what changed, not because of how sure the model sounds. On client builds we write those triggers down as plain rules before any threshold tuning: a new payee, an amount above a limit, a field that disagrees with the source document. The score only decides among the remaining cases.
The comment about deferral queue capacity matters too. If reviewers can handle forty items a day, that number sets your threshold more than any calibration curve does.
Did the 1.0 edge case Arhan found change how you bin the top bucket, or did you simply clip it?
I'd make the last bin closed on the right rather than clip the scores. Clipping changes the observed values, while including 1.0 in the final bin preserves what the model actually returned. And definitely agree on the hard rules coming first. If review capacity is fixed, the threshold has to account for that too, but it shouldn't override rules for cases that must be reviewed regardless of the score.
Calibration tells you whether the model's reported probabilities match empirical frequencies on average, but setting an operational threshold requires knowing the loss matrix. If auto-approving a clean document saves ten cents of human review while auto-approving a defective one costs five hundred dollars in manual remediation, your implied odds ratio is five thousand to one.
At that degree of asymmetry, the threshold is dictated entirely by the cost of the tail event. A system calibrated to ninety-five percent accuracy across standard bins is irrelevant if the five percent failure mode carries a catastrophic downside. When error costs are asymmetric enough, the optimal operating point moves far past the region where empirical binning has enough sample density to guarantee calibration.
In practice, that means teams cannot rely on probability curves alone to manage tail risk. The only way to bound exposure without setting coverage to zero is structural capping, such as hard transaction limits or deterministic policy checks that constrain the maximum loss of any single automated decision regardless of the score.
Absolutely. The loss matrix changes the whole picture, and I like the idea of structural caps because they limit what a bad prediction can actually cost you. I'd add that some decisions probably shouldn't be automated at all based on a model score, especially when the downside is hard to bound. Calibration tells you about reliability, but it doesn't make every risk acceptable.
One slice missing from the usual list is age. In people and company data a match score is calibrated on the day the record was written, and then the world moves: the person changes jobs, the company renames, the mailbox is closed. On a standard title-and-industry list I measured this week, a third of the records pointed at people who had already left the role. The match was not wrong when it was written; it was never re-run. So for anything that decays, I would log the confirmation date next to the confidence, bin by age as well as by confidence, and let the threshold rise with the age of the record instead of trusting one number for a row from last year and a row from last week.
This is a really good point. A score can be well calibrated when the record is created and still be misleading months later, even if the matching logic hasn't changed at all. I'd track record age as a separate dimension and measure how accuracy degrades over time. One nuance is that I'd probably prefer re-verification or an expiry rule for high-impact records over just raising the threshold, since an old record can be confidently wrong.
The coverage side of the trade-off deserves as much attention as accuracy. Raising the threshold only helps if the deferred cases actually get timely human review; otherwise deferral becomes a queue that grows silently and the system ends up answering under time pressure anyway. That makes the threshold a capacity decision as well as a calibration one. How should the curve be read when review capacity is fixed — is there a point where a slightly lower selective accuracy is preferable because it keeps the deferral path usable?
Yeah, that's a good way to look at it. I'd treat review capacity as a constraint when choosing the threshold, not something to figure out afterward. If the queue can't keep up, the whole deferral mechanism breaks down. The best operating point is probably the one that balances error cost, review capacity and turnaround time, rather than maximizing selective accuracy in isolation.
I hit this in a small way in my own tool. It sorts comments on my posts: feedback I should act on, a discussion worth a reply, thanks, or spam.
This week I tested it against what I actually did. I took 27 comments I had already dealt with myself (replied, skipped or hid) and ran the decision again with two models: a dedicated decision model (TypeSafe's Jev) and a general LLM (Gemini Flash Lite).
The most useful thing was disagreement. The two models disagreed 4 times, and every time the decision model was unsure (0.52 or lower), which is exactly where I'd want to look myself. So I'd add something to your list. The checks usually start from the threshold: above it is trusted, below it is reviewed. But nobody asks whether the cases above it were good. For decisions that matter, I'd run a second check with different reasoning over all the cases, not only the ones under 0.9. Then I'd route on disagreement, not on either score.
27 is far too few for real calibration bins. What makes it workable is that I see every comment anyway, so my reply, skip or hide is ground truth logged for free.
That example where the decision model was most confident on its worst mistake is exactly why I wouldn't trust disagreement alone either. I like using it as a trigger for review, but agreement doesn't automatically mean correctness, especially if both models share the same blind spot. And having your actual actions logged is a huge advantage here. You can keep checking the system against what you really did as more comments come in.
The same 0.9 trap sits one layer down in eval judges. Mine returns a confidence with every verdict, and I trusted it like your invoice team trusted 0.9 until I put it through bins like your calibration_bins function. When you re-run after a model swap, do you reuse the old ground-truth labels or relabel a fresh sample, since the input mix has usually moved too?
I'd keep the old labels for a clean apples-to-apples comparison, but also label a fresh sample from the current input mix. Otherwise, you can't really tell whether the model got worse or the data changed underneath it. For eval judges especially, I'd also track agreement with human reviewers separately, since a judge can be consistently confident and still consistently wrong.