Judge-only intervals miss the full-human value 18 of 21 times. PPI++ saves ~9% of human labels; disagreement routing costs 2.6x — pre-registered and coverage-gated.
Scope note, first paragraph as promised: 7 English-heavy annotation tasks, 3 mid-tier judges (2 local, 1 API), one run each, $0.29 API spend. Numbers below are scoped to the pre-registered study (846 configs × 2,000 replicates).
Your LLM judge says the system scores 72%. The true number — what full human
labeling would conclude — is 65%. And the ±1% interval you computed describes
the judge's opinions, not the truth. I measured exactly this: judge-only
intervals missed the full-human value in 18 of 21 task-and-panel
combinations across 7 real annotation tasks.
The usual fix — "label a random 100 by hand to check" — is ad hoc. Nobody
knows whether 100 is enough, which 100 to pick, or how to combine them with
the 5,000 judge labels. FewHumans is a library that answers all three
with valid statistics: it routes which items humans label, combines the two
sources with prediction-powered inference, and tells you when to stop looking
at anytime-valid intervals.
Setup
Three judges (two local Qwen 9B/4B through Ollama, one gpt-4o-mini — $0.29
total) labeled 7 tasks: HelpSteer2 and SummEval ratings, MT-Bench and Arena
preferences, WMT translation scores, toxicity flags, and OracleBench math
verdicts. Prompts frozen before any simulation. Ground truth = the human
labels; strategies only "see" a label when they buy it.
Results
Pre-registered, 846 configurations × 2,000 replicates each:
- PPI++: 0.91 [0.87, 0.95] — saves ~9% of human labels. Modest, real.
- Stratified PPI++: 0.91 [0.84, 0.98] — same savings, simpler assumptions.
- Fixed PPI: 2.27× — costs labels. With mediocre judges (agreement 45–77%), untuned λ=1 adds noise. Never run PPI without power tuning.
- Disagreement routing: 2.64× — costs labels on 6 of 7 tasks, even though disagreement predicts errors (AUROC up to 0.80). Inverse-probability weights blow up variance faster than the signal helps. My prior (from Sfyraki & Wang) said this might fail. It failed.
- Judge-only: misses 18/21. Verbalized confidence is overconfident everywhere (calibration error up to 0.98).
Every default interval passed a 196-configuration coverage gate first
(Student-t intervals; stratified needs n≥100; skewed truth needs n≥200).
Limitations
Savings are ~9%, not 10× — garbage in applies. 7 English-heavy tasks, 3
mid-tier judges, one run each. Ratio estimands (F1) not shipped. Active
routing needs logged probabilities some workflows lack. High-stakes domains:
this reduces human review, never removes it.
Reproduce
pip install fewhumans
pytest # 47 tests
python examples/toy_end_to_end.py
Repo: github.com/raihan-js/fewhumans · Dataset: JudgeBank (8 tasks, every
judge output + cost + latency) · Paper draft: arXiv soon.
Top comments (3)
The disagreement-routing result is the part I'd want more people to sit with. It's such an intuitive move — "humans should look at the cases where the judge is unsure" — and the AUROC up to 0.80 makes it look like it should work. But inverse-probability weights blowing up variance faster than the signal helps is the trap nobody warns you about until you've already shipped the dashboard. We hit a milder version of this routing human review toward model-disagreement cases and watched our confidence intervals get wider, not tighter, before we understood why.
The thing I keep coming back to is that "label a random 100 by hand" isn't just ad hoc — it's ad hoc in a way that feels rigorous, which is worse. You get a number, it has a ± on it, and the ± is describing the judge's opinions rather than the truth. Quick question: for the PPI++ ~9% savings, did the gain hold up as judge agreement improved, or is the whole method basically a correction for mediocre judges that stops mattering once your judge is actually good?
A 9% saving is not a property of PPI++ — it is a property of your judges, and it is computable before the 846-configuration experiment rather than inferred from it.
With a judge that is the human label plus a bias plus independent noise, the correction term's variance is Var(y−f)/m, so the human-label-equivalent is m_eff/m ≈ ρ²/(1−ρ²), where ρ is judge–human correlation on the items you would actually sample. Measured on a fixed population with a biased judge (b=0.4), resampling the labelled subset 4,000 times:
Three consequences for your numbers.
0.91 is about ρ ≈ 0.72, not about the method. Inverting 1.09 gives ρ = 0.722, which sits inside your agreement band (45–77%) — so the modest saving you found is what that judge quality must produce, and a reader could have predicted it from your agreement table without the replicates. Worth saying in the post, because "saves ~9%" reads as a verdict on PPI when it is a measurement of the panel.
Your routing result was decided before it ran. Buying labels where the judge is wrong is selection on |y−f|, which is the numerator of the variance in the law above; the sample's Var(y−f) rises, so m_eff falls. Measured, sampling weighted by squared residual: 2.12× cost at ρ=0.72, 1.74× at 0.80, 1.23× at 0.90 — same order as your 2.64×. In the naive version I also saw the routed mean shift (−1.10 against a true −0.40); inverse-probability weights remove that bias and leave the variance cost, which is the outcome you report. So the failure is a property of the selection, not of the weights implementation, and it would fail the same way at ρ=0.95.
Which makes ρ per task the missing column. Above ~0.85 PPI pays 4×; below 0.707 it cannot pay at all; your seven tasks almost certainly do not share one ρ, and a single 0.91 [0.87, 0.95] reads as though they do. Two hundred human labels on one task each decides it, and it decides the routing question per task — the one you actually have to answer operationally — before any replication budget is spent.
Scope: synthetic population, one-dimensional, Gaussian noise, λ tuned by plug-in on the labelled subset; ratio estimands like F1 and skewed truths move the constants, not the ordering. And the AUROC 0.80 you found for disagreement is not contradicted by any of this — disagreement does predict error, it just predicts it in the quantity that decides the variance.
The 18 of 21 misses are 7 tasks times 3 judges, and the three judges label the same items, so the 21 are not 21 independent trials. I'd report it as "every task had at least one judge whose interval missed" next to the 18/21 and give a Wilson interval on the task level (n=7), because a reader will otherwise read 86% as a rate with a tight error bar. The 2,000 replicates fix the Monte Carlo error, not the number of tasks.
A related check for the 9% saving: it comes from judge agreement of 45-77%, so I'd print the saving against agreement per task. If it is roughly flat in agreement the 0.91 is a property of the method; if it climbs, it is a property of your judge, and users can estimate it from a small pilot before buying labels.