This one first ran on mrsaynothing.dev — What decision models can't do, the against-the-grain companion to yesterday's explainer.
Everyone is piping the new decision models into production because the release notes were exciting. Nobody has published an accuracy bench. That gap is this post.
The short version: use them for decisions, never for reasons. Two of the six are non-commercial. The printed limits — option counts, token ceilings, language gaps — are real and fail silently. And the confidence number is a claim until somebody benches it on real labels.
The six limits
- There are no reasons in the box. A decision model returns a label and a number. Why that label — the evidence, the counter-argument, the doubt — is not in the output, at any price. The moment your pipeline needs a why, you are back to a chat model or a human.
- Agreement is not verification. Ask three models and get three matching answers, and you have one opinion in triplicate. A unanimous wrong answer arrives in the confident tray — sorted, stamped, and wrong at 0.99. Consensus needs an independent reference to mean anything, and none of these models brings one.
- The license walls are real. Julia-1, Laya, lev and Kev-4B are Apache-2.0. Nimble and OpenJev are CC BY-NC 4.0 — non-commercial, full stop. The llama.cpp runtime is MIT, and that means nothing about the weights: the checkpoint card outranks the repo's license badge every time.
- The modality map has holes. Laya is English-only — don't route multilingual traffic through it. Nimble is text-only locally. OpenJev's vision input needs its own projector. Kev-4B ships without the reference-date preprocessing it expects. Each gap is printed; none of them is visible from the API's side.
- The limits are printed on the box. Julia-1 takes 2–20 options and an 8,192-token combined question — go past either and the failure is silent, not an error. Feeding a decision model a 12k-token document is not stretching it; it is using a different product than the one that shipped.
- Serving is not proof. A new endpoint returning numbers is a plumbing fact, not an accuracy fact. The published per-million-token prices and the serving support say nothing about whether the probability is honest on your labels. Only a bench on real traffic answers that — and no one has published one yet.
What holds despite all six
The scope gets smaller, not the value. One-pass routing, gating and triage — where the output is a decision and a human owns the consequences — is still the cleanest fit local AI has offered in a year. Small, fast, free of per-token bills, and honest about being a stamp. The failure mode to design for is the confident tray: set a threshold, escalate below it, and never let the stamp be the last word on anything that matters.
Remember this: a probability is a claim. Stamped ≠ sorted right. Bench on your labels before the tray is the truth.
The bench is next — Julia-1 against Laya on real routing labels, calibration per size, latency per hardware. If the confidence numbers hold, the wave earns it. If they don't, this post already said so.
Which of the six limits bites hardest in your stack — and has anyone seen a published calibration bench for any of these models yet?
Top comments (9)
The license split surprised me most: 4 of 6 Apache-2.0, but Nimble and OpenJev are CC BY-NC — none of the release-notes excitement mentions that a commercial pipeline legally stops at those two. Which of the six limits did I underweight?
@deanlee "Shared covariance masquerading as confidence" is the phrasing limit two deserved — nothing in the release notes claims the checkpoints disagree independently, and you are right that mostly they cannot. The asymmetric-loss half is the one we keep circling: a scalar confidence ships with no cost matrix attached, so until one does, the honest routing rule stays boring — under the calibrated bar, it goes to a human queue. A calibration bench on the six checkpoints is on the list; if the numbers hold, they land in a ledger, not a launch post.
48 hours, the real numbers: 5 reactions, 10 comments — this thread is carrying the piece — and 2 sessions on the site copy all week. Ten to one, the argument beats the click-through.
The second limit, treating agreement across models as verification, is where the balance-sheet risk concentrates. In financial pricing, running three correlated models that share underlying training distributions does not buy you an ensemble; it gives you the exact same tail blindness at three times the inference cost. If all three checkpoints were pre-trained on the common web crawl, unanimous agreement is just shared covariance masquerading as confidence.
The other structural blind spot is calibration under asymmetric loss. A raw softmax score or confidence scalar treats a false positive and a false negative as having identical payoffs. In real production triage, the cost matrix is almost never symmetric. Letting a corrupted payload pass downstream can break an entire pipeline, while routing a borderline request to human review is a fixed queue cost. Until a decision model lets you condition the decision boundary on the operational loss function rather than raw likelihood, the confidence number is just an unhedged probability claim.
The explainability gap is the one that bites hardest in practice. I've watched teams ship a local model because the benchmark numbers looked clean, only to get stuck when compliance or a customer asks "why did it pick that label?" and the only honest answer is "the weights said so."
Confidence scores are especially treacherous — they're calibrated on the training distribution, not your production drift. Seen too many dashboards where 0.92 confidence meant "wrong but confident" once the input shifted slightly. The only thing that's saved us is shadow-evaluating against a held-out slice of real traffic for at least two weeks before any model touches a user-facing path.
Non-commercial licensing helps adoption, but it doesn't solve the operational burden of maintaining your own eval harness. That's the hidden cost nobody budgets for (site: labagent .tech)
Compliance asking 'why' with nothing but weights to point at is the gap no benchmark table captures. The boring defense holds up best: log the calibrated score, the inputs, and the routing decision together, and treat SHAP-style stories as debugging aids — never as the audit artifact. A score calibrated on the training distribution is a claim about yesterday's traffic; a reliability diagram on a live slice humbles more dashboards than any drift alert. When your teams caught it, did recalibrating on production data save the model, or did it end up behind a human queue?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.