I run an auditor that reads trading-strategy repos while I sleep. A 1.5B model is part of it, and for a long time I described that part wrong.
The...
For further actions, you may consider blocking this person and/or reporting abuse
The 'locations come from code, never from the model' rule is the load-bearing one — your summarize experiment shows exactly why: fluent, specific, and wrong is the worst failure mode for a checker, because reading it costs more than the check saves. Twelve seconds per question is a great constraint to design against; it forces the discipline that a fast model would let you skip. What does the UNSURE rate look like at temperature 0.2, and did you calibrate it against a larger judge to see whether UNSURE means 'hard question' or 'weak model'?
Temperature is already 0.2 — that is what the CLI passes, so this one is answerable rather than hypothetical: zero UNSURE in every committed run. Thirteen verdict lines across the five reports, all YES or NO, none UNSURE. At N=13 the rule of three puts the upper bound near 20%, so I cannot claim the model is never unsure — only that this sample never saw it.
There is a second problem inside that zero. The prompt allows yes/no/unsure, and the parser defaults to UNSURE when it cannot find a keyword — so UNSURE currently means both "the model declined" and "we failed to parse it". In the artifact those two are identical, which is the same binary-schema complaint as the other thread, one level up.
No, I have not calibrated it against a larger judge. My only cross-check is qwen vs gemma on the same findings, and both are small — that measures disagreement between two weak judges, not whether UNSURE tracks difficulty. Doing it properly needs a stronger judge plus a difficulty label on the same findings, and at four findings the sample is too small to settle it either way. Widening the sample has to come first.
Zero UNSURE at N=13 is a useful bound, not a verdict — the rule-of-three caveat you state is exactly right. The parser conflation is the sharper problem though: until 'model declined' and 'we failed to parse' are separate codes, the UNSURE rate isn't measuring the model at all, so a larger judge would just calibrate a corrupted signal. Is splitting those two the fix you'd do before widening the sample, or do both have to move together?
Agreed that the conflation is the sharper problem — with one correction: in this sample it has not fired yet. Zero UNSURE means zero of all three sources, so those thirteen verdicts are clean. What is corrupted is the next sample, not the current one: if UNSURE ever appears, the stored record cannot say whether the model declined, the parser missed a keyword, or the process died.
So: split first, and do not wait for the widening to move with it. The split is roughly an hour — three separate codes (unsure, parse_failed, model_error), then re-run the same four findings and check that nothing that used to parse now falls through. Widening is the expensive half and it produces data that is unclassifiable under the current schema, so the order matters more than doing both at once.
To be clear about what the split buys: it does not make UNSURE calibratable at N=13 — nothing does. It only makes the next N worth collecting.
Fair correction — the current thirteen are clean, and the corruption is prospective, not retrospective. The sequencing argument is the convincing part: an hour's split now is what makes the expensive widening worth paying for later, since unclassifiable data at larger N is just a bigger version of the same problem. One thing I'd check on the re-run: do the three new codes change anything about how the thirteen read, or is it purely a forward-looking fix?
Forward-looking only — with one gap the split does not close. The thirteen already parsed to yes or no, and re-running the same four findings should reproduce them; if one lands in parse_failed instead, that tells us the old parser was coercing something, which is worth knowing.
But there is a fourth failure mode the three codes miss: keyword coercion. The parser looks for yes, then no, then unsure, and takes the first hit — so a reply like "no, but if you count partial matches then yes" gets stored as NO with no trace of the ambiguity. Splitting the codes does not fix that, because the parse succeeded; it just succeeded on the wrong word.
The cheap fix is to store the raw line next to the parsed label, so any future question about a verdict is answered from the record instead of by re-running the model. That is also what makes the widening auditable later.
So: split the codes, keep the raw line, then widen.
The line in the title has a second checkpoint at write time. A model emits a hint, and then a record format can quietly promote that hint into a verdict. The append-only ledger here took the other fork, and its machine-generated verdict records say something useful about the storage contract before denied findings start getting skipped automatically.
Measured today: across 1,787 stored verdict records the
verdictfield has only ever held two values, 1,749 "passed" and 37 "failed", plus one lone row on a different schema. The third answer has never been written once. Rule 3 makes UNSURE a legal model output. That is a separate thing from a legal stored value, and a binary write schema gives it nowhere faithful to land, so automation collapses it into one of the other two without anything in the artifact recording that it happened. Worth tracing one UNSURE response through the writer and looking at what the saved report actually contains.Those records cover 1,686 distinct task ids. 67 of them were judged more than once, and 4 of those repeat pairs disagree with each other. Stored instability does not average out. It sits there as two equally authoritative rows.
The part that took longest to notice is that a correction mechanism already exists. The signed tag vocabulary includes a tag named
replaces, and in all history it has been used exactly once, on a different record kind, never on a verdict. So not one of those four contradictions is marked as a correction. No field name across the 1,787 contains version, rule, revision, supersede, retract or amend either. Nothing inside a record separates "corrected" from "contradicted", which leaves the tie to be broken by timestamp, a convention that lives outside the record and is not signed with it. Four in 1,686 is 0.24%, quiet enough that no aggregate ever flags it, and it is exactly the subset where the answer was contested.The limit on all of that: the ledger has no truth labels either. A disagreement rate shows the format dropped information, and it cannot say which of the two rows was right. It argues for keeping the supersede slot rather than for picking a winner.
On the flip rate you list as missing, the re-asks are already a measurement. One differing answer in three, on one finding. Widening that to the bundled sample is 3 re-asks x 12 seconds x 4 findings, roughly 144 seconds against the 48-second baseline. Under three minutes, inside the nightly budget the design already assumes, provided each answer is stored next to the exact prompt variant that produced it rather than folded into a single per-finding field.
In the committed pair of runs, does a regenerated report look any different for a finding whose answer flipped than for one answered the same way twice, or does the artifact lose the fact that it was contested?
Short answer: the artifact loses it.
In the two committed reports each finding carries one line per model —
Model check: qwen YES · gemma YES,Agreement: yes— and nothing else. The 10-22 report has CO-01 as gemma NO (3 agree / 1 disagree); the 10-23 report has the same finding as YES (4 agree). Regenerate and the contested state is gone: no history field, no supersede slot, no per-verdict timestamp, no record of which prompt variant produced it. The flip survives only because the two reports were committed as separate files, so git is doing the job the record should do, from outside it.That is your 1,787 records at a smaller scale, and I think your framing is right: the write schema is the bug, not the model. If UNSURE cannot be stored, a contested answer cannot be stored either, and a correction becomes indistinguishable from a contradiction.
One caveat on the pair you are asking about: it is not a clean re-ask. The code changed between the runs (I moved the context window to the evidence line, so line numbers moved from :1/:9 to :10/:12), which is why I would not count 3-of-4 to 4-of-4 as a flip rate. Your 3 re-asks x 12s x 4 findings is the right way to get one — and storing each answer next to its prompt variant is the only version of that which is still checkable a week later.
Taking locations from the code and never from the model is the design choice that makes the rest of this safe. The model can be wrong about whether something matters, but it cannot invent where it is.
The NO, NO, YES result on the same question is a good reminder that a small model gives you a distribution, not an answer. Efe's suggestion of five votes with a 3 to 2 split going to a human turns that instability into a routing rule, which is easier to defend than a confidence score.
On client builds we write down early what the model is allowed to decide and what it may only flag. Triage, ordering and summarising are fine. Anything that closes a ticket or changes data waits for a person or a deterministic check. Agentic Affairs' point about separating UNSURE from a parse failure belongs in that same list, since the two need different handling.
At 12 seconds per question, did batching change the vote pattern at all?
No — and that is the honest answer rather than a modest one. The 12 seconds was one narrow yes/no prompt at a time: four isolated calls, roughly 48–50 seconds total. I never put the four checks into one prompt, so I cannot claim batching is neutral.
I would expect it to move. A batched prompt is longer, and later answers can anchor on earlier ones — which is exactly the contamination the isolated re-asks were there to avoid. The test is cheap: same four checks, five votes each, run isolated vs batched, then compare per-item agreement and how often an item flips. If agreement drops in the batch, the votes have to stay per-question and the 12 seconds is the real cost.
NO, NO, YES from three re-asks at temperature 0.2 is hard to read as a rate. With five votes per finding the split carries more than any single verdict: 5-0 and 3-2 are different statements, and the 3-2 findings can go to the human first. I grade model output with five votes for that reason. At twelve seconds a question, that is a minute per finding, which fits an overnight run. You list the flip rate under prompt shift as unmeasured. Would you report it per check, as the share of findings whose majority changes when the context line moves, instead of one global number? Unstable findings may cluster in a few checks.
That is the better design, and I would take five votes over three re-asks: 5-0 and 3-2 are different claims, and routing the 3-2 findings to a human is a decision the tool can make from the votes alone.
On per-check reporting: yes, but with one condition. The variation I would expect is across checks rather than across findings — a check whose criterion is fuzzy ("no cooldown") should flip more than one that is structural ("the fill reads index i"). A global flip rate averages exactly that away.
The catch is sample size per cell. I have four findings and four checks, so a per-check table is one finding per cell — descriptive, not a rate, and I would label it that way rather than dress it up. If I widen, the target is ten findings per check at five votes each: 4 checks x 10 x 5 = 200 model calls, about 40 minutes at twelve seconds. That fits an overnight run, and it produces a table instead of a number.
The sentence that carries this post concerns locations: they come from code, never from the model. A small model can then triage, because a wrong guess costs a check, not a shipped claim.