We are building a local agent that can write short descriptions and topics from source material, then check its own work. The goal is not just to get an answer out. We want it to finish the task without inventing facts, manufacturing problems or damaging something it already got right.
That last part has been harder than it sounds.
During our own tests, the reviewer sometimes reached the right decision and supplied a false reason for it. Other times it rejected a useful answer while missing a defect we could check directly in the text.
The hallucination was not always in the answer. Sometimes it was in the review.
The rejection was right but the accusation was wrong
In one of our evaluation cases, a description claimed an improvement that its source did not establish. The source described an activity without measuring the claimed benefit.
The reviewer correctly identified the unsupported result. But it also accused the description of directly copying the source.
That was not what happened. The claimed improvement was absent from the source, and the text was not a copied source sentence. There was a real factual problem, but the copying accusation did not explain it.
We also found a review that reached the correct unsupported-result conclusion while mis-describing the evidence it cited. The cited passage concerned the lack of measurement; the explanation described a different passage about volunteers distributing forms.
Those are actual findings from our tests, not hypothetical examples of how a critic might fail.
This matters if the critic will guide another model's revision. A false copying accusation could encourage the writer to paraphrase the unsupported result instead of removing it.
Counting the rejection alone would hide the difference between finding a problem and accurately explaining it.
Suspicion was not the same as accuracy
The reverse problem appeared too. The reviewer objected to a source-faithful description for omitting a volunteer count and a detail about handing out forms. Neither detail was necessary to support what that description actually said.
We wanted concise summaries, not miniature copies of every source detail. Our evaluation allowed harmless omissions. Treating every missing detail as a factual failure pushed against the task we wanted the agent to complete.
Meanwhile, the same paired test included an actual complete source sentence copied into a candidate. Both reviewer conditions missed that copying defect. They also missed structural violations that a separate program could identify.
That combination was the important finding: false accusations about copying in some outputs, missed copying in another. Turning up the critic's suspicion was not a reliable substitute for getting the distinction right.
The same score hid a win and a loss
We tested 12 inputs under two prompt conditions: the existing instructions and a version with additional procedural guidance. This comparison changed the guidance, not the model weights.
Each condition got six final decisions right. Once we required the checks and explanations to be supported as well, each condition had only three fully qualifying reviews out of 12.
The total stayed the same, but the cases did not. The added guidance gained one fully qualifying review and lost another that had qualified under the original instructions.
Looking only at the improved example would have given us a progress story. Looking at the paired results gave us a different conclusion: this change had not earned promotion.
This comparison was part of several days of development, hands-on human review and cross-family critique. Two additional model families independently regraded all 12 saved candidates and 24 critiques with condition labels and earlier scores hidden. Both reproduced the same fully qualifying results, including the same gain and regression. We preserved disagreements about individual judgments and checked literal and structural claims directly against the source text.
We are evaluating the critic as a component
The useful change in our work has been separating questions that a single pass rate can blur together. Did the critic make the right decision? Were its accusations true? Did its cited evidence say what it claimed? What did it miss? Did the change break a case that previously passed?
We kept the failed outputs and the regressions. We did not replace the posting model or call the broader agent qualified because one part of a review looked convincing.
Adding a critic has not removed the need to verify. It has given us another component whose mistakes need to be understood before we let it steer revisions.
The question we are carrying forward is simple:
If the writer followed this explanation, would it fix the actual defect?
A right verdict is a start. A truthful explanation is a separate achievement.
*AI disclosure: This article was drafted by an AI assistant from our own development records.
Top comments (4)
"The hallucination was not always in the answer. Sometimes it was in the review." This is the eval trap that most judge setups walk straight into — scoring the verdict and ignoring the justification, so a right answer for a fabricated reason counts as a win.
Your downstream point is the one that makes it urgent: if the critic feeds a reviser, a false "you copied the source" accusation teaches the writer to paraphrase the unsupported claim instead of deleting it. The wrong explanation doesn't just mislead you, it actively steers the next model toward the wrong fix. We've seen that exact loop.
The move that helped us was splitting the critic's job into two scored dimensions: was the defect real and does the cited evidence actually support the accusation. Grading them separately surfaces precisely the "verdict right, explanation wrong" cases your paired test caught — and it stops a suspicious critic from getting credit for noise. Your point that the same aggregate score hid a gained review and a lost one is the strongest argument I've seen for never trusting a single top-line number on an eval. Are you doing the paired/per-case diff by hand, or have you automated detecting a swap like that?
Thanks, Max. You put your finger on the part that matters most: a critic can identify a real defect and still give the writer a false reason to change the draft.
We’ve tested that feedback-to-revision loop, not just the critic’s verdict. In a 12-case development run, one critic correctly challenged an invented benefit but also falsely accused the draft of copying. The writer’s revision then changed “we haven’t measured a difference” into “we found no difference”—a new factual error. Because the writer received multiple findings, we can’t say the copying accusation alone caused it. We can say the final answer got worse after the feedback.
And yes, we pair before-and-after outputs to flag gains and regressions. Literal faults can be checked mechanically; source-grounded and close-paraphrase judgments still need separate review. Your two-part test—whether the defect is real and whether the evidence supports the accusation—is one we should apply to every finding. We also check the third part: what happened when the writer acted on it.
Scoring the explanation separately is the right call, and the 3-of-12 vs 3-of-12 table is the honest way to show it. One number I'd add to the paired view: the change gained one review and lost one, so the discordant pairs are 1 vs 1. Even a 4-0 split on 12 inputs only gives an exact two-sided sign-test p of 0.125, so at this size the comparison can rule out a big regression but cannot show a small gain. Your decision not to promote it is the correct one, and the pairing is what makes it visible.
The false copying accusation is a nice case for a cheap programmatic check: if the critic says 'copied from the source', test it with longest-common-substring or n-gram overlap against the source. That turns one class of explanation into something you can score without a second model.
Did the two other model families agree with each other on which critiques were unsupported, or mostly on the verdicts?
Thanks, Arhan. Counting the discordant pairs is exactly right: the guidance gained one fully qualifying review and lost one. That’s no net gain, and I wouldn’t use 12 cases to rule out regressions on unseen work. It didn’t earn promotion.
The two other families agreed on more than the final verdicts. Their true/false calls matched on every finding in 20 of the 24 critiques, and both identified 21 specific false findings. They agreed on exactly which six critiques fully qualified—three under each condition. They disagreed on findings in four critiques, though, and their decision-only totals differed, so I wouldn’t say every explanation was independently confirmed.
We also ran an exact source-sentence and structural check. It caught a copied sentence both critic conditions missed, and flagged several “direct copy” accusations with no complete sentence match. Your overlap suggestion could make that triage stronger. I’d still keep close paraphrase and whether a source supports the meaning of a claim as separate judgments; an overlap threshold can’t settle those by itself. Good question—it gets right to what “the graders agreed” actually means.