DEV Community

Cover image for Bugs Are Innocent Until Reproduced: Building Verdict, an Evidence-First Agent Harness
Himanshu Kumar
Himanshu Kumar Subscriber

Posted on

Bugs Are Innocent Until Reproduced: Building Verdict, an Evidence-First Agent Harness

#ai #testing #opensource #showdev

Separates LLM guesses from hard proof

Most flaky bug reports end in one of two places: "cannot reproduce" or a patch nobody can prove fixed the problem.

I built Verdict around a stricter idea:

Bugs are innocent until reproduced.

Verdict turns a GitHub issue into a bounded investigation. It runs an approved command repeatedly under approved conditions, keeps every observation and refuses to claim a reproduction unless the evidence crosses a deterministic threshold.

This is not an autonomous patch generator. It is an evidence-producing agent harness for the difficult step that comes before a patch.

Why another bug investigation tool?

An LLM can read a stack trace and propose a plausible explanation quickly. Plausible is not the same as reproduced.

For an intermittent failure, the questions that matter are concrete:

  • Which condition actually triggers it?
  • How often does it fail under that condition?
  • What happens under a contrasting control?
  • Which repository range does the evidence support?
  • What regression test would prevent the same failure from returning?

Verdict treats those as an experiment rather than a conversation.

The three-act investigation

Verdict uses three bounded subagents:

GitHub issue
    |
    v
Hunter: find the trigger
    |
    v
Surgeon: localize the change
    |
    v
Insurance: keep it fixed
    |
    v
Maintainer review
Enter fullscreen mode Exit fullscreen mode

Hunter

Hunter searches only the condition matrix and command budget approved by the maintainer. Successful, failed, partial and unresolved runs all stay in the evidence ledger. An inconvenient result cannot disappear just because it weakens the story.

Surgeon

Surgeon narrows the reproduced condition to the smallest suspect range the records support. Static inspection stays visibly different from a proven execution boundary. Surgeon does not author a patch.

Insurance

Insurance converts the reproduction into a regression plan: the test name, fixture, failing assertion and publication manifest. A draft pull request can only be created through a workflow the maintainer explicitly approves.

Each act is allowed to claim less than the act before it. None can talk the deterministic reducer into a stronger verdict.

The reproduction is a record, not a screenshot

Verdict reproduced TrueForge issue #417, where snapshot registration can wait indefinitely when an upstream request never resolves.

The pinned runtime used @truefoundry/[email protected]#DaytonaSandboxProvider and ran two conditions:

Condition Result
daytona-stalled-endpoint 10 of 10 runs matched, REPRODUCTION_PINNED
daytona-responsive-endpoint 0 of 10 matched, NOT_REPRODUCED

The control is the important half. A condition that fails every time next to one that never fails is stronger evidence than twenty failures with no contrast.

Anyone can recompute the record:

pnpm --filter @verdict/agent verify:runtime-evidence
Enter fullscreen mode Exit fullscreen mode

The verifier returns:

{
  "verdict": "REPRODUCED",
  "stalledRuns": 10,
  "responsiveControls": 10,
  "provider": "@truefoundry/[email protected]#DaytonaSandboxProvider",
  "canonicalSha256": "a8bb5dd22e083782bd7782fccb0a1343b59fc77ea8525b6358fecc9b5b8baffa"
}
Enter fullscreen mode Exit fullscreen mode

The evidence binds the observations to the TrueForge session, Hunter thread, repository commit, npm provenance commit and shared source blob.

Exploration and proof are different systems

The model gathers candidate observations. It does not decide what those observations prove.

Verdict's evidence contract is simple:

  1. Issue text and repository content begin as untrusted input.
  2. The investigation may use only approved commands, knobs and budgets.
  3. Every accepted observation must match the evidence schema.
  4. Pure reducers decide which claim the records support.
  5. Missing or conflicting evidence produces an honest partial result.
  6. A maintainer controls the only public write.

The same records always produce the same verdict. That is the boundary between an agent exploring a problem and a system making a claim.

What is live and what is a fixture

The recorded case renders the executed artifact. It includes both conditions, all twenty runs and the recomputable hash.

The interactive workspace is a conceptual fixture. Every generated value is labelled. It is not quietly presented as live runtime evidence.

That distinction matters for a product whose entire argument is that a claim needs a record.

Approval remains a maintainer decision

Verdict also exercised its publication boundary against real GitHub. After explicit approval, a nonce-bound workflow ran, verified the external reproduction reference and created a draft pull request in Verdict's repository. The upstream TrueForge repository remained read-only.

The workflow proof says runtimeReproducedByThisWorkflow: false. That is deliberate. The provider run reproduced the bug. GitHub Actions verified the harness and published the independently checkable proof. Combining those into one vague "verified" flag would erase the boundary.

Qodo reviewed the claims as well as the code

Every substantive change went through a pull request reviewed by Qodo before merge.

The most useful findings were not dramatic crashes. They were mismatches between the implementation and what the project claimed:

  • A landing card still described the real reproduction as simulated.
  • CI trusted the recorded verdict instead of recomputing its hash.
  • README copy claimed coverage on every push while the workflow covered main and pull requests.
  • A malformed CSS selector silently failed after a cleanup.

Those reviews fit the product philosophy perfectly: do not ship a stronger claim than the evidence supports.

Current verification

The repository runs the same gate locally and in CI:

pnpm lint
pnpm typecheck
pnpm test
pnpm build
pnpm --filter @verdict/agent verify:runtime-evidence
Enter fullscreen mode Exit fullscreen mode

The current suite contains 220 tests across the agent, protocol and web packages.

Try Verdict

Verdict is open source under the MIT licence and was built for the WeMakeDevs x TrueFoundry Agent Harness Hackathon.

The goal is not to make an agent sound certain. The goal is to make certainty inspectable.

Top comments (7)

Collapse
 
mansio profile image
Mikhail •

Himanshu, this is another masterclass in evidence-first harness architecture. The execution-proof separation (LLM gathers observations, pure deterministic reducer decides the claim) targets the exact "verification theater" that plagues most agent toolchains.

Three details stand out from an evidence-boundary perspective:

  1. The Control Condition (Dual-Arm Contrast): Requiring 10/10 failed vs 0/10 on responsive control is the only way to eliminate environment artifacts. Twenty failures without a contrasting baseline prove nothing except that the environment might be broken. The negative control is what turns an observation into a scientific proof.
  2. Decoupled Verification Boundary: Setting runtimeReproducedByThisWorkflow: false while CI verifies the harness proof is a brilliant move. Collapsing execution (who ran the code) and verification (who checked the canonical hash) into a single aggregate "verified" flag erases the trust boundary.
  3. Evidence vs Human Documentation: The Qodo finding where the README drifted away from actual CI behavior is the ultimate real-world proof that human/LLM text naturally hallucinates over time, while execution hashes hold the ground truth.

Quick question on the deterministic reducer design: How do you plan to handle flaky / non-deterministic bugs (e.g., race conditions that reproduce at a 15% frequency)? Would the reducer shift toward a statistical confidence threshold (e.g., binomial test against control), or stay strictly deterministic with a larger probe budget?

Incredible work on both Verdict and Airlock!

Collapse
 
himanshu_748 profile image
Himanshu Kumar •

Thank you, this gets to the exact boundary I want Verdict to preserve.

The submitted reducer is deliberately deterministic and conservative. Each condition needs at least 10 valid observations. It returns REPRODUCTION_PINNED only when every valid observation matches. With a full budget, 4 or more matches becomes PARTIAL_REPRODUCTION, 1 to 3 becomes WEAK_SIGNAL, zero becomes NOT_REPRODUCED and an underfilled budget stays UNRESOLVED. The recorded case is accepted only because the failing arm is pinned and the control arm is not reproduced.

For a 15% race, I would not quietly weaken that contract or call one hit proof. The next reducer should be a separately versioned statistical policy with a predeclared trial budget, a minimum effect size, a confidence interval and an exact two-arm comparison against the control. The model would still only collect observations; a pure reducer would make the claim. If the budget cannot reach the declared confidence, Verdict should return partial or unresolved.

That statistical policy is a next step, not something the submitted build already implements.

Collapse
 
mansio profile image
Mikhail •

Perfect. That is the exact line between a scientific agent harness and a prompt-engineering illusion.

Having a dedicated, versioned statistical policy (with a pre-declared trial budget and binomial confidence intervals against the control) while keeping the LLM strictly in the observation collection layer is the cleanest way to handle non-determinism without corrupting the evidence ledger.

If the budget can't cross the threshold, returning UNRESOLVED is the only honest claim a system can make.

Great job on keeping the execution boundaries this sharp!

Collapse
 
vinhnguyenthanhdn profile image
Vinh Nguyen •

Excluding UNRESOLVED from the valid denominator is the right default for most bugs, but it leans one way for this one. The symptom in TrueForge #417 is a wait that never resolves, so on the control arm a run that genuinely stalls and gets killed at the budget before the matcher concludes leaves as an unresolved run rather than a match, drops out of the denominator, and the 0 of 10 contrast comes out stronger from the same bad data. Since the reducer already carries the unresolved count, the cheap tightening is to gate on it for the control arm specifically: a control with a high unresolved share is not a clean zero, it is a second UNRESOLVED.

Collapse
 
eduzsh profile image
Edu Peralta •

The part that landed for me is treating reproduction as a record with a control run, not a chat conclusion. I have watched coding agents invent a plausible root cause, patch around it, and declare victory while the intermittent case never fired once under the same conditions. Pinning the fail case next to a responsive control is exactly what most agent loops skip, because it costs tokens and patience. Curious how often Hunter burns the whole command budget without crossing the threshold, and whether you surface that as a hard stop or as a soft still unproven for the maintainer.

Collapse
 
himanshu_748 profile image
Himanshu Kumar •

That is exactly where Verdict stops the search without promoting the claim.

Hunter has a hard cap of 8 tool calls and at most two sandbox commands: one checksum-pinned bootstrap and one exact reproduction command. I do not yet have a cohort large enough to quote how often the full budget is consumed.

In the deterministic reducer, UNRESOLVED runs are excluded from the valid denominator. If the budget ends with fewer than the required 10 valid observations, the terminal result is UNRESOLVED, carrying the matched, observed and unresolved counts. It is not a failed reproduction and does not authorize a reproduction claim or source-fix PR.

So the mechanics are a hard execution stop with an honest evidence outcome: “still unproven” is surfaced to the maintainer with the missing evidence. If there are 10 valid observations but only some match, the result is WEAK_SIGNAL or PARTIAL_REPRODUCTION, not UNRESOLVED. That separation prevents budget exhaustion from being mistaken for absence.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.