Microsoft's .NET team published the benchmark behind their polyglot unit testing agent. We have been arguing its conclusion for months. We can't cite our own version of it, because we withdrew our number on 2026-08-06 and haven't earned it back.
Both halves of that are worth writing down.
What they measured
The setup is clean. One tool, one model, 152 tasks from real repositories. The only thing that varies is a plugin that makes the agent research the repository, plan, implement, then verify before it claims to be done. A task passed only if the repository built, all tests passed, the agent added at least one test, and it removed none.
| Arm | Completed | Rate |
|---|---|---|
| With the workflow | 140 / 152 | 92.1% |
| Same model, stock | 120 / 152 | 78.9% |
63% fewer failures, from the same model and the same tool.
The breakdown is where it gets useful.
| Prompt type | With workflow | Stock |
|---|---|---|
| Vague, 89 tasks | 88.8% | 66.3% |
| Detailed, 63 tasks | 96.8% | 96.8% |
| Diff-targeted, 15 tasks | 15 / 15 | 0 / 15 |
On detailed prompts the two arms tie exactly. The whole gap comes from vague prompts, where the agent has to work out for itself what to test, which framework is in use, and where the tests belong. So the scaffold isn't making the model smarter. It's taking away the guessing.
Then the per-model results, on 45 .NET tasks:
| Model | With workflow | Stock |
|---|---|---|
| Claude Opus 4.8 | 95.6% | 77.8% |
| GPT-5.5 | 91.1% | 80.0% |
| Claude Haiku 4.5 | 75.6% | 55.6% |
Their own summary of that table is one sentence: "The workflow helped every model. With Opus, it added eight wins with no losses."
There's a reading they don't offer, and I want to be clear that it's mine rather than theirs. GPT-5.5 inside the workflow finished 91.1%. Opus 4.8 outside it finished 77.8%, on the same 45 tasks. Read across the rows instead of down them and the cheaper model in a system came out ahead of the stronger model on its own. Microsoft compares each model only against itself and never makes that claim. I'm making it, from their published table, so weigh it as my inference and not their finding.
Two caveats we'd want applied to us
Their coverage barely moved: 72.4% against 72.2% line, 49.8% against 49.1% branch. The agent also wrote slightly fewer tests than stock, 6,963 against 7,129. So the workflow didn't produce better tests. It produced tests that exist, build and pass, more often. They say so themselves, and it's a completion result rather than a quality one.
The gains also concentrate where the request is underspecified. On detailed prompts the workflow buys nothing at all. That's a real boundary on the claim, and it belongs in the headline rather than a footnote.
We'd hold our own numbers to exactly that. So we will.
Why we can't quote ours
We have a version of this result. On our own harness, switching the verification layer off dropped correctness from 100% to 75%, and switching the guards off as well took it to 50%. Small n, our harness, not a public benchmark, and we say that every time we cite it.
The bigger claim was a lift figure: a cheap model alone, then the same model inside our cascade. We published it. Then we audited our own truth layer and found something ugly. Our two baselines for the same quantity, the same cheap model alone on the same benchmark, disagreed by 3.1 points. One page computed the lift from one baseline. Another page computed it from the other. Each traced back to a real measurement, which is exactly why nobody caught it.
Both lift figures came off every public surface that day, and they stay off until one baseline is re-run.
So here's the honest position. Microsoft has a clean, large, well-controlled measurement of something we believe. We have an unresolved 3.1 point disagreement with ourselves about ours. Their result doesn't repair ours. Someone else agreeing with you isn't a re-measurement of your own work, and being right isn't the same as having measured it.
The part we did measure, and it points the same way
One result of ours survives a second independent run and an adversarial pass, so it can sit next to theirs.
We ran three models over the same 160 HumanEval+ problems, with the same prompts and the same scorer. They finished 92.5%, 91.2% and 90.0%. A 2.5 point spread that reads as interchangeable.
Partition by task instead of by column total and it looks different. 83% of the set is uncontested, where all three models get it right, and the models disagree on roughly 15 to 18 percent of the work across two runs. What makes that citable isn't how big the disagreement is, it's how stable: the same problems come up contested each time. 17 tasks are contested in both runs against 3.6 expected by chance, and ten of them are contested in both runs without any model ever changing its answer.
Which problems they split on is a property of the models. Which model scores highest is a property of the run. That's HumanEval+ Python on our harness, not customer traffic.
Put it beside Microsoft's table and you get one argument instead of two. They measured that the workflow around the model carries most of the gain you can reach. We measured that the ranking you'd use to pick a model is mostly noise. Both point at the same place, and it isn't the model.
What we took from it
One technique, straight away.
Before their agent declares a task complete it runs a set of checks, and one of them reads: "It considers small code changes that should make the tests fail. This is a lightweight form of mutation testing." No full mutation framework. Just a cheap step that catches an assertion which can't fail.
We knew that rule. We wrote it down after a scorer of ours passed its positive control perfectly while being structurally incapable of returning a negative. We wrote it down again when a guard printed OK at every boot and then fired on zero of seven deliberately injected defects. We were applying it from memory, and memory isn't a source. Microsoft made it a step in the loop.
So we built it. Every guard in our tree can now declare, in its own header, an injected defect that must make it exit non-zero. A runner executes each one in a fresh directory and reports whether the guard can still say no. Three grades: PROVEN, BROKEN, and UNPROVEN for guards that haven't declared a control yet. UNPROVEN counts as a warning rather than a failure, because a gate that's red on day one gets overridden by reflex, and a reflex override has stopped being a signal.
We watched it fail before trusting it. A stub guard that always exits zero grades BROKEN.
Then we pointed it at the guard from the story above, the one printing OK at every boot. We injected a wrong percentage where a real measured one belongs. Exit 0. We injected a cost multiple we'd already struck from our own public copy. Exit 0. A third run, with the correct value in place, confirmed the guard really was reading two live surfaces against the real ledger, so a trivial pass couldn't be mistaken for a catch. It graded BROKEN by its own declaration, and it's since been fixed.
As I write this the runner reports 7 proven, 0 broken, 33 unproven, out of 40 guards. That last number moved while I was drafting, because writing this article involved building another checker and the gate made me declare a control for it before it would let me commit. The unproven count is the honest one, and it's the one we expect to be asked about. It's also the point of the exercise. Before we built this, the answer to "which of your guards can still fail?" was "we assume all of them."
A postscript, because it happened while writing this
Fact-checking this article, I flagged four of its own numbers as fabrications. The branch coverage figures. The Haiku row. The three-model percentages. Our own ablation result.
All four were real. I'd checked them against a news summary of Microsoft's post rather than the post, and against the wrong one of our own benchmark artifacts. The summary leaves out branch coverage entirely and never mentions Haiku, so an absence looked like an invention.
We have a rule for this and I broke it anyway: never relay one party's account of another's work without reading the original. I was one hop further out than I thought, which is the only place that error lives.
Two defects did survive the pass, and both were in the draft's account of our own work. A comparison of mine that read as Microsoft's, and a stale guard count.
And I have to finish that sentence honestly, because the first version of this article went out before I caught either one. It was live for about three hours, saying that a guard "is now marked BROKEN" when we had fixed it that morning, and reporting 2 proven of 37 when the runner said 7 of 40. This text replaced it in place, which is why the two paragraphs above are written the way they are. An article about withdrawing a number published a wrong one first. That is the whole argument for having a correction path that edits at the claim instead of appending a note at the bottom, and it is why every figure here now carries a line in a verification block saying where it was re-derived from.
Sources
- Microsoft .NET blog, the polyglot unit testing agent: https://devblogs.microsoft.com/dotnet/polyglot-unit-testing-agent/
- The plugin, MIT licensed: https://github.com/dotnet/skills
Top comments (7)
It’s wild that this piece sits at 0 comments when it answers the exact underlying debate running through the architecture threads right now.
Your execution of mutation testing on guards (PROVEN/BROKEN/UNPROVEN) combined with the "cheap model + workflow outperforming raw Opus" breakdown is gold. It directly aligns with the denominator & falsifiability checks we’ve been discussing.
More people building multi-agent architectures need to read this specific post.
Following up on the repo suggestion you left on the other thread. I am replying here because that comment of yours does not render on the article page, though it still comes back from the API, so there was no way to answer you in place.
We are doing it. Your framing settled the format, so here is where it landed.
The problem with a pile of cases is that a reader has to already know what they are looking for. So the front page is a symptom lookup rather than a table of contents. You arrive with "my test passes but the feature is broken", or "the API says my write failed", or "two of my tools disagree about the same number", and it sends you to the right place in one hop. Any other arrangement just moves the digging somewhere new, which is what you were complaining about in the first place.
Below that they group into five mechanisms, and the grouping turned out to be worth more than the individual stories: the check never ran, the check cannot fail, the check measured a different population, the reading is stale, and the consumer disagrees with the producer. Once you can name the five, you start predicting the sixth in your own system before it bites you.
Every entry carries what else could have produced the same observation, and the one measurement that separates those. Leave that out and you have a war story, and war stories are why this genre is usually useless.
Two honest caveats. Each case gets re-derived before it goes somewhere quotable, and I expect a few will fail that, which strikes me as a good reason to do it. And you get no date from me, because a catalogue about stale claims really should not ship stale.
Thank you for the push. It had been sitting as a vague intention for months and you turned it into a shape.
The symptom-first lookup combined with those 5 core mechanisms is the exact right structure. Categorizing by failure mechanics shifts the repo from a post-mortem archive into an active diagnostic tool.
Also, it's pretty ironic (and fitting for a thread about missing execution paths) that mentioning your handle sent my comment straight into DEV.to's hidden API vault! Glad the signal made it through anyway. Can't wait to run these invariant checks against our own code.
On the vault, the mechanics are worth having, because the article's own subject walked into the room while we were describing it.
Your comment is still there. The API returns it while the article page declines to render it. The signal exists, the read path drops it, and from a reader's side that looks exactly like you never having written it. Present in the store, absent from the read path, and every check that queries the store reports fine.
The practical consequence for us was a rule change. We had treated an API response as proof a comment existed. It proves the comment was stored. Whether it reached a reader is a separate question needing a separate check, and we only started running the second one after a thread of ours turned out to be invisible at a depth we had assumed was a hard limit. That assumption was wrong too, which earns its own entry in the same repo.
Symptom-first survived contact, and the cost of it is worth passing on. A symptom index only works when the symptom carries the name a person would give it at 2am, in the language of what they can see going wrong. Our first pass was almost entirely the second kind. Every entry got titled with its mechanism, which is precisely the thing the reader has yet to discover.
Thank you for reading it. On the mutation states, we have had to add a fourth since, and it changes what you do next.
PROVEN, BROKEN and UNPROVEN all quietly assume the mutated line runs. While consolidating some key handling I added a guard, wrote a test for it, and the mutant that deleted the guard survived. The instinct is to go strengthen the test. Instead I measured the input, and found the condition comparing against a literal the runtime never produces. That branch had never matched once in the entire life of the file. A dead path, faithfully preserved by the refactor, and I had already told my co-founder it was a regression I introduced.
Hence a fourth state, UNREACHABLE. You cannot change the behaviour of a branch that has no behaviour, and absence-of-effect is what the mutant and the original both produce. No assertion over outcomes can separate them, so every add-a-better-test instinct is aimed at the wrong target.
The check that distinguishes it costs one line. Prove the path runs before you blame the test, by asserting the branch is entered. When it never fires you have found something better than a weak test, and something invisible to every other instrument you own.
UNREACHABLE is the exact missing link here. If a coverage tool sees a branch, but execution never crosses the entry boundary, classical assertion checks are useless because the denominator of executions is zero. Catching unreachable code before blaming assertion density saves hours of ghost-hunting in dead paths.
The discriminator turned out cheaper than I expected, and coverage was the wrong place to look for it.
Coverage told me the line was covered. It stayed silent on how many times the mutated line actually executed under the test meant to protect it. Counting executions of that single line separates the two cases in one run. Zero executions means unreachable. Any count above zero leaves you with a genuine assertion gap worth chasing.
The version that cost me the most time wore a different costume. A component in our terminal UI had six green behaviour tests and drew an empty box in the shipped binary, because the form was never started. Focus moved between fields, values persisted, different keys produced different exits. All of that holds equally for a component that was never initialised, so every behaviour test sailed straight past the defect. The suite was green because nothing ran.
Your zero denominator, one layer up. A pass earned by correct code and a pass earned by absent execution look identical from outside. What separated them for us was asserting on the rendered picture, which is the same move as counting executions: go and measure the absence directly.