A push gate that quizzes you on your AI's code ships its own unresolved false negatives

npm install -g sphinx-understand, then git push. Git stops on the branches you protected. A model that never saw your chat session reads the diff and asks you one question about what the change actually does. Get it wrong and it explains why, then asks a different scenario built from the same concept. Get it right and the push continues. sphinx yolo lets the push through and logs the skip.

That's Sphinx, MIT licensed, 117 stars, a Git hook for agent-assisted development. The pitch on its site is a Sun Tzu quote: you can outsource your thinking, but not your understanding. The Show HN thread hit 242 points and 85 comments and split almost evenly between "who is this for" and people who built the same thing. There's a second project in that thread doing a PR-level version.

I don't have much to add to that argument. What I care about is that the repo ships an eval/ directory, and the eval directory is a better artifact than the product.

The gate has a grader, and the grader has false negatives

A comprehension gate is a test suite where the thing under test is a human explanation written in free text. The grader is a model. So you've built LLM-as-judge, pointed it at prose, and wired the result to a hard stop on your push. That's a design with a very specific failure mode, and most projects shipping one never write down what it is.

Sphinx wrote it down. eval/RESULTS.md ends a live run with this line: "The result is not an all-green semantic evaluation, despite all 560 guarded unit/integration tests passing."

560 green tests, and the author is telling you the semantic evaluation is not green. That distinction is the whole piece. Unit tests verify the pipeline moves bytes around correctly. They say nothing about whether the grader would have accepted your answer.

Two answers that were right and got marked ambiguous

The repo keeps a file called eval/fixtures/known-false-negatives.json. Its scope field reads: "Synthetic fixture questions and authored answers only. These are known failures, not passing model assertions." The eval README calls them "unresolved regressions, not passing tests."

Both entries are a human answer, the verdict the model gave it, the model's feedback, and an independent finding that the verdict was wrong.

Case one is a CSS question. The diff adds a .card .button rule that sets only padding-block, on top of a base .button rule. The question asks you to predict per axis what changes for a card button and for a toolbar button, and why. The authored answer says the toolbar button is unaffected because the new selector needs a card ancestor, and that horizontal padding is unchanged because the base rule still supplies it.

The grader marked it ambiguous, with this feedback: "It never states that the vertical (block-direction) padding is what changes for Complete task, so the required per-axis prediction is missing."

Read that again against the answer. The answer described new card padding plus unchanged horizontal padding, which leaves exactly one axis. It named the ancestor scope. It named the unchanged toolbar boundary. The grader's pass condition had collapsed into a vocabulary check for the literal word "vertical."

That isn't a fluke of one cheap route either. A later run replayed the same question and answer against a stronger model, and it reproduced the false negative in one request, 4.6 seconds. The writeup notes the stop condition ended the comparison there, so there's no claim the stronger model fails the broader matrix.

Case two is authorization. The patch merges a missing-document check and a cross-tenant check into one condition returning the same { status: 404 }. The authored answer says a different tenant gets a bodyless 404 "just like a missing id." The grader wanted an explicit "no" and the literal combined-condition mechanism.

The part that should bother you: the writeup says the source-audited acceptance criteria themselves unnecessarily named a single combined condition. The audit that was supposed to make grading fairer had baked the vocabulary requirement into the key. The author is explicit that both question keys and the accepted replacements were source-correct, and that source correctness did not make the sufficient answer fairly graded.

The replay protocol is what I'd actually copy

Here's the part I think is worth more than the tool. The eval README documents how to replay a known false negative:

To replay one, select its source.caseId from CASES, overlay the saved source.definition, and call createFixture(fixture, config). Read immutable evidence with createReviewWorkspace(snapshot, config, { env }) and canonicalEvidence(snapshot, record.source.requiredPaths, workspace), then build a review bound to that new snapshot.id and call evaluator.gradeAnswer(...). Use the existing metered evaluator transport and an explicit live budget.

Then the constraint that makes it a real test instead of a demo: "do not reuse an old temporary snapshot ID, re-audit to an easier question, or replace the saved human answer."

That's the difference between a benchmark and a marketing page. The saved answer is frozen. You can change the grader, the criteria, the prompt, the model route, and re-run against the identical input. If you rewrite the question until it passes, you've changed the test, and the file says so out loud.

There's a matching guard on the other side. npm test and npm run check preload a network guard into Node and its spawned children, rejecting outbound fetch, HTTP, and socket calls while allowing loopback. The stated reason is that a partial model mock shouldn't be able to silently reach a real provider. For a project whose whole job is making deterministic claims about a nondeterministic grader, that's the right instinct.

A failure you can't read is worse than no failure

The configuration case is the other lesson. The fixture asked about a BATCH_SIZE precedence rule using the phrasing "BATCH_SIZE is set to 5." The independently authored probe adapter only recognized BATCH_SIZE=number, so it answered a different scenario with a different value.

The writeup's conclusion is the useful bit: "That is an uncovered scenario, so this result does not establish how the model would grade a sufficient answer to the stated input."

A red result from a fixture that never presented the stated input tells you nothing about the grader. It tells you about the fixture. If your eval report doesn't distinguish those two, every red number is noise you'll spend a week chasing.

The same run also produced a raw response that called the prediction ambiguous, acknowledged no material misconception, and then emitted incorrect with no required transfer. Validation rejected the malformed grade. That's a separate failure class again, a response-shape bug wearing a semantic failure's clothes.

One more line worth stealing

After the false-negative replay, the writeup says the ambiguous result "leaves the checkpoint pending and grants no approval; safe noncompletion is separate from fair grading."

Those are two different properties and they get conflated constantly. A gate that fails closed when the grader is confused is safe. It is not correct, and it is not fair to the person who wrote a good answer. Teams adopting any comprehension gate, or any AI review gate, tend to only verify the first one.

The same question applies to AI code review tools generally. When you're comparing options, ask what the tool publishes about the runs it got wrong. A changelog of fixes is not the same artifact as a frozen list of answers that are still being graded incorrectly. Some projects publish the second thing and some don't, and the ones that do are easier to trust with a blocking check in your pipeline. It's the first question I'd put to Kodus or any other reviewer you're considering wiring into CI.

The repo also bumped its assessment policy to contract version 3 so questions and receipts prepared under the earlier contract are not silently reused. If you version your prompts, version the artifacts they produced too, or a stale receipt will authorize a push under rules that no longer exist.

What I'd check before wiring a quiz gate into your push path

Four things, and they map to the failure classes above.

Whether the project keeps a file of things it currently gets wrong. A pass list is easy. A frozen list of unresolved failures is the artifact that tells you the authors are measuring.

Whether you can re-run a saved input through a changed grader with an explicit cost budget. If replaying a fixed answer requires re-authoring the question, you can't measure a fix.

Whether an uncovered input phrasing is reported as uncovered instead of failed. Otherwise your dashboard fills with failures that never happened.

Whether "it blocked the push" is being reported as a success metric on its own. Blocking is easy to achieve by grading badly. The metric that matters is how many sufficient answers it accepted.

None of that argues against the tool. It argues that the eval folder is the reason to read the repo, and that a gate which grades your understanding needs its own understanding checked first. The authors published the two cases where theirs wasn't.