This is the first contest I have entered. I built the honesty controls before I built the agent, which is probably backwards for a hackathon and ex...
For further actions, you may consider blocking this person and/or reporting abuse
I find interesting that the failure is not really in the finding itself, but in the system’s model of its own history. That seems like a particularly difficult class of bug because every individual component may still behave reasonably. The detector finds something, the memory layer contains valid records, and the comparison logic returns a result. The error appears only when those pieces jointly construct the wrong interpretation of state.
I think this is one of the reasons provenance becomes increasingly important as systems become more autonomous. It is not enough to know what the system knows; you also need to know where that knowledge came from, when it was created, and under what assumptions it should still be considered valid. Otherwise memory stops being context and starts becoming an unchecked source of truth.
your middle paragraph is exactly the run. every component behaved reasonably in isolation. the detector quoted real bytes that exist verbatim and exactly once in the named file. the frozen prior was valid. the validator returned a structurally correct result and was right to. the error only exists in the joint construction, and no single piece is the one that failed. that is the part i had trouble writing down and you put it in one sentence.
the diagnosis i would push back on is "the system's model of its own history," because the history was clean and i can show it.
prior committed 5bf10acd 2026-08-29 05:55:49Z
run executed 2026-08-29 12:02:41Z
governing_prior_sha256 93820ea5...
prior_freeze_commit 5bf10acd
six hours and seven minutes, hashed before the run, and the receipt binds that exact commit and that exact digest. origin, creation time, and validity conditions all present, all checkable by someone who does not trust me. that is the provenance you are describing, and it was already there.
it did not help. the record was right and the agent's use of it was wrong. K1 as frozen reads "a verdict computed over an empty collection returns success... treated as nothing wrong rather than nothing checked." the code it cited exits 2 on empty, which is a hard failure, the inverse. the agent described that inversion in its own reasoning field and filed the finding under K1 anyway.
so the distinction i would draw is that provenance is a property of the record. this defect lives in the edge between a new observation and a record, and nothing about a record's pedigree constrains whether a fresh finding actually belongs to it. a perfectly sourced, correctly dated, cryptographically bound condition can be cited by something that is not an instance of it, and every provenance check still passes.
where your point does land, and i think it is the harder half: once a wrong association is written down, it inherits the record's credibility. that one i have not solved.
Yeah, that's a fair correction. The prior seems exceptionally well grounded. The failure happens one step later: the agent observes something new, retrieves a valid prior, and then incorrectly decides that the observation is an instance of that prior.
That makes me wonder whether the association itself needs to become a first-class auditable object. Not just “which prior did the agent cite?”, but why did this observation satisfy the conditions for that prior? In other words, provenance tells us where the record came from, but we may also need something closer to traceable classification evidence for the edge connecting observation and record.
And I think your last point is the really hard one. Once the bad association is persisted, it can inherit the credibility of a perfectly valid prior. At that point the system is not merely making one wrong classification but it is potentially manufacturing trusted history from an incorrect edge
That seems much harder than detecting a bad record, because the components on both sides can remain completely correct.
you are describing something that already exists in the artifact, and i went and read the repo rather than answering from memory. the answer is more uncomfortable than i expected.
the finding object carries confidence_basis. it is required by the response schema and it is exactly the field you are asking for: the agent's own account of why this observation satisfies that prior. verbatim from the run:
"The code structure directly matches the pattern: a filter operation that can produce an empty
collection, an explicit check that treats empty as failure, but no corresponding check that
non-empty means success."
and K1, frozen and hashed before the run:
"A verdict computed over an empty collection returns success. An empty array ... is treated as
'nothing wrong' rather than 'nothing checked'."
treats empty as failure, against treated as nothing wrong. the justification for the edge states the inverse of the condition the edge points at, in the same object, and it has been sitting in the published receipt since the run.
what enforces it in code: the schema requires it non-empty, and the validator rejects it if it is a bare number like "87%". that is the whole of it. no code path compares it to the condition it is justifying. i grepped the entire repository for this rather than one file, after someone pushed me on whether i had actually checked.
but here is the part that answers your question properly. the contract does specify the comparison. row B3 of the frozen contract names the hazard as "a wrong-reason pass, a result obtained for a cause unrelated to the check," and its stated control is: every finding records confidence_basis, and a named breaker seat compares observed reason to intended reason on every row.
so the check is not missing from the design. it is assigned to a person. and in this run it either did not happen or did not catch it, because the contradiction shipped.
that is the actual shape of the gap, and it is one we already have a name for: a rule that depends on someone choosing to perform it is a request, not a control. the field is first-class. the comparison is a human promise wearing the contract's authority.
the mechanized version is small and i have not built it: make the cited condition's text and confidence_basis both required inputs to a check that must agree they describe the same direction. what stops me is that "do these two pieces of prose agree" is itself a semantic judgment, and putting a model in that seat gives the checker and the checked a shared failure mode again. that is the same wall, one layer out.
on inheritance you are right and i have nothing. the priors are hashed, and there is a test that rejects a successor prior which quietly rewords a condition, digest mismatch, verified. none of that stops me from writing a wrong association into the next prior deliberately because it looked right. it would then be frozen, dated, cryptographically bound, and true forever as far as any check can tell. a false thing with perfect provenance is worse than a false thing without it.
The wall might be avoidable here if the semantic judgment moves into the frozen prior instead of happening after the model writes prose. K1 can carry an executable predicate such as
empty_count == 0 -> exit_code == 0. The new finding can carry the observed predicate from the quoted bytes,empty_count == 0 -> exit_code == 2. The association is then rejected by a truth-table check; no second model has to decide whether two explanations “mean the same thing.”That doesn’t solve arbitrary prose classes, but it changes the contract: a class without a machine-checkable discriminator can remain context, but it can’t be a
CONFIRMS_KNOWNtarget.confidence_basiscan stay for humans, while the edge only inherits credibility after the predicate passes. In this run, that would have failed before the wrong K1 link could be written into the next prior.this gets past the thing i said i could not get past, and it does it by removing the part i said would poison it.
i had argued that mechanising the check meant asking a model whether two pieces of prose describe the same thing, which hands the checker and the checked a shared failure mode. you deleted the model from the seat entirely. a predicate on the class and a predicate on the finding is a truth table, and truth tables do not have opinions.
i ran your example against the actual run before answering.
K1 predicate empty_count == 0 -> exit_code == 0
observed predicate empty_count == 0 -> exit_code == 2
antecedents match, consequents do not, association rejected. so yes, that specific wrong link dies before it can be written, and it dies without anything reading confidence_basis.
two costs worth naming, because neither is fatal but both are real.
first, neither field exists. a frozen known_condition currently carries description, name, why_listed, blind_spot_evidence, instances_already_found_by_humans and measured_model_blind_spot. no predicate. and a finding carries condition, path, exact_bytes, why_it_matters, evidence, novelty, known_condition_id, confidence_basis, not_established and repair. also no predicate. so this is two new required fields on two frozen schemas, and the prior is hashed, which means it is a successor prior rather than an edit.
second, and this is the part i want your read on: the observed predicate still has to come from somewhere. for this case it is nearly mechanical, an if on a length comparison and a literal exit code. for a class like "a derived label outranking recoverable evidence" i do not know how to extract a predicate from arbitrary bytes without a model doing the extraction, which puts the model back in the loop one step earlier.
but your contract clause is the answer to that and i think it is the strongest sentence in your comment. a class without a machine-checkable discriminator can remain context but cannot be a CONFIRMS_KNOWN target. that does not require solving prose. it requires admitting which classes are decidable and refusing to let the undecidable ones carry a verdict. K1 becomes checkable, K3 stays advisory, and nothing pretends otherwise.
that is a smaller and more honest system than the one i was trying to build, and it is the first proposal in this thread that would have failed the run rather than explained it afterwards.
The distinction between a model proposing a candidate predicate and a model adjudicating the match is what keeps the loop clean. The finding-generating model can emit the predicate alongside the quoted bytes, but the evaluator doesn't trust the model's prose explanation—it just executes the predicate or checks it against the AST trace. If the generator emits a bogus predicate that doesn't match the AST of the quoted bytes, a syntactic checker rejects it immediately.
For classes like derived labels outranking evidence, the hard truth is that if the property can't be reduced to an AST pattern or a deterministic test harness, it is fundamentally an open-ended semantic claim. Putting a model in charge of deciding whether arbitrary bytes fit that concept is just reintroducing the judge that has the same blind spots as the worker.
Treating those as advisory context rather than gating verdicts is the right trade-off. It forces you to write executable invariants for the classes you actually want to gate on, and keeps the undecidable ones from quietly minting false confidence.
the split is right and i can tell you exactly where my system already draws it, including where it does not hold.
on a finding, four fields get validated. two are executed against the artifact and two are only checked for grammar. path has to exist in the frozen corpus. exact_bytes has to occur exactly once in that file, not roughly, exactly once, or it throws. those two are real. confidence_basis only has to be non empty and not a bare number, so you cannot say 85 percent, you have to say something. not_established only has to be a list of strings. neither of those is verified against anything. so the honest count is two executed, two grammar, and i think that ratio is the actual state of most systems that claim to check their own output.
and there is a limit even on the executed one that i had to sit with. exact_bytes occurring exactly once proves locatability, not relevance. the citation is provably real and provably unique in the file. nothing anywhere establishes that those are the bytes that matter to the claim. so the deterministic check closes half the gap, the half where someone cites something that is not there or is ambiguous. it does not touch the half where someone cites something true and irrelevant.
on the derived label class, i want to push back a little because i went and reread it. thats K3 in my frozen set, and it reads: a conclusion stored in place of the raw values it was derived from, where the raw values remain available on the same record. that last clause is structural. does the record carry both the label and its inputs, and does anything recompute it. you can gate on that without a model deciding what counts as a conclusion. so that one is closer to decidable than you gave it credit for.
where you are right, and harder than you put it, is my novelty rule. a reported condition is NEW only if it matches no known entry by mechanism, not by wording. matching by mechanism and not by wording is not reducible to an AST pattern, and it is the field that gates whether a run counts as a finding at all. so the undecidable thing is not sitting in an advisory corner. it is the gate.
and to be straight, there is no predicate or ast field on my frozen schema at all right now. what you are describing is not something i have and turned advisory. it is something i have not built.
That K3 clarification is really sharp. If the record schema enforces both the raw inputs and the derived field on the same object, checking whether they match or if recomputation passes isn't semantic judgment at all, it's just an executable equality check. That's a much cleaner boundary than I gave it credit for.
The locatability versus relevance gap is where almost every automated evaluation loop gets bruised. Unique byte matching at least guarantees the model isn't hallucinating nonexistent files or phantom symbols, but it can still cite completely unrelated code that just happens to look plausible. The closest thing I've seen work reliably without an LLM judge is binding citations to execution artifacts, like exact stack frame deltas or tool call input/output diffs, so the cited bytes have to sit inside the active mutation path rather than anywhere in the repo.
The novelty gate is the real kicker though. Once novelty requires matching on mechanism rather than text similarity, an LLM judge usually sneaks back in at the most critical choke point. If you ever build that out, I'd be curious whether you end up trying semantic clustering or if you end up fingerprinting structural traces to keep it deterministic.
your execution artifact idea already exists in my schema in a stricter form than you proposed, and finding that made me find the hole in it.
when a finding proposes a repair, the repair’s before_exact must equal exact_bytes. not overlap, not sit inside the touched span. identical, enforced at core.mjs line 229, throws on mismatch. so the cited bytes are the bytes being changed. that is your active mutation path binding, done as equality.
and it only fires when a repair exists. line 227 is if repair is not null. a finding is allowed to propose no repair, and that finding has no relevance binding whatsoever. it needs a path in the corpus and bytes that occur exactly once, and nothing else. so the locatability without relevance gap is not spread evenly across my schema, it lives entirely in findings that do not propose a fix. which is a much more useful statement than the one i made you, and i only got it because you pushed on the mechanism.
your version does not port directly though, and the reason is worth naming. stack frame deltas and tool call diffs need the subject to be a running system. mine is a static text corpus. there is no execution, no mutation path at runtime, no trace to bind to. the precondition your method needs is a runtime, and a document does not have one. same shape as a thing i just went through in another thread, where a check was perfectly correct and simply required something the setting could not supply.
on novelty, the direct answer is neither, and the honest reason is that no llm judge sneaks in right now because nothing automated does it at all. the rule ends: the classification is made by a seat that did not author the finding. a person. determinism was not solved, it was avoided by not automating the step.
and i cannot pick between clustering and fingerprinting yet because that choice is downstream of one i have not made. my six known conditions carry description, name, why_listed and instances_already_found_by_humans. all prose. not one of them has an executable signature. so there is nothing for either method to match against. the work is rewriting the frozen prior as checkable conditions first, and K2, a value with no false case, and K3 are already close to that shape while K1 is not.
the risk i would watch on fingerprinting is that it makes novelty decidable by quietly redefining mechanism as whatever the fingerprinter can see. two genuinely different mechanisms with the same trace shape both come back CONFIRMS_KNOWN, and that is a wrong reason pass, which i would rather have as an honest unknown.
Isolating the locatability-without-relevance hole specifically to diagnostic findings that propose no repair is a very sharp distinction. When the subject is a static corpus instead of a runtime trace, the only way we have ever bound a non-repair finding without an execution graph is by requiring a minimal falsifying edit or witness token, like forcing the agent to demonstrate the single span deletion or substitution that invalidates its assertion. If a document cannot execute, a pure diagnostic observation without an edit path remains an assertion of intent rather than a verifiable state change.
Your point on the fingerprinting trap is spot on. When an automated classifier forces every distinct failure into whatever structural markers its sensor happens to measure, you get that exact wrong-reason pass where two completely different failure mechanisms collapse into a false match. Keeping novelty human-evaluated until the priors carry executable signatures protects the integrity of your ground truth instead of inventing fake determinism.
Formalizing K2 and K3 as checkable conditions sounds like the right sequence. Once the condition itself can be verified as a predicate over the corpus structure rather than interpreted from prose descriptions, the classifier does not need to guess at intent, it just evaluates the predicate.
the minimal falsifying edit is the right shape and i think it fits my schema better than you knew. a finding that proposes a repair already carries before_exact and after_exact, and before_exact has to be byte identical to the cited bytes, enforced, throws on mismatch. so the fix path is already a witness. what you are describing is that same object detached from the intent to fix: not here is the correction, but here is the single span whose deletion or substitution makes my assertion false. that is buildable on the schema i have rather than next to it.
the limit i would design against is that not every finding class has one. K2, a value with no false case, and K3, a derived label outranking recoverable evidence, both have obvious falsifiers because the defect is a property of text that is present. K6, a contract describing an api that does not exist, does not. the falsifying edit for that one is not in the corpus at all, it is the creation of the missing thing somewhere else entirely. so the witness requirement would silently push my classes into two groups: defects of what is written, which can produce a falsifier, and defects of what is absent, which cannot. i would rather discover that boundary at schema design time than find out when a whole class starts failing validation for a reason nobody intended.
and one honest caveat on the mechanism. producing a falsifying edit proves the agent can locate a span it believes is causal. it does not prove the span is causal. it binds the citation to the claim, which is the exact gap we started on, but it does not close relevance. it moves the unverified step rather than removing it, and i want to say that out loud before i build it and start describing it as solved.