DEV Community

Rain S
Rain S

Posted on

AI Audit Evidence, Part 5: Five Sources in One File Do Not Check Each Other

Has the same conclusion been answered by two paths that share nothing?

Our evidence package puts five sources into one file: the authorization snapshot, the decision evidence, the audit row, the side-effect row and the operation audit row. A root digest pins them together. The package verifies offline and the signature matches.

What that proves is that the five were signed together. It does not prove that they said anything to each other. A digest only answers whether the group was swapped out.

One sentence: cross verification is not "find more sources". It is one conclusion answered independently by two non-sharing paths, with the system itself able to notice when they disagree.

If that is enough, the rest is a table, a code fragment and three admissions.

  1. What counts, and what is only packaging Three criteria. Miss one and it does not count.

The two paths do not share a source. Not two views of the same data, and not the same code run twice.
A disagreement is detectable. Code compares them and returns a distinguishable conclusion. Two values placed side by side, waiting for the reader to spot the conflict, does not count.
It happens at runtime. A comparison that only exists in a test does not count, and neither does one a reader performs by hand.
By those criteria, bundling five sources and hashing the bundle is packaging. Its value is real: the bundle cannot be altered without breaking the hash. But "cannot be altered" answers "was it changed", not "do the parts corroborate each other". Those two questions get conflated because both travel under the word integrity.

  1. Our own account: two and a half This write was authorized. The second path recomputes from the policy version recorded in the row. What it compares is a single item, whether the tool was enabled. Role, writable field domain, destination and approval requirement are not recomputed. This write actually happened. The ledger, the business table and the audit row each carry a view. They are displayed side by side and never judged against each other. The revocation actually completed. The second path compares what this call declared against what the ledger holds, in both directions. This is the only place where a disagreement is detectable. The record was not altered. No second path in this deployment. The external anchor exists as a mechanism; this deployment has no receiver for it, so it is off rather than absent. The coverage statement holds. No second path. The conclusion is derived from one set of facts. Out of five conclusions, one meets all three criteria.

The authorization row is half. A recomputation does happen, but for one of five checks. A reader sees "authorization was reviewed" and what was reviewed is a single item.

Read that list as three grades: complete, meaning a second source that is actually compared; half, meaning a second source with one item compared; and none, meaning no second source at all.

There is a fourth grade that is not on the list, and it costs the most: both sources are present and nobody compares them.

We run two chains, an AI audit chain and an operation audit chain, and the same request lands in both. The material is there, the write points do not overlap, and it looks exactly like two independent paths. But no line of code pairs them by action. They are juxtaposed rather than corroborating.

It is not on the list of five because we never treated "the two chains describe the same event" as a conclusion the package has to answer. The five are what the package claims. This one nobody claimed. The expensive kind of false cross is not missing material. It is material that nobody remembers to compare.

  1. Why the one that counts, counts declared targets vs targets the ledger holds only declared → onlyDeclared only held → onlyStored either non-empty ⇒ mark the group disputed, refuse to report the revocation complete Independence: one side is what this call said it would write, the other is what the ledger already recorded. They are produced in different places, so an error in one does not automatically appear in the other.

Detectability: a mismatch becomes a state, disputed, with the difference kept as evidence. Not two fields waiting to be noticed.

The cost, stated plainly: it triggers in one situation, when a retry with the same arguments hits the idempotency key.

  1. Something a reader asked out of us This section is about us.

We published a five-layer evidence model: that a record exists, its source and integrity, its link to execution, coverage, and the verification of the result. A reader replied that a layer was missing. Some claims have no record at all. "This system complies with X", with nothing attached.

He was right, and it was worse than he put it. Our first layer says that a record exists is not the same as the event happened, and it presupposes that a record exists. A claim with nothing attached sits outside the model: it neither satisfies nor violates it. It cannot enter.

We added the layer, numbered L0. To be explicit: it was added after publication, and the published version does not contain it. If you go back to the article, you will not find it there.

That exchange was itself a cross check. Our own conclusion was asked again by someone who does not share our view, in a different form, and the different form found the gap. The same question in the same wording returns the same blind spot.

  1. Four checks you can run What is the second path for this conclusion? If you cannot name one, it is single source. Label it that way instead of calling it corroborated. Are the two paths independent, or is it the same data dressed differently? Bundling, merging and digesting all fall on the second side. When they disagree, does the system return a state, or does it place two values next to each other? Does that comparison run in production? If only in a test, it proves that test rather than the steady state. Closing Back to the file. It can prove it was not altered. It cannot prove the parts corroborate each other.

To prove that, each conclusion has to name its second source, and the ones that cannot get labelled single source. The list will look bad. It will at least be accurate.

The mechanism above is taken from KeelBase, an open-source runtime that sits between an AI agent and a business system, re-checks identity, authorization, scope, risk and confirmation state before a tool runs, and writes every decision into a record that can be verified offline. Cross verification is a way of reading here, not a shipped feature; KeelBase is its implementation and evidence carrier.

Repository: https://github.com/rain6fish/KeelBase

Three things stated plainly. One of five conclusions is a complete cross check; the rest are half or nothing. The authorization row reviews a single item out of five. And L0 was added after publication, so anyone citing it should carry that sentence along.

Drafted with an AI assistant. The system, the positions and the mistakes are mine.

Top comments (5)

Collapse
 
reidmarlow profile image
Reid Marlow •

Most teams confuse an audit trail with an audit check. Bundling an agent's declared tool arguments next to the database receipt in one signed payload looks like verification, but if no code diffs them before commit, you only bought post-incident forensics. The moment you make that mismatch a blocking state, you also discover how many retries trigger false disputes unless the idempotency key binds the exact argument hash.

Collapse
 
rain6fish profile image
Rain S •

You are right that a trail is not a check, and right about where the value sits: a signed bundle nobody diffs is forensics.

One correction on timing, because it decides which side of commit this falls on. The diff runs on the write path, not at read time. When a retry hits the idempotency key, the ledger compares what this call declares it will write against what the group already holds, before replaying the existing row. So it is a pre-commit check, but only on the path where a second observation exists. A first write has nothing to compare against, and I would rather say that plainly than imply the check runs everywhere.

On the false disputes: the key binds the exact arguments, and more than that, the canonicalised ones. It is sha256 over user, conversation, tool and the sorted-args JSON, so identical arguments always land on the same key and different arguments never do. The false disputes we actually get come from somewhere else, tools that declare different targets for identical arguments, generated ids being the usual case. That is why the mismatch does not block the write. It blocks the claim that revocation completed, and keeps the difference as evidence.

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

Bundling five sources and hashing them together proves they traveled together, nothing more. What counts as a second path in your setup, a different source or a different code path?

Collapse
 
rain6fish profile image
Rain S •

Different source first, and then a test that settles it. Two answers count as separate paths when I can construct a fault that moves one and not the other. If every way I can break the system moves both together, they are one path wearing two names, and their agreement means nothing.

Source difference is necessary, since two views of the same row can never disagree. A different code path usually follows, but it is not sufficient: two paths can read the same table and fail together.

In our code the one case that passes is the compensation diff. One side is what the call declared it will write, the other is what the ledger already holds. The fault that separates them is a non-deterministic tool that declares different targets for identical arguments, and it moves only the declared side. A bug in the writer feeding both would move them together, which is the limit of that check.

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

That fault test is the part I will keep. A break that moves both sides together means the agreement was never a check.