DEV Community

Jordan Huang
Jordan Huang

Posted on

Your Agent Said Pass. Show Me the SHA256.

#ai #agents #testing #git

Did your agent actually finish the work? Or did it only narrate a convincing win? Review threads keep mixing those two events.

A spoken pass feels complete. A hashed file does not feel magical. Which one would you trust during an incident?

This FAQ attacks claims developers repeat. Each entry has a claim, weak evidence, and a corrected model. No beauty contest. No unnamed latency charts.

What this piece refuses to do

This is not another green-chat sermon. You have read that slogan already. The gap is operational, not rhetorical.

I will not invent model names. I will not invent quotas. I will not invent hardware. Availability is not a benchmark.

Myth: "I watched pytest scroll by, so tests ran"

The claim. If tokens look like pytest, pytest ran. The live stream is the proof.

What people paste.

  • A green-looking summary paragraph
  • A copied traceback with no file
  • A cheerful "all 41 tests passed"

What actually holds. A stream is ephemeral theater. You cannot hash a vanished scrollbar. You cannot re-run a memory.

Did the process write junit.xml? Did sha256sum see that file? If both answers are no, you have a story.

Corrected model. Proof is a file with a digest. Narration is a courtesy, not a receipt.

Myth: "It quoted my filename, so it read the file"

The claim. Path mention equals file access. Citation equals comprehension. Sound familiar?

What people paste.

  • A function name from memory
  • A line-shaped guess
  • A confident in src/app.py clause

What actually holds. Models can recite paths from prompts. They can invent nearby names. Quoting is cheap. Opening is not.

Ask for a content hash of the cited file. Compare it to sha256sum on disk. Mismatch? The citation was fan fiction.

sha256sum src/app.py
# require the agent to write the same digest into receipt.json
Enter fullscreen mode Exit fullscreen mode

Corrected model. Reading is an I/O event. Talking about a path is not I/O.

Myth: "Throwaway box, so skip the receipt"

The claim. A free server is disposable. Disposable means no artifacts. Why keep XML on a box you will burn?

What actually holds. Disposable compute still produces durable lies. A bad patch can leave your laptop. The box dies. The commit may not.

Would you merge because a hotel TV looked green? Same energy. Same missing hash.

A throwaway host is still useful for rehearsal. MonkeyCode offers free model access and a free server option, which fits that rehearsal role. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free compute does not waive evidence. Demand the same hashed receipt there.

Corrected model. Burn the machine. Keep the digest. The receipt outlives the host.

Myth: "The transcript is my audit log"

The claim. Save the chat. That is provenance. Someone can read the vibes later.

What actually holds. Transcripts omit tool failures. They reorder steps. They summarize stderr away. They cannot be replayed as a build.

Can you point git at a chat bubble? Can make consume a paragraph? Then it is not an audit log.

# a transcript cannot answer these
git rev-parse HEAD
git status --porcelain
test -f junit.xml && echo "report exists"
Enter fullscreen mode Exit fullscreen mode

Corrected model. Audit logs are append-only, timestamped, and parseable. Chat is neither a compiler nor a ledger.

Myth: "I'll paste the summary into the PR body"

The claim. The model's recap documents the change. Reviewers need prose, not XML.

What actually holds. Prose helps humans. Prose does not pin bytes. Reviewers cannot regenerate a recap into a test graph.

Put the digest in the PR. Link the report. Keep the story if you want. Do not swap them.

Corrected model. Narrative is a comment. The artifact is the contract.

The artifact: a receipt you can hash

This recipe is a proposed drill. It is not a production SLA. Run it on a tiny repo first.

Create a disposable project:

mkdir /tmp/receipt-drill && cd /tmp/receipt-drill
git init -q
python3 -m venv .venv
. .venv/bin/activate
pip install -q pytest
Enter fullscreen mode Exit fullscreen mode

Drop a test that can fail on purpose:

# test_math.py
def add(a, b):
    return a + b

def test_add_happy_path():
    assert add(2, 3) == 5

def test_add_agent_must_not_skip():
    # Flip this to False to watch the receipt fail closed.
    assert True
Enter fullscreen mode Exit fullscreen mode

Write receipt.py. Label it as a drill script, not magic:

#!/usr/bin/env python3
"""Proposed receipt drill. Unexecuted until you run it."""
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path

ROOT = Path.cwd()
REPORT = ROOT / "junit.xml"
RECEIPT = ROOT / "receipt.json"


def sha256(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def git(*args: str) -> str:
    out = subprocess.check_output(["git", *args], cwd=ROOT)
    return out.decode().strip()


def run_pytest() -> int:
    if REPORT.exists():
        REPORT.unlink()
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", "--junitxml", str(REPORT)],
        cwd=ROOT,
    )
    return proc.returncode


def main() -> int:
    code = run_pytest()
    porcelain = git("status", "--porcelain")
    head = git("rev-parse", "HEAD") if Path(".git").exists() else "NO_GIT"

    if not REPORT.exists():
        print("FAIL: no junit.xml. The stream is not a report.")
        return 2

    payload = {
        "generated_at": datetime.now(timezone.utc).isoformat(),
        "git_head": head,
        "git_porcelain": porcelain,
        "pytest_exit": code,
        "junit_sha256": sha256(REPORT),
        "junit_bytes": REPORT.stat().st_size,
    }
    RECEIPT.write_text(json.dumps(payload, indent=2) + "\n")

    if porcelain:
        print("FAIL: working tree dirty. Chat cannot hide this.")
        print(porcelain)
        return 3
    if code != 0:
        print("FAIL: pytest exit was not zero. Ignore the pep talk.")
        return code

    print("PASS: hashed receipt written.")
    print(f"HEAD {head}")
    print(f"JUNIT {payload['junit_sha256']}")
    return 0


if __name__ == "__main__":
    sys.exit(main())
Enter fullscreen mode Exit fullscreen mode

Run the drill:

python3 receipt.py; echo exit:$?
sha256sum junit.xml receipt.json
cat receipt.json
Enter fullscreen mode Exit fullscreen mode

Now flip the second test to assert False. Run it again. Does a spoken apology still tempt you? The new SHA should change. The exit code should change. If either stays still, your receipt is fake.

Decision table: claim versus proof

Claim you heard Common "proof" Require instead
Tests passed Chat summary junit.xml plus SHA-256
File was read Path quoted in prose Digest matching disk
Patch is clean "LGTM from the model" Empty git status --porcelain
Build is reproducible Recalled command in chat Script in repo that exits 0
Server was throwaway "I deleted the box" Receipt committed or uploaded
Review is done PR body recap Digest pasted, report linked

Print that table next to your agent window. Ask one rude question. Where is the file?

Commands that belong in the loop

Do not debate the model. Interrogate the tree.

# 1. What changed for real?
git status --porcelain
git diff --stat

# 2. Did a report land on disk?
ls -l junit.xml

# 3. Can another machine verify the bytes?
sha256sum junit.xml receipt.json

# 4. Would CI accept this working tree?
test -z "$(git status --porcelain)" && echo clean || echo dirty
Enter fullscreen mode Exit fullscreen mode

Want a one-liner gate before git commit?

python3 receipt.py && git add -A && git commit -m "$(jq -r .junit_sha256 receipt.json)"
Enter fullscreen mode Exit fullscreen mode

That commit message is ugly on purpose. It is a hash, not a vibe.

A 20-minute walkthrough, labeled as a drill

  1. Branch from main. Do not work on main.
  2. Give the agent one bounded task. One test file is enough.
  3. Refuse merged output that exists only in chat.
  4. Demand junit.xml and receipt.json on disk.
  5. Run receipt.py yourself. Do not delegate the hash.
  6. If porcelain is dirty, stop. Do not negotiate.
  7. Only then read the prose summary, as a comment.

Why run the hash yourself? Because the agent has a conflict of interest. It wants the turn to end. You want bytes that survive the turn.

Limitations, said plainly

A hashed report does not prove tests were meaningful. Garbage assertions hash too. You still need review of the tests.

This script does not replace CI. It is a preflight. Remote runners can still disagree with your laptop.

Free model access can change. A free server option can change. Do not pin a release process to a courtesy tier.

The receipt does not attest the model identity. It attests files you kept. Those are different facts.

JSON in receipt.json is not a supply-chain SBOM. Do not pretend it is.

Who should not use this approach

Skip this drill if you handle production secrets on a shared box. A free server is a stranger's laptop. Do not paste credentials into it.

Skip it if you need contractual uptime. Courtesy compute is not an SLA.

Skip it if your org already has mandatory CI attestations. Do not invent a second source of truth.

Skip it if you will not run the script yourself. A receipt you never execute is fan fiction.

Corrected mental model, one page

  • Tokens are not processes.
  • Paths are not reads.
  • Summaries are not reports.
  • Hosts are not evidence.
  • Digests are evidence.

Ask it in one breath. Can I hash it, store it, and re-run it? If no, the agent did not prove the claim.

If you run the receipt drill, tell me which myth cracked first. I want the file, not the pep talk.

Top comments (0)