There is a specific kind of confidence failure that shows up in agentic AI systems. It is not the dramatic hallucination - the confident fabrication of a person who never existed. It is quieter than that. The model understands the task. It understands the data. And then it decides it has a better idea.
We ran into this repeatedly while building the content health maintenance workflow for Synthadoc, an open-source AI-native wiki engine. The workflow's job sounds modest: scan wiki pages for structural problems like duplicate headings or thin sections, propose targeted fixes, and let the user approve them before applying anything. What we found, and eventually fixed, is a catalogue of ways an agent can go wrong even when the model is capable and the intent is clear - and why the solution is never just a better instruction.
The Problem We Were Solving
Synthadoc ingests raw documents - PDFs, web pages, transcripts, spreadsheets - and uses an LLM to synthesize them into a structured Markdown wiki. Over time, the same topic gets ingested from multiple sources. Without intervention, a single wiki page about Alan Turing can accumulate ten copies of ## Bletchley Park and Cryptanalysis and eight copies of ## Early Life and Education. The page is coherent in isolation but structurally broken: duplicate content, contradictory emphasis, and thin sections that exist only because a heading was created before the body text was written.
The content health scanner addresses this. It is deliberately a pure-Python heuristic with no LLM dependency - it detects duplicate_section, empty_section, thin_section, unclosed_code_block, orphaned_heading, and broken_list_nesting through deterministic regex. This part is reliable. The next part - using an LLM to fix the issues - is where the story gets interesting.
What We Intended the Workflow to Do
The design was explicit. For each flagged page:
- Load the page content and the list of detected issues
- Send both to the LLM with a carefully constructed prompt
- Show the user a side-by-side BEFORE/AFTER diff
- Wait for approval before writing anything
- Snapshot the previous version so a rollback is possible
Every step in this design has a purpose. The BEFORE/AFTER review is a human gate, not a formality. The snapshot is a reversibility guarantee. The heuristic scanner runs before the LLM so the issue list is precise, not speculative.
What actually happened when we first ran it told us exactly which parts of our hardening were missing.
What Went Wrong
Failure 1: The Model Rewrote the Page
The first problem emerged immediately. The LLM received the page content and a list of issues. There were twelve structural problems on the page about the Open Source Movement - three duplicate sections and nine thin sections. The AFTER content we received back bore no structural resemblance to the original. The original heading ## Richard Stallman and GNU (1983) was gone. The heading ## The Cathedral and the Bazaar (1999) was gone. In their place: ## Historical Framing and Origins, ### Research Computing Culture and Principled Advocacy, ### GNU Toolchain and the Missing Kernel. The content was factually reasonable. It was also something the user never asked for and could not safely approve, because we had lost all traceability between the original and the proposed version.
The model had, in effect, written its own wiki page from its pre-trained knowledge, using our wiki page as a rough reference. The word "fix" in our prompt was interpreted by the model as license for rewriting.
Failure 2: The Model Kept the Wrong Duplicate
For a page about computing history, the scanner flagged ## Moore's Law appearing three times. Two occurrences were thin summaries of about forty words each. The third was a detailed paragraph with four footnote citations to the source material. Our initial prompt said "keep the first occurrence." The model faithfully kept the first - one of the forty-word summaries - and discarded the version with citations.
Technically correct. Practically, the wrong choice. We had a rule that was precise but not meaningful for the actual goal.
Failure 3: CLI-Provider Artifacts in the Proposed Content
Synthadoc supports multiple LLM backends. One of them is a CLI-based provider - an agentic tool (like opencode or Claude Code) that is itself an agent with its own tool-calling loop. When we routed the content health workflow through this provider, the response text included shell command output: <|func_arg_name|>command<|func_arg_value|>cat /Users/user1/wikis/alan-turing.md. The model had read the actual file from disk, bypassing our carefully constructed context, and its internal tool invocations leaked into the response text we were about to write to the wiki.
Failure 4: Planning Notes Presented as Final Content
In some runs, the proposed content contained lines like:
### LISP (1958) - actually LISP is more third gen but...
### Introduction paragraph (existing)
Expand to 50+ words
These were the model's internal planning notes - draft scaffolding that it had not cleaned up before terminating its response. These notes would have been written to the wiki without detection. It is possible that a user reviewing a diff in a modal at speed might miss them.
Failure 5: Heading Renamed to "Correct" a Date
After tightening the prompt, one run still returned a subtle violation. The original heading was ## The Cathedral and the Bazaar (1999). The proposed content had ## The Cathedral and the Bazaar - the year dropped. The model "knew" the essay was presented in 1997 and silently corrected what it interpreted as an error in the user's own heading. It was an editorial judgment we had not authorized.
The Failure Mode Map
These failures cluster into two categories that are worth naming precisely.
Category A - DRIFT: Model understands task but applies wrong judgment
Category B - HALLUCINATION: Model substitutes its own knowledge for the task
It is important to distinguish. As a result of drift, the right process is being used to make wrong decisions - "keep the first" instead of "keep the best." As a result of hallucination, the right process is being replaced completely - generating a page from weights instead of fixing the one we hand it. Different mitigations are required for each.
How We Hardened the System and Built the Harness
These fixes fall into two distinct layers. Hardening addresses what the model produces: tighter prompts, output parsing, provider-specific guards. Harness engineering addresses what the model can affect: human gates, rollback, scope exclusions. Hardening reduces the probability of a bad output; the harness limits the consequence of one that gets through. Both are required - hardening alone cannot guarantee zero failures, and a harness without hardening just moves the review burden to the human gate.
Hardening the Agent
1. Artifact Extraction as an Input Boundary
The CLI-provider issue required a structural fix, not a prompt fix. We introduced _extract_md_from_response() - a pre-processing step that strips everything before the first # H1 heading in the response. CLI-provider tool call output always appears before the actual content begins. By treating the first H1 as the clean boundary, we recover the markdown content without the artifact noise.
This is an example of boundary enforcement at the output interface: treat the model's raw response as untrusted input to a parser, not as a value ready to write.
2. Draft Detection as a Post-Output Gate
For planning notes, we added _is_draft() - a regex that catches linguistic markers of incomplete thinking: TODO, FIXME, Expand to, Hmm,, Wait,, I'll, Let me, existing content. If any match, the proposed content is flagged and the page is skipped automatically with a clear explanation in the UI.
_DRAFT_RE = re.compile(
r"^\s*(TODO|FIXME|Expand to\b|Hmm[,.\s]|Wait[,.\s]|I'll\b|I will\b|Let me\b"
r"|existing content|combine existing|add more content|placeholder)",
re.IGNORECASE | re.MULTILINE,
)
This is an output validation gate - a deterministic check applied after the model responds and before the result reaches the user approval screen. The model cannot disable it by rephrasing the note.
3. Surgical Prompt Constraints
The original prompt said: "Fix the structural issues listed below." That sentence, to a language model, is nearly unlimited scope. We replaced it with an explicit constraint hierarchy:
ABSOLUTE RULES - violating any one is a failure:
• PRESERVE every existing section heading EXACTLY - same text, same # level,
including any dates or years in parentheses. Do not 'correct' headings.
• PRESERVE the section ORDER exactly as in the original.
• Do NOT add new top-level or second-level sections.
• Do NOT rename, merge, split, or reorder any sections.
• Do NOT alter factual claims in sections you are not fixing.
PERMITTED CHANGES (only these, only where the issue list directs):
1. Duplicate sections: keep the most complete occurrence (most words and/or
citations); delete the others. Place the kept version where the first
occurrence was. Do NOT merge or rewrite - keep it verbatim.
2. Thin/empty sections: add factual prose UNDER the existing heading.
The heading text must remain unchanged.
Changing from "fix issues" to "permitted changes" is intentional. The model cannot rationalize around a list of what it is allowed to do when permissions are enumerated rather than prohibitions.
4. Issue Count Threshold for Nondeterministic Providers
Even with the tighter prompt, pages with more than six issues on a CLI-provider run produced hallucinated new sections. The CLI provider is itself an agentic tool with its own reasoning loop - it does not behave like a simple completion endpoint, and the compound nondeterminism (our model + its internal model) overwhelms a long issue list.
We added a hard skip at six issues for the CLI path:
# CLI providers (opencode, claude-code) are agentic - they read files,
# plan before writing, and hallucinate new sections when the issue list
# is long. API providers follow the prompt reliably and don't need this
# guard. This path is CLI-only, so the cap applies only there.
_MAX_ISSUES_AUTO_FIX = 6
API-key providers (Anthropic, OpenAI) use a separate code path with no cap, since they are stateless completion endpoints that see the prompt exactly as we construct it. Rather than a model capability, this is a property of the architecture.
Building the Harness
5. System Page Exclusion
Thin sections on pages named index and dashboard were correctly identified by the scanner. These are navigation pages, not content pages - thin sections are structurally appropriate. The workflow would propose expanding them without exclusion, which the user would reject correctly, wasting their time.
We use the existing SYSTEM_PAGE_SLUGS constant to exclude them from the fix loop. No duplicate definition, no magic string. The scanner still runs on them - the issues appear in lint reports - but the auto-fix workflow skips them.
6. The Human Gate as a Structural Requirement
Every proposed fix must pass through a BEFORE/AFTER diff review before anything is written. This is not a feature - it is the architectural guarantee that no automated process can corrupt wiki content without explicit consent. The confirm card shows:
- The page slug and issue summary
- BEFORE (first 2000 chars of the original)
- AFTER (first 2000 chars of the proposed content)
- "Approve" / "Skip this page" buttons with a 5-minute timeout (auto-decline)
The scrollability of code blocks in the review modal was a concrete UX issue we had to address - users need to be able to read what they are approving. Gates that are theoretically present but practically unusable are not gates.
7. Snapshot and Rollback as a Safety Net
Before any approved fix is written, snapshot_if_changed() takes a versioned copy of the original content. If the approved result turns out to be wrong, the user can request a rollback by name: "roll back alan-turing." The workflow restores the snapshot, re-runs the scanner, and updates the issue list to reflect the restored state. It is not a rollback to recover from system failure - it is a first-class workflow feature that acknowledges the model may produce something subtly wrong that was not detected during review.
The Broader Principle: Both Layers Are Architectural, Not Textual
The pattern we followed across all of these fixes is consistent: every control - whether it hardens the agent's output or constrains the harness around it - is implemented in code, not in prose.
"Do not hallucinate" in a system prompt is a behavioral suggestion. The model considers it, weighs it against other pressures, and sometimes overrides it. That is not a flaw, it is how language models work. The token prediction process does not have a hard stop for "this would be a hallucination."
Both hardening and harness engineering mean building controls that are not subject to the model's interpretation:
| Layer | Control | What it does | Why it cannot be bypassed |
|---|---|---|---|
| Hardening | _extract_md_from_response() |
Strips pre-H1 content | Runs on raw response bytes, before any model interpretation |
| Hardening |
_is_draft() regex |
Blocks planning-note content | Deterministic regex match - not in the model's loop |
| Hardening | PERMITTED CHANGES prompt | Enumerates allowed scope | Narrows the action space; drift must contradict the list |
| Hardening | Issue count cap | Skips complex pages | Hard code gate; model never sees the page |
| Harness | System page exclusion | Skips non-content pages | Slug lookup against a constant; not in the prompt |
| Harness | BEFORE/AFTER human gate | Requires user approval | No write path bypasses this |
| Harness | Snapshot / rollback | Preserves prior state | Written before the fix is applied |
These controls form layers. A fix that evades the prompt constraints is caught by the draft detector or the human gate. A fix that passes the human gate can be rolled back. No single layer is sufficient. The layers together are.
Why This Matters Beyond Wiki Maintenance
Incorrectly updating a wiki page is a recoverable mistake. A user clicks rollback, the page is restored, and no lasting harm is done. But the failure modes we observed - a model substituting its own knowledge for the task, a model including its internal tool calls in the output, a model silently correcting data it considered wrong - are not domain-specific. The same failure patterns appear in:
- Agentic coding tools that refactor code the user did not ask to touch
- Document automation systems that "improve" a legal clause beyond the approved scope
- Data pipelines where an LLM normalizes values using world knowledge rather than the provided schema
- Financial report generators that extrapolate figures not present in the source data
In each case, the consequence of unhardened agentic behavior scales with the stakes of the domain. A hallucinated section in a personal wiki is recoverable. A hallucinated figure in a financial disclosure is not.
The argument for hardening is not that models are bad. It is that models are nondeterministic, and production systems require deterministic guarantees at the boundaries where AI output meets persistent state.
What to Apply From This
If you are building an agentic system that reads and modifies persistent data, consider these as minimum requirements:
Boundary enforcement. Treat the model's output as untrusted input at every handoff. Parse it. Validate it. Strip what does not belong before it reaches the next step.
Deterministic post-output gates. Add validation logic that runs after the model responds and is not subject to the model's behavior. Draft detection, schema validation, output format checks. These are deterministic; the model cannot reason around them.
Enumerated permissions over enumerated prohibitions. In prompts, tell the model what it is allowed to do, not only what it must avoid. The scope of permitted actions is easier to reason about than an open-ended task with a list of exceptions.
Acknowledge provider heterogeneity. Not all "model" configurations behave the same way. A CLI-based agentic provider and a stateless API completion endpoint have fundamentally different behavioral properties. Design your hardening layer to account for both, or constrain which providers can reach which actions.
Human gates as structural requirements. For any workflow that modifies persistent state, a human review step is not a UX nicety - it is a control. Make it usable. If the review UI cannot display the full diff, the gate is not functioning as intended.
Reversibility as a first-class design constraint. Snapshots and rollback should be designed in from the start, not added after the first production incident. Assume the model will sometimes produce a subtly wrong result that passes review, and design the recovery path accordingly.
Closing
It took multiple iterations for Synthadoc's content health maintenance workflow to behave predictably. Rather than rephrasing the prompt, each iteration addressed a specific failure mode. As a result, the team trusts the workflow - not because the model is perfect, but because the system is designed to catch and contain errors.
Synthadoc is open source. The content health scanner, the resolver workflow, and the draft detection logic are all available at Synthadoc git repo. If you are building AI-native systems that write back to persistent state, the patterns described here are worth examining in the context of real code, not just as abstractions.
The nondeterministic machine can be harnessed. The harness is engineering, not prompting.
Top comments (9)
The "it decides it has a better idea" framing is spot on — that's the failure that burned us the most, because it never trips an error. The model does something reasonable, just not the thing you asked for, so the output looks fine until you check traceability.
The pattern we landed on matches yours: constrain the edit surface, not the prose. For your rewrite-the-whole-page case, we had luck forcing the model to emit a list of targeted operations ("merge sections X and Y", "delete empty heading Z") against stable anchors, then applying them deterministically in code. The LLM proposes, Python disposes. It can't silently reorganize what it isn't allowed to touch, and your BEFORE/AFTER diff stays meaningful because the anchors never move.
Two questions: did you find a measurable quality hit from constraining the model to operations instead of free rewrites? And does your snapshot/rollback kick in automatically on a failed human gate, or is rollback a separate manual step?
Hi Max, thanks for your comments. The "LLM proposes, Python disposes" framing is exactly right, and the anchored-operations approach is the cleaner architecture for structural edits. We landed somewhere adjacent but not identical, the model still does free rewrites, but the harness constrains the surface it can touch (slug-guarded writes, must call get_page_content before write, diff shown before commit). The LLM can still reorganize prose inside a section; it just can't invent new pages or write without reading first.
On your quality question: yes, there's a ceiling. For structural issues - merge duplicated sections, delete empty headings - targeted ops are strictly better. For thin sections that genuinely need new prose, you can't express "add substantive content about X" as a deterministic operation against an anchor. We hit that wall with the content-health resolver; structural issues respond well to constraints, content gaps need latitude. The hybrid I'd reach for next time: ops for structure, free prose for content, different model calls for each.
On rollback: the snapshot is taken before the approved write, so a declined gate means nothing was written, no rollback needed. If the user approves, sees the result, and regrets it, rollback is a separate manual step (roll back in chat or CLI). Auto-rollback on gate failure felt like it would cause more confusion than it solved - better to just never write without a yes.
One thing I'd add to the harness list is reproducing the bad run itself. A failure like the full-page rewrite usually doesn't happen every time, so after a prompt fix it's hard to know whether you fixed it or just didn't hit it this time.
What helped me: record the exact request (rendered prompt, tool results, sampling settings) and replay it on an engine that gives the same output for the same request. Then you can look at where the rewrite starts. Often there's one early token where "keep the heading" and "rename the section" were nearly tied, and the rest follows from that choice. You can also re-run that one request with the changed instruction and see whether the same point now goes the other way, instead of sampling a few runs and hoping.
(The engine I use for this is one I wrote, GeulPU: it replays recorded runs bit for bit and can fork a run at a chosen token. github.com/wwoosshh/GeulPUv2)
The question of "did I fix it or just didn't hit it" is an important one, and it is exacerbated when the failure rate is on the order of one in ten runs. The majority of prompt debugging occurs in this probabilistic fog.
The idea of forking about a token is particularly interesting. Instead of asking "does this prompt produce the right output on average," you're asking "at the specific choice point where things went wrong, does the new instruction shift the probability mass?" That's a much more thorough test.
Basically, deterministic replay is required, which means either controlling sampling (temperature=0 still varies across frameworks) or recording at the logit level as GeulPU does. The recording of the full rendered prompt and tool results is also not trivial if your harness does not already expose that - in Synthadoc, we log the raw messages array, but not the sampling parameters, so I could replay the content, but could not guarantee the same token distribution.
We'll look at GeulPU - the fork-at-token primitive seems hard to build and undervalued until you need it.
Thanks Paul. Agreed that the fork is the hard part to build, and it only comes almost for free once replay is exact.
A cheap step that works even without changing engines: log the sampling settings next to the messages array (model id and version, temperature, top_p, seed, max_tokens, stop), and ask for logprobs wherever the provider returns them. With the chosen token's logprob and the runner-up's, you can find the low-margin steps of a bad run, the near-ties where "keep the heading" and "rename the section" were close, even if you can't replay them exactly. Temperature 0 alone isn't enough: on most stacks batching, the prompt cache and kernel choice can still flip a near-tie.
For exact replay you need a deterministic engine underneath. GeulPU is one for local GGUF models: it records each request with its token ids and per-step logit hashes, replays it bit for bit, and branches from any token or from an edited request, so the only difference you see comes from the instruction you changed. The debugger guide walks through it: github.com/wwoosshh/GeulPUv2/blob/...
Failures 1, 2 and 5 share a property: your scanner is deterministic, so the expected shape of the AFTER page is already known before the model runs. Headings in AFTER should equal headings in BEFORE minus the duplicates the scanner flagged, in the same order, with the header text byte-identical. That is a plain set-and-sequence comparison, and it rejects the rewritten Open Source page and the "(1999)" dropped from the Cathedral heading without any prompt wording or regex for what the model might say. A proposal that fails it never reaches the human diff.
Failure 2 I would take away from the model altogether: choose which duplicate survives in code (most citations, then longest) and let the LLM only merge the text. "Keep the first" and "keep the best" are both rules the model will follow literally, so the selection is safer as a measured property than as an instruction.
Do your logs show how often the structural check would have fired per run, as opposed to the draft detector? I would guess the heading check is the one that fires most.
Thanks, Arhan. These are the kinds of observations that directly address the structural issue.
Regarding the heading invariant check: you are correct, and we should have built this first. Failures 1, 2, and 5 have a fully computable post-condition: headings(AFTER) == headings(BEFORE) − flagged_duplicates, in the same order, byte-identical. The output of a model that fails that check is structurally corrupt, regardless of how well it reads. The "Open Source" rewrite passed our diff-based review heuristic since the text looked reasonable - the heading count check would have rejected it immediately. There is no heuristic judgment involved in this 10-line post-output validation.
The principle of removing survivor selection from the model is sound, but the implementation boundary matters in our case. Our contradiction resolver usually resolves conflicts based on page-versus-raw-source (a new PDF, URL, or scraped article contradicts an existing wiki page), not page-versus-page. As raw sources generally lack any citations and are long, the rule "most citations, then longest" will almost always select the existing wiki page as the survivor - which is not always appropriate when the source is newer and more authoritative. The heuristic does apply cleanly in the case of cross-page duplicates (two wiki pages covering the same topic), and we would apply it there rather than to all types of conflict. The underlying split - code selects, LLM only synthesizes - is the correct one; we only need to pay more attention to which scoring function fits which type of conflict.
We can say with certainty that the heading check would have caught failure 5 - the "(1999)" drop is a literal heading text mutation that a byte-identical comparison rejects outright. For failure 1, it depends on whether the Open Source rewrite changed any headings or only the content within sections; if it was a content-only rewrite, the heading check would have missed it. For failure 2 it almost certainly would not have fired - that is a survivor-selection error, not a structural mutation. As for which check fires most often at scale, that is genuinely a guess; we do not have aggregate data yet, which is exactly why instrumenting the two signal sources separately is worth doing.
the keep the first occurrence failure is the one that stuck with me. a rule that is precise but not tied to the goal gets followed perfectly and still picks the forty word summary over the version with four citations.
we moved that kind of decision out of the prompt and into a scoring step, so the model chooses between candidates ranked by citation count and length instead of position. the model only does the merge, which also shrinks the rewrite problem from failure one a lot.
have you tried constraining the output to a patch format so a rewritten heading simply fails to apply?
The fix we shipped here was not scoring - it was merging, and that's a meaningfully different fix from what was described in the earlier reply.
The duplicate-section fix runs entirely in Python, no LLM call: it keeps the position of the first occurrence, but appends every paragraph from later copies that isn't an exact duplicate. We don't discard anything to pick a "winner," so the failure you describe - forty-word summary beats four citations - can't happen anymore, not because we scored the candidates correctly, but because we stopped selecting a survivor. Regardless of which copy had the catchier opening paragraph, the citations are appended to the kept section.
As you pointed out, this path completely shrinks the rewrite problem: Since Python does the entire merge, the model never gets caught in the loop. However, that is only applicable to duplicate headings on the same page. As of now, we don't have an equivalent for page-to-page duplicates or page-to-source duplicates, where two documents may differ in a number of ways rather than just in detail, and naive paragraph-union cannot handle that, since it would merge content that may conflict. For that case specifically, scoring (citation count, recency, length) is the most likely direction, not appending everything.
The patch-format question: no, we haven't tried it, and we want to gently push back on the framing. A unified diff will not make a heading mutation "fail to apply" on its own - patch application is just context-line matching; it has no notion of which lines are off-limits. If the surrounding text matches, a hunk that drops a year from a heading applies cleanly. Instead of an all-or-nothing rewrite, we could run the heading check per-edit and reject only the bad ones. In general, we prefer search/replace blocks (exact text match, not line-numbered diff syntax) over a literal patch format, since models are unreliable at generating correct line counts and a bad header fails for no apparent reason.