Your Docs Gate Has Never Failed, So It Proves Nothing: Mutation-Testing a Model-Drafted Page

A documentation gate that has only ever printed pass is not evidence of quality; it is evidence that nobody has tried to break it. Before you let a free model draft a page into a review queue, spend twenty minutes proving that your checks actually reject bad drafts. This post builds a small gate, then a mutation harness that injects known defects and asserts the gate fails on each one. The conclusion up front: the mutation sheet, not the gate, is the artifact that tells you which parts of the page a human still has to own.

1. The failure mode this fixes

Most AI-drafting pipelines fail in the same quiet way: the checker looks for things the model already produces. If your gate only verifies that headings exist and links resolve, then a draft that invents a version number and deletes a data-loss warning will pass with a green check. The reviewer then reads a green check instead of reading the section, and the deletion ships.

Mutation testing gives you a cheap way to measure that blindness. You take a draft that legitimately passes, introduce one defect at a time, and require the gate to exit non-zero for every single mutation. Any mutation that survives marks a rule you believed in but never implemented.

2. Decide ownership before any drafting happens

Write the ownership split down once, then encode the machine-checkable half of it. The table below is the split I keep coming back to, because each row on the right is something a model writes fluently and gets wrong silently.

Two rules fall out of this table and they drive the whole pipeline. First, anything in the right column must be physically present in the draft as a marked block, so deletion is detectable. Second, anything in the left column must trace back to a file that runs, so invention is detectable.

3. Build the gate: five rules, five exit conditions

The gate reads one draft page and one facts file, and it does not care which model wrote the prose. Keep it dependency-free so it runs anywhere a free server can run Python.

#!/usr/bin/env python3
# tools/check_docs.py
import json, re, sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
PAGE = ROOT / "docs/draft.md"
FACTS = json.loads((ROOT / "docs/facts.json").read_text(encoding="utf-8"))

FENCE = re.compile(r"```

[a-zA-Z0-9]*\n(.*?)

```", re.S)
COMMAND = re.compile(r"^\$ (.+)$", re.M)
VERSION = re.compile(r"\b\d+\.\d+(?:\.\d+)?\b")
HUMAN_BLOCK = re.compile(r"<!-- owner: human -->\n(.*?)\n<!-- /owner -->", re.S)

def check(text: str) -> list[str]:
    errors = []
    blocks = FENCE.findall(text)
    if not blocks:
        errors.append("D1: no fenced blocks; page cannot be verified")
    for block in blocks:
        for cmd in COMMAND.findall(block):
            if cmd.strip() not in FACTS["tested_commands"]:
                errors.append(f"D2: untested command: {cmd.strip()}")
    for version in sorted(set(VERSION.findall(text))):
        if version not in FACTS["versions"]:
            errors.append(f"D3: unregistered version: {version}")
    humans = [b.strip() for b in HUMAN_BLOCK.findall(text)]
    if not humans:
        errors.append("D4: no human-owned block found")
    if any(not b for b in humans):
        errors.append("D4: empty human-owned block")
    for phrase in FACTS["required_human_phrases"]:
        if phrase not in text:
            errors.append(f"D5: missing caveat: {phrase[:40]}")
    return errors

if __name__ == "__main__":
    errs = check(PAGE.read_text(encoding="utf-8"))
    print("\n".join(errs) or "pass")
    sys.exit(1 if errs else 0)

The matching facts file is small on purpose, because every entry is a promise you have to maintain:

{
  "tested_commands": ["pytest -q", "python tools/check_docs.py"],
  "versions": ["3.11", "20.11.1"],
  "required_human_phrases": [
    "Do not run this against production data.",
    "Backups are kept for 30 days."
  ]
}

4. Mark the human-owned region in the draft itself

Put the ownership marker in the page rather than in a side manifest, so deletion shows up in the same diff a reviewer reads. The marker is an HTML comment, which renders as nothing on DEV or GitHub but is trivially greppable in CI.

<!-- owner: human -->
Do not run this against production data.
Backups are kept for 30 days.
<!-- /owner -->

Give the model exactly this instruction: you may write around the block, you may not edit inside it, and if the block looks misplaced you must stop and report it. That single constraint removes the most expensive review failure, which is a model politely rewriting a warning into something friendlier and less true.

5. Number the steps and keep the draft in a fixed shape

1. Write docs/facts.json from the test suite, not from memory.
2. Write the human-owned blocks yourself, before the prompt, in the empty draft.
3. Ask the model for prose only, with the block contract above.
4. Run python tools/check_docs.py and fix commands rather than prose.
5. Run the mutation harness and require every line to print caught.
6. Review the diff, then publish; never publish from the draft directory.

6. The mutation harness is the real test

The harness copies the draft, applies one textual defect, and asserts a non-zero exit. It restores the file in a finally block, because a crashed run must not leave a mutated page behind for the next person.

#!/usr/bin/env python3
# tools/mutate_docs.py
import subprocess, sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
DRAFT = ROOT / "docs/draft.md"
GATE = ROOT / "tools/check_docs.py"

MUTATIONS = {
    "M1_untested_command": lambda t: t.replace("$ pytest -q", "$ pytest -q --all", 1),
    "M2_version_swap": lambda t: t.replace("Python 3.11", "Python 3.9", 1),
    "M3_empty_human": lambda t: t.replace(
        "Do not run this against production data.", "", 1),
    "M4_drop_caveat": lambda t: t.replace("Backups are kept for 30 days.", "", 1),
    "M5_placeholder": lambda t: t.replace("$ pytest -q", "$ <your test command>", 1),
}

def gate_rc() -> int:
    return subprocess.run([sys.executable, str(GATE)], capture_output=True).returncode

def main() -> int:
    original = DRAFT.read_text(encoding="utf-8")
    survivors = []
    try:
        if gate_rc() != 0:
            print("M0_baseline=fail — fix the draft before mutating")
            return 2
        print("M0_baseline=pass")
        for name, mutate in MUTATIONS.items():
            DRAFT.write_text(mutate(original), encoding="utf-8")
            caught = gate_rc() != 0
            print(f"{name}={'caught' if caught else 'SURVIVED'}")
            if not caught:
                survivors.append(name)
    finally:
        DRAFT.write_text(original, encoding="utf-8")
    if survivors:
        print(f"survivors: {', '.join(survivors)}")
    return 1 if survivors else 0

if __name__ == "__main__":
    sys.exit(main())

The baseline line matters more than the rest. If M0_baseline fails, every later caught is meaningless noise, so treat a red baseline as a stop condition rather than a warning. Run the harness on every change to the gate, and treat a surviving mutation as a bug report against the rules, not against the draft.

7. What the harness still cannot see

Surviving mutations are honest, but passing mutations are not proof of a safe page. Five defect classes have no mechanical check in this design: whether a warning is strong enough for the risk, whether a deprecation date is politically acceptable, whether migration ordering is correct for real upgrade paths, whether the tone suits an incident postmortem, and whether a caveat was omitted entirely rather than deleted.

Keep those five on a written human checklist attached to the pull request. The gate's job is to make sure the reviewer spends attention on judgement calls instead of on verifying that pytest -q still exists.

8. Limitations and who should skip this

This pipeline assumes you can express facts as files, and that assumption fails in several common situations. Teams without a runnable source of truth will spend more time maintaining facts.json than reviewing pages. Pages dominated by narrative, positioning, or opinion gain nothing from command and version rules. Documentation under regulatory or contractual review often needs per-sentence citation, which this gate does not provide. Single README projects should stay with a human reviewer and a link checker.

There is also a maintenance cost that is easy to underestimate: every new tested command, version, and required caveat is a line somebody must keep true. Budget that work explicitly, or the facts file becomes the new stale doc.

9. Where a free model and a free server fit

Drafting prose from a fixed facts file and a marked block contract is low-risk model work, and it is the part worth automating first. I used MonkeyCode's free model access to produce the prose-only drafts and its free server option to run the gate and the mutation harness in a throwaway environment before opening a review. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Availability and limits of the free tiers are whatever the operator currently offers, so confirm them yourself rather than reading them into this post. If you already have a CI runner you trust, use that instead; the harness has no dependency on any particular model or host.