Let the Free Model Write Inputs, Not Assertions: A Mutation Gate Before Your Refactor

A passing test suite can hide a refactor bug for months. That happens when the tests assert nothing real.

Free coding models make this worse. They produce plausible tests fast. Most of those tests pin the current behavior only by accident.

Here is the rule I use now. Let the model propose inputs. Take expected values from the code as it runs today. Then filter the suite with a small mutation gate.

Only then do I touch the function.

The problem with model-written assertions

A model writing a characterization test makes two guesses. It guesses an input. It also guesses the output.

Those guesses fail differently.

Bad inputs are obvious. They crash, or they never reach the branch you care about. Bad assertions are silent. They lock in behavior the code never had.

So split the job:

That split is the whole method. Everything below is mechanics.

Step 1: Freeze the target function

Pick one function. Not a module, not a package. One function with one job.

In this example it is a legacy discount calculator. It reads five module-level constants and has never had a test.

# legacy/pricing.py
def compute_discount(cents, tier, months, coupon=None):
    ...

Do not edit it yet. Note the import path. That path is your contract for the rest of the exercise.

Step 2: Build a golden corpus from real inputs

Pull inputs from logs, fixtures, or the bug tracker. Twenty to sixty cases is enough to start.

Store them as data, not as code:

[{ "cents": 1999, "tier": "pro", "months": 12 },
 { "cents": 0, "tier": "free", "months": 1, "coupon": "HALF" }]

Then record what the current code returns. Serialize deterministically, or the goldens will drift on their own.

# goldens.py
import json, pathlib
from legacy.pricing import compute_discount

def serialize(result):
    return json.dumps(result, sort_keys=True, default=str)

cases = json.loads(pathlib.Path("fixtures/pricing_cases.json").read_text())
goldens = {str(i): serialize(compute_discount(**c)) for i, c in enumerate(cases)}
pathlib.Path("fixtures/pricing_goldens.json").write_text(
    json.dumps(goldens, indent=2, sort_keys=True))

Run it once. Commit both files. The goldens are now the ceiling of what you are allowed to change.

Step 3: Ask the model for inputs only

The prompt matters more than the model. Ask for cases, never for expected values.

Below is the signature and a short description of a legacy function.
Propose 25 input combinations that are most likely to break a refactor.
Prioritize: zero and negative values, boundary tiers, coupon interactions,
and values that appear only in one branch. Output JSON list of kwargs only.
Do not include expected outputs. Do not explain.

I run this on MonkeyCode's free model access, because the task is cheap and repetitive and I do not want a paid quota in the loop. Free tokens at the time of writing are advertised by the operator as a generously sized allowance; check the project page for the current number, since terms change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Paste the returned JSON into fixtures/pricing_cases.json. Regenerate goldens. Read every new row.

Reject anything that raises. Reject anything that hits a network or clock path. Keep the rest, even the weird ones.

Step 4: Turn the corpus into characterization tests

One parametrized test. No hand-written assertions.

# tests/test_pricing_char.py
import json, pathlib, pytest
from legacy.pricing import compute_discount
from goldens import serialize

CASES = json.loads(pathlib.Path("fixtures/pricing_cases.json").read_text())
GOLDENS = json.loads(pathlib.Path("fixtures/pricing_goldens.json").read_text())

@pytest.mark.parametrize("idx", range(len(CASES)))
def test_matches_golden(idx):
    assert serialize(compute_discount(**CASES[idx])) == GOLDENS[str(idx)]

Green. Now the important part: this suite might still be worthless.

Step 5: Score the suite with a mutation gate

Coverage tells you the test ran the line. Mutation tells you the test would notice a change.

You do not need a framework. Sixty lines of stdlib is enough for one function.

# mutate.py
import json, pathlib, re, shutil, subprocess, sys, tempfile

TARGET = pathlib.Path("legacy/pricing.py")
TESTS = "tests/test_pricing_char.py"

MUTANTS = [
    (r"<=", "<"),
    (r">=", ">"),
    (r" == ", " != "),
    (r" and ", " or "),
    (r"return 0\b", "return 1"),
]

def suite_green(root):
    p = subprocess.run([sys.executable, "-m", "pytest", "-q", TESTS],
                       cwd=root, capture_output=True, text=True)
    return p.returncode == 0

def main():
    src = TARGET.read_text()
    killed, total = 0, 0
    for pattern, repl in MUTANTS:
        for match in re.finditer(pattern, src):
            total += 1
            with tempfile.TemporaryDirectory() as tmp:
                root = pathlib.Path(tmp)
                shutil.copytree(".", root / "repo", dirs_exist_ok=True)
                mutated = src[:match.start()] + repl + src[match.end():]
                (root / "repo" / TARGET).write_text(mutated)
                if not suite_green(root / "repo"):
                    killed += 1
    print(json.dumps({"killed": killed, "total": total,
                      "score": round(killed / total, 3) if total else None}))

main()

Run it. Read the score, not the percentage sign.

python goldens.py && python mutate.py
# {"killed": 5, "total": 19, "score": 0.263}

The numbers above are illustrative shape, not a measurement of any product. Your score is the only one that matters.

Step 6: Use the score to decide

Low mutation score means your corpus is thin, not that your code is bad. Add inputs, not assertions.

Surviving mutants are free documentation. Each one shows a branch your corpus never exercises.

Loop twice. Three passes rarely pay for themselves.

Step 7: Make the smallest safe change

Now, and only now, edit. One extraction. One rename. Nothing else.

def _tier_rate(tier):
    if tier == "pro":
        return PRO_RATE
    return BASE_RATE

Re-run both scripts. Goldens must match byte for byte. The mutation score must not drop.

If the score drops, your extraction changed a branch. Revert. Do not reorder.

Where a free server fits

The mutation loop is slow and dumb. It copies the repo once per mutant and runs pytest.

That is a batch job, not an interactive one. A free server instance from the same project is a reasonable place to park it, because the workload is short and bursty. Specs and availability are operator-declared and can change, so confirm them before you depend on the box.

Keep the loop in CI as well. A gate you run by hand is a gate you skip.

Limitations

• Equivalent mutants. Some substitutions cannot change behavior. They cap your score below 1.00 forever. Read survivors before chasing the number.
• Cost grows fast. Mutants multiply with operators. Bound the run at one function and a few hundred cases.
• Goldens freeze bugs too. A characterization test commits you to current behavior, defects included. Fix bugs in a separate, labeled commit.
• Generated inputs need review. A model can propose cases that touch the filesystem or the clock. Filter them before they enter the corpus.
• This is not a correctness proof. A high mutation score means the suite is sensitive. It does not mean the code is right.

Who should skip this

Skip it if the function has no callers you can enumerate. Skip it if the output is non-deterministic and you cannot stub the source.

Skip it for code under regulatory change control. There, the specification is the artifact, not a recorded snapshot.

And skip it if you cannot commit the golden files. A corpus you regenerate on every run measures nothing.

The takeaway

Free models are strong at enumeration and weak at ground truth. Point them at the first problem.

Record reality with your own runtime, then measure how hard your tests push back. A mutation score of 0.7 before the refactor is worth more than 100 percent coverage after it.

If you want the cheapest place to run step 3, start with a free model endpoint and keep the gate in your own repo.