Pin the Broken Input: Characterization Tests Before an OSS Parser Patch

On a mid-size Python CLI used by data teams, a contributor shipped a quoting fix that looked obviously correct. The local suite stayed green because every fixture used short fields without embedded line breaks. A week later a maintainer reproduced a production CSV that contained a quoted newline inside one field. The revert itself was small, yet review time was large because the failing input never became a project test.

That pattern is getting more common as assistants draft patches quickly and fill undocumented gaps with guesses. The model treats a missing fixture as unused behavior instead of untested behavior, then lands a library swap that the issue never requested. Maintainers inherit a green CI run and a silent change to stdout, exit codes, or dependency pins. A fail-closed loop freezes the broken bytes first, so a later diff cannot drift away from the report.

Why a green suite still misses the regression

Maintainers often inherit patches that optimize the happy path and leave the reported failure unfrozen. Characterization tests record what the program actually does on a captured input before anyone rewrites logic. They do not claim correctness up front; they pin observed behavior so a later diff cannot drift silently. For open source work, that pin is the cheapest artifact a contributor can hand a reviewer who lacks the original file.

When an assistant proposes a patch, it tends to assume missing fixtures are unused rather than untested. That assumption matches a broader pattern in agent-style coding: the model fills gaps instead of stopping. A fail-closed loop treats those gaps as blockers, not as invitations to invent a flag change. The working unit is not a chat transcript; it is a file a clone can run with one command.

A fail-closed contribution sequence

The sequence below is a worked example for a fictional csvsplit command-line tool, not a claim about a live repository. Each step produces a file a maintainer can run without reconstructing the original bug report from memory. The same files later become the only context a review model should see. Extra repository history stays out of the prompt unless a command proves it is required.

1. Minimize the failing input

The first job is not to edit product code; it is to freeze the smallest input that still fails in production. Store that byte sequence under tests/fixtures/ so later clones do not depend on a screenshot or a chat paste. Avoid pretty-printing the fixture, because formatters often normalize the exact bytes that expose the parser defect. A hex dump is cheap insurance when the file contains mixed line endings or an unmatched quote.

mkdir -p tests/fixtures
printf 'id,note\n1,"line one\nline two"\n' > tests/fixtures/embedded_newline.csv
xxd tests/fixtures/embedded_newline.csv
file tests/fixtures/embedded_newline.csv

Keep the fixture in the patch from the first commit, even before product code changes. Reviewers can then replay the defect without asking the reporter to resend a private spreadsheet. If the original file cannot be shared, reduce it until every remaining byte is still required for the failure.

2. Write a characterization test that fails for the right reason

A characterization test should name the behavior under dispute, not the implementation the contributor hopes to land. For csvsplit, the disputed behavior is that a quoted field may contain a newline without splitting the record. The test below is a worked example and is intended to fail on the default branch until a later patch makes it pass. A test that already passes on main is characterizing a different program.

# tests/test_embedded_newline.py
# Worked example for a fictional csvsplit CLI. Not executed against a live project.
from pathlib import Path
import subprocess
import sys

FIXTURE = Path(__file__).parent / "fixtures" / "embedded_newline.csv"
CONTRACT_COLUMNS = ["id", "note"]


def run_csvsplit(path: Path) -> subprocess.CompletedProcess:
    return subprocess.run(
        [sys.executable, "-m", "csvsplit", str(path)],
        capture_output=True,
        text=True,
        check=False,
    )


def test_quoted_newline_stays_in_one_record():
    result = run_csvsplit(FIXTURE)
    assert result.returncode == 0, result.stderr
    lines = [line for line in result.stdout.splitlines() if line]
    # Characterization: one header plus one data record, even with an embedded newline.
    assert len(lines) == 2, lines
    assert lines[0].split(",") == CONTRACT_COLUMNS
    assert "line one" in lines[1] and "line two" in lines[1]

Run that file against the default branch before generating any fix.

git fetch origin
git switch -c fix/quoted-newline origin/main
python -m pytest tests/test_embedded_newline.py -q

Stop if the test passes. The fixture is wrong, the bug is already gone, or the CLI entry point in the test does not match the project. Continue only when the failure message matches the original report in exit code, record count, or parsed fields.

3. Freeze the public contract beside the test

AI-authored patches frequently "fix" a parser by changing exit codes, flag names, or stdout shape. Those changes look local in the diff and then break scripts that already parse the CLI in user pipelines. Record the contract in a short file the review prompt can load without scraping the entire README tree. Bugfix pull requests should not add flags or rewrite dependency pins unless the issue asked for that work.

# contract/cli.md
- command: python -m csvsplit <file>
- exit 0 on valid CSV; exit 2 on usage errors
- stdout: header row, then one line per record
- quoted newlines must not create extra records
- stderr stays empty on success
- do not add flags in a bugfix PR
- do not change lockfiles or runtime dependencies in a bugfix PR

Keep that file in the same commit as the fixture. The contract is not marketing copy; it is a reject list for later review. If the project already documents these rules, quote the existing section instead of inventing a parallel spec.

4. Patch only against the frozen failure

Contributors should edit the smallest parser path that makes the characterization test pass, then rerun the full suite. A passing characterization test plus a failing unrelated test means the patch leaked into another contract. Dependency diffs, generated lockfile noise, and drive-by refactors belong in a separate proposal, not in the bugfix branch. If the issue can be fixed with a one-line guard, the review should say so before anyone accepts a parser rewrite.

python -m pytest -q
git diff --stat origin/main...HEAD
git diff origin/main...HEAD -- pyproject.toml poetry.lock requirements.txt

Empty dependency diffs are a feature of this loop, not an omission. A model that "helpfully" bumps a CSV library is expanding the blast radius beyond the frozen fixture. Maintainers can then review one behavior change instead of a hidden stack upgrade.

5. Review the diff against the frozen failure, not against style

A review prompt should receive the fixture, the test, the contract file, and the diff, and nothing optional. The model is useful when it hunts for contract violations, extra dependency edits, and tests that were deleted to stay green. It is not useful when it rewrites style, renames internals, or proposes a library swap that the issue never requested. Treat every finding as a hypothesis and re-run the commands above before changing the branch.

You are reviewing an OSS bugfix, not generating a new feature.
Read contract/cli.md, tests/fixtures/embedded_newline.csv,
tests/test_embedded_newline.py, and the git diff.
Report only:
1. contract violations (exit code, stdout shape, flags)
2. files that are unrelated to the frozen failure
3. tests skipped, deleted, or weakened
4. dependency or lockfile edits
Do not suggest style renames or library replacements.
Do not invent benchmarks.

Paste the four artifacts, not the whole repository. Wide context invites the model to assume undocumented callers and "fix" them. Narrow context keeps the review inside the bytes the reporter actually supplied.

Decision table for shipping, splitting, or stopping

Use the table as a gate before opening the pull request. The actions are mechanical on purpose, so a tired reviewer does not negotiate with a green but unrelated diff.

The table is the original artifact of this workflow, not a scoreboard. It does not measure model quality. It measures whether the pull request still talks about the same failure the reporter filed.

Where free model access and a free server actually help

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A coding environment with free model access can run the review prompt against the four artifacts without turning the whole tree into context. A free server option is useful when the contributor's laptop has the wrong Python version, or when the suite needs a clean checkout that local virtualenv residue would contaminate. Neither substitute for the fixture, the failing test, or the contract file.

Keep product code generation secondary. The model should spend its pass on the diff after the test already fails on main for the right bytes. If the server run and the laptop run disagree, trust the clean checkout and treat local packages as suspects. Publish the commands, not a screenshot of a chat window, so maintainers can replay the same gate.

Limitations

This loop does not characterize timing bugs, GUI event order, or failures that need live network credentials. Fixtures that contain personal data, license keys, or production hostnames must never enter the repository, even in minimized form. Characterization tests can pin a wrong behavior if the reporter's file is not actually invalid, and then the patch cements that wrongness. Parser bugs that span several files may still need a maintainer-written design note after the first failing test exists.

The workflow also assumes the project has an automated test runner a stranger can invoke. A repository that only documents manual clicks will not gain much from a pytest harness. Models will still hallucinate flags that look plausible in CLI tools; the contract file is a filter, not a guarantee. Free model access and a free server option do not make an unlicensed contribution acceptable, and they do not replace CODEOWNERS review.

Who should not use this approach

Drive-by typo patches do not need a characterization harness, and adding one creates noise for maintainers. Contributors without a signed CLA, a DCO trailer, or permission to use the reporter's file should stop before committing fixtures. Security issues that become exploitable once a public test includes the payload belong on the private maintainer channel, not in tests/fixtures/. Teams that already have a strict golden-file suite can skip the extra contract file if the existing goldens already freeze exit codes and stdout.

People chasing a large refactor should also skip this loop. The method is for a reported behavioral defect with a replayable input, not for "clean up the parser" work that lacks a failing byte sequence. If no input can be shared or reduced, the honest next step is to ask the reporter for a redacted fixture, not to let a model invent one.

The useful outcome is a pull request whose first commits contain the broken input, a test that failed on main, and a diff that only answers that test. Maintainers can replay those commits without reconstructing a chat. Contributors who need a clean machine for the suite and a bounded model pass over the four artifacts can run the same loop with MonkeyCode's free model access and free server option, then leave the harness in the PR for the next reviewer.