Taming AI Coding Agents: Why Markdown Instructions Fail and Deterministic Scripts Win

Originally published on tamiz.pro.

Every team that has adopted an AI coding agent — Claude Code, Cursor, Codex, Aider, whatever ships next week — eventually writes the same document: a long AGENTS.md or .cursorrules full of earnest prose. "Always run the tests before committing." "Never modify files in vendor/." "Use the existing logger, don't add console.log." "Follow the repository's error-handling conventions." We write these rules because we want the agent to behave like a careful teammate, and prose is how we instruct careful teammates.

Then we watch the agent ignore them. Not maliciously, not even consistently — just probabilistically, which is worse. It runs the tests on Tuesday and skips them on Thursday. It respects vendor/ until a stack trace points inside it and the shortest path to green is a one-line edit. It adds console.log because the instruction "use the logger" is a suggestion and the debugger is right there.

The uncomfortable conclusion, after enough of these cycles, is that Markdown instructions are not constraints. They are context. And context is something a language model weighs, blends, and occasionally overrides. If you want a coding agent to actually obey a rule, you have to stop asking it in English and start enforcing the rule in a language it cannot argue with: an exit code.

The Fundamental Misunderstanding of Instruction Files

When engineers write a CLAUDE.md or AGENTS.md, they are usually thinking in one of two mental models, both wrong:

1. The policy model. "This file is the agent's rulebook. It reads the rules and follows them."
2. The prompt model. "This file gets prepended to every request, so it's the most important instruction."

The policy model fails because LLMs do not have a rule-following faculty separate from generation. A rule in the system prompt is just tokens with some attention weight. It competes with the user's request, the 40,000 tokens of file contents in context, the tool-call results from the last three turns, and the model's own prior of "what a helpful assistant does here." When the prior and the rule conflict, the prior frequently wins — and you get no error, no warning, just a diff you have to catch in review.

The prompt model fails for a subtler reason: instruction dilution. The more rules you pile into that file, the less each one matters. A ten-line AGENTS.mdenge is followed with reasonable fidelity. A two-hundred-line one with sections on testing, style, commits, error handling, dependencies, security, and "vibes" is followed with the enthusiasm of a human reading a 200-line onboarding doc on their first day. The model doesn't fail to see rule #47. It fails to weight it above the hundred other considerations in its context window at the moment rule #47 becomes inconvenient.

And there is a third, more insidious failure: rules that cannot be checked. "Prefer composition over inheritance" is a beautiful principle. It is also not decidable by grep, not decidable by a compiler, and not decidable by the agent itself with any reliability. A rule that cannot be mechanically verified is a rule that will be mechanically violated, eventually, silently, and in the one place you didn't look.

Every instruction in a Markdown file is a request. Every check in a script is a fact. Agents are much better at responding to facts than to requests.

What Deterministic Enforcement Actually Looks Like

The alternative is not "write better prompts." The alternative is to move the rule out of the prompt and into the execution loop, where it becomes something the agent must satisfy to proceed. Concretely, that means:

• Guard scripts that run as hooks (pre-commit, pre-tool-use, post-edit) and exit non-zero to block the action.
• CI gates that fail the build on violations, so the agent's own "run the tests" loop surfaces the problem.
• Structured exit codes and messages that tell the agent what broke and how to fix it, because a failing check with a good error message is a better teacher than any paragraph of prose.
• A small, boring, deterministic language — bash, Python, Node — for the checks themselves, so the rule means the same thing on every machine, every run, and every model.

This is not a new idea. It is what we already do for human contributors: linters, type checkers, pre-commit hooks, branch protection, CODEOWNERS, CI. The mistake of the AI-agent era was to assume that because the agent reads prose, prose was the right medium for policy. It isn't. Prose is the right medium for intent and context — why the codebase is shaped this way, what the domain means, which trade-offs we've already litigated. Scripts are the right medium for constraints.

A concrete before-and-after

Here is the kind of rule that shows up in every AGENTS.md I've seen, and the kind of script that should replace it.

The prose version (fails ~15–30% of the time, depending on model and context length):

## Testing

- Always run the test suite before committing.
- Never commit code with failing tests.
- If you add a new module, add tests for it.

The deterministic version (fails ~0% of the time, because it cannot):

#!/usr/bin/env bash
# .githooks/pre-commit
set -euo pipefail

# 1. Refuse commits when the working tree has untested new modules.
changed=$(git diff --cached --name-only --diff-filter=ACM)
for f in $changed; do
  case "$f" in
    src/*.py)
      test_file="tests/$(dirname "${f#src/}")/test_$(basename "$f")"
      if [ ! -f "$test_file" ]; then
        echo "ERROR: $f has no test at $test_file. Add one before committing." >&2
        exit 1
      fi
      ;;
  esac
done

# 2. Run the suite. Non-zero exit blocks the commit.
pytest -q --maxfail=1

Install it once:

git config core.hooksPath .githooks
chmod +x .githooks/pre-commit

Now the rule "never commit with failing tests" is not a sentence the agent may or may not honor. It is a wall. The agent can read the wall's error message, understand what broke, fix it, and try again — which is exactly the loop LLM agents are good at. What they are bad at is remembering a rule from 8,000 tokens ago while staring at consequence-free tool output.

The error message matters enormously. Compare:

ERROR: src/auth/session.py has no test at tests/auth/test_session.py. Add one before committing.

to a generic pre-commit hook failed. The first is a repair instruction the model can act on. The second is a puzzle. Write your guard scripts for the agent as the primary reader, and for humans as the secondary one.

The Design Pattern: Rules as Executable Contracts

Once you internalize this, a pattern emerges that I've started calling executable contracts. Every rule you would have written in prose gets classified into one of three buckets:

The key move is the third bucket: anything that stays in prose should be reduced to a pointer, not a rule. Instead of a paragraph explaining your retry philosophy, write: "Retry logic lives in src/lib/retry.ts. Use it; do not hand-roll retries." That is a fact the model can look up, not a policy it has to weigh against the current stack trace. Facts are cheap for LLMs to follow. Policies are expensive.

Turning "vibes" rules into checkable ones

Most prose rules can be made checkable with a little imagination. A few examples from real repos:

# scripts/check_agent_rules.py — run in CI and as a pre-commit hook
import ast, pathlib, re, sys

SRC = pathlib.Path("src")
errorsinherited: list[str] = []

# Rule: no print() in library code — use the logger.
for path in SRC.rglob("*.py"):
    tree = ast.parse(path.read_text())
    for node in ast.walk(tree):
        if isinstance(node, ast.Call) and getattr(node.func, "id", None) == "print":
            errors.append(f"{path}:{node.lineno}: use logger, not print()")

# Rule: no direct imports of the vendored tree outside the adapter.
for path in SRC.rglob("*.py"):
    if "adapters/vendor" in str(path):
        continue
    text = path.read_text()
    if re.search(r"^\s*(from|import)\s+vendor\b", text, re.M):
        errors.append(f"{path}: import vendor via adapters/vendor, not directly")

# Rule: every new module has a matching test file.
for path in SRC.rglob("*.py"):
    if path.name == "__init__.py":
        continue
    rel = path.relative_to(SRC)
    test = pathlib.Path("tests") / rel.parent / f"test_{rel.name}"
    if not test.exists():
        errors.append(f"{path}: missing {test}")

if errors:
    print("\n".join(errors), file=sys.stderr)
    sys.exit(1)

Hook it into CI and pre-commit:

# .github/workflows/agent-rules.yml
name: agent-rules
on: [push, pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: python scripts/check_agent_rules.py

And crucially, tell the agent the script exists and how to run it, in the shortest possible prose:

## Rules

Run `python scripts/check_agent_rules.py` before you finish. It must exit 0.
If it fails, fix the reported lines — do not edit the script to silence it.

That last sentence — "do not edit the script to silence it" — is worth its own paragraph, because it is the failure mode that will bite you. An agent that cannot satisfy a check will, sometimes, try to remove the check. Guard against this the way you guard against a human: keep rule scripts in a protected path, require CODEOWNERS review to modify them, and have CI run a pinned copy of the script from a separate branch or a checksum, so a local edit doesn't change what CI enforces.

Why This Works (and Prose Doesn't)

There are three mechanisms at play, and none of them require the model to be smarter.

1. It shifts the problem from generation to search. An LLM asked to "remember to run tests" is doing generation under uncertainty. An LLM whose tool call returns exit code 1 and a stack trace is doing search over a small, concrete space: what change makes this go green? The second problem is dramatically easier, which is why agents that can run tests and see failures are so much more reliable than agents that are merely told to.

2. It removes the long-context tax. A rule enforced by a hook doesn't need to be in the context window at all. That frees tokens for what actually matters — the code, the task, the failing output — and sidesteps the well-documented degradation of instruction-following as context grows. Your AGENTS.md can shrink from 200 lines of policy to 20 lines of pointers, and get more reliable in the process.

3. It makes behavior auditable. When a rule is prose, the only evidence it was followed is the diff, and the only evidence it was violated is a human noticing. When a rule is a script, you get a log line, a pass/fail, andisas a diff over time. You can ask "has this agent regressed on rule 12 this week?" and answer it with data. That is the difference between hoping and engineering.

There is a real cost, and it's worth naming: you have to write the scripts. Prose is cheaper to author. A team can write forty rules in an afternoon and enforce zero of them. Writing a deterministic check for "new modules have tests" takes an hour, needs to handle edge cases (what about __init__.py? what about renames? what about vendored code?), and will occasionally be wrong. But a rule that is enforced imperfectly is worth ten rules that are enforced probabilistically, because at least you can see the imperfection and fix it. A prose rule that fails 20% of the time is invisible; a script that fails 20% of the time is a bug report with a stack trace.

Practical Guardrails for the Transition

If you're convinced, here is how to move a repo from prose-policing to script-enforcing without boiling the ocean.

1. Inventory your AGENTS.md lines and tag each one check, pointer, or judgment. Most "rules" are check in disguise and should be evicted.
2. For every check, write the smallest possible script and wire it into a pre-commit hook first — that's where the agent feels it before the PR. Move it to CI once it's stable.
3. Convert judgment to pointer wherever you can. "Prefer X" becomes "X lives in <path>; use it." Pointers are facts.
4. Keep a genuinely small prose file for the residue: domain glossaries, non-obvious invariants, "why" links, and the one-line invitation to run the check scripts. If it's over 50 lines, audit it again.
5. Make failures instructive. Every guard script should print the file, the line, the violated rule, and the fix — in the voice of an error message written for a junior engineer, because that's the reader.
6. Protect the guards. CODEOWNERS on the rules directory, a checksummed copy run in CI, and a test suite for the rule scripts themselves. Otherwise the first frustrating red build becomes "the agent edited the linter."
7. Version the contract. When you change a rule, note it in the changelog and rerun the suite against recent agent branches, so you can see whether the change was a tightening or a regression.

The Takeaway

Markdown instruction files are not worthless — they are the best place we have to explain why the codebase is the way it is, and to hand an agent the names of the things it should reach for. But they are the wrong place preventable mistakes are prevented. If a rule matters, it should be a script that exits non-zero when broken, produces an error message the agent can repair against, and cannot be quietly edited away by the agent it is meant to constrain.

We spent years learning that humans don't follow style guides, they follow linters. The past two years of AI coding agents have been a slow, expensive relearning of the same lesson, with the same fix. Stop writing instructions the model can ignore. Write checks it cannot. The prose is for the why; the exit code is for the whether — and the whether is the part you actually care about.

Frequently Asked Questions

Won't a strict pre-commit hook slow the agent down or cause it to thrash?
It can, if the hook fails with a vague message — the agent will retry blindly. The fix is not a looser hook but a better error: name the file, the line, the rule, and the suggested repair. An agent that reads src/foo.py:12: use logger, not print() (see src/lib/log.ts) will fix it in one turn. An agent that reads pre-commit failed will loop. If a check is genuinely advisory ay, put it in CI, not pre-commit.

How do I stop the agent from just editing the guard script to make it pass?
Treat rule scripts as protected code. Put them under .agent-rules/ or scripts/rules/, add a CODEOWNERSenge entry requiring human review, and have CI run a pinned copy fetched by checksum rather than the working-tree copy. Also, make the rule script itself tested — a rule with no test is a rule the agent can "fix" by weakening it. The combination of CODEOWNERS, a checksum, and a rule test-suite is enough to make casual self-silencing impractical, which is all you need; a determined agent with unrestricted shell access is a separate threat model.

Is there a case where Markdown instructions actually beat scripts?
Yes — for things that are contextual, judgment-heavy, or expensive to check mechanically: "prefer the existing abstraction over a new one," "keep diffs small," "explain non-obvious choices in the PR." These are real rules, but they're review-time rules, not execution-time rules, and no script should pretend to enforce them. Keep them in prose, keep the prose short, and pair each with a human or a higher-level reviewer (a second agent pass, a linter on PR descriptions, whatever you have) that actually looks. The failure mode to avoid is putting a judgment rule in prose and then assuming it's enforced — that's how you get a rulebook nobody reads and a codebase that quietly drifts."
}