48-Hour Field Notes: My Homemade Eval Set Was Scoring My Prompt, Not the Model

I spent two days comparing models on my own little eval set, and the winner changed every time I reordered the examples. That is not a model difference, that is my eval set leaking.

Here is the setup. Free model access means I can afford three repeats per item instead of one, and a free server option means the sweep can run overnight without me babysitting a laptop. Cheap inference makes homemade evals attractive. But cheap inference also makes it easy to build an eval that measures nothing.

These are my field notes: what I tried, what broke, and the two scripts I would keep.

What I thought I was measuring

I had 60 labelled items for a boring extraction task: pull three fields out of a short support message and return JSON. I wrote eight few-shot examples, picked four candidate models, and ran each item once.

Three numbers came out of that first sweep, and only one of them was about correctness:

• Format compliance — did the reply parse as JSON with the three keys present?
• Abstention behaviour — when the message did not contain a field, did the model say so or invent a value?
• Agreement — did the extracted value match my label?

Agreement is the number everyone quotes. Format compliance and abstention are the ones that page you at 2am. And all three were lying to me, because of a bug that had nothing to do with any model.

The bug: my prompt examples were inside my eval

I built the eval set first, then wrote few-shot examples by copying the clearest items. That is the whole bug. Five of my eight examples were near-duplicates of eval items, so any model that copied an example verbatim got a free point.

You can spot this by eye at 60 items. You cannot spot it at 600, and you definitely cannot spot it after you have paraphrased your prompt three times and forgotten what moved.

So I stopped trusting my eyes and wrote a checker.

Artifact 1: a shingle overlap gate for prompt/eval leakage

The check is deliberately dumb: tokenise, build k-word shingles, and fail when any eval item shares too large a fraction of its shingles with the prompt examples. No embeddings, no LLM judge, nothing you have to trust.

#!/usr/bin/env python3
"""leak_check.py - fail when eval items share too much text with prompt examples.

Usage:
    python leak_check.py prompt_examples.json eval_set.json --threshold 0.30
Exit code 1 means the eval set is contaminated: fix the split before trusting a score.
"""
import argparse
import json
import re
import sys
from pathlib import Path

TOKEN = re.compile(r"[a-z0-9]+")


def shingles(text: str, k: int = 5) -> set[str]:
    words = TOKEN.findall(text.lower())
    if not words:
        return set()
    if len(words) < k:
        return {" ".join(words)}
    return {" ".join(words[i:i + k]) for i in range(len(words) - k + 1)}


def load_items(path: str) -> list[dict]:
    data = json.loads(Path(path).read_text(encoding="utf-8"))
    return data["items"] if isinstance(data, dict) else data


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("prompt_examples")
    ap.add_argument("eval_set")
    ap.add_argument("--threshold", type=float, default=0.30)
    ap.add_argument("--k", type=int, default=5)
    args = ap.parse_args()

    prompt_grams: set[str] = set()
    for item in load_items(args.prompt_examples):
        prompt_grams |= shingles(item["text"], args.k)

    bad = []
    for item in load_items(args.eval_set):
        grams = shingles(item["text"], args.k)
        if not grams:
            continue
        hit = grams & prompt_grams
        ratio = len(hit) / len(grams)
        if ratio >= args.threshold:
            bad.append((item["id"], ratio, sorted(hit)[:2]))

    for item_id, ratio, sample in sorted(bad, key=lambda row: -row[1]):
        print(f"CONTAMINATED {item_id}: {ratio:.2%} overlap, e.g. {sample}")
    print(f"{len(bad)} flagged item(s) at k={args.k}, threshold={args.threshold:.0%}")
    return 1 if bad else 0


if __name__ == "__main__":
    sys.exit(main())

Two rules made this useful in practice. First, run it in CI on every prompt change, not once at design time. Second, when an item trips the gate, rewrite the item instead of deleting it, so your coverage does not quietly shrink every time the gate fires.

The second bug: one sample per item is a coin flip

After the leakage fix, agreement numbers dropped and got noisier. Why? Because I was reading a single sample per item and treating it as a property of the model.

Extraction tasks are boring until the message is ambiguous, and ambiguity is exactly where sampling temperature does its work. A model that is "88% correct" might be getting the same 12% wrong every time, or a different 12% each run. Those are different products.

So I changed the unit of measurement from accuracy to flip rate: the share of items whose label changed across three repeats of the same input.

# sweep.py - N repeats per item per model, one JSONL row per call.
import json
import time
import uuid


def sweep(model_id, items, repeats=3, out="runs.jsonl"):
    with open(out, "a", encoding="utf-8") as fh:
        for item in items:
            for r in range(repeats):
                row = {"model": model_id, "item": item["id"], "repeat": r, "call": str(uuid.uuid4())}
                t0 = time.monotonic()
                try:
                    text = call_model(model_id, item["text"])   # your client, 1 concurrent call
                    row["label"] = parse_label(text)
                    row["error"] = None
                except Exception as exc:
                    row["label"] = None
                    row["error"] = type(exc).__name__
                row["ms"] = round((time.monotonic() - t0) * 1000)
                fh.write(json.dumps(row) + "\n")
                fh.flush()   # a restart must not eat finished work

Summarising is six lines, and it tells you more than the accuracy table did:

import collections, json

rows = [json.loads(line) for line in open("runs.jsonl", encoding="utf-8")]
labels = collections.defaultdict(set)
errors = collections.Counter()
for r in rows:
    if r["error"]:
        errors[(r["model"], r["error"])] += 1
    else:
        labels[(r["model"], r["item"])].add(r["label"])

for model in sorted({m for m, _ in labels}):
    sets = [v for (m, _), v in labels.items() if m == model]
    flipped = sum(1 for v in sets if len(v) > 1)
    print(f"{model}: {flipped}/{len(sets)} items flipped across repeats")
print(dict(errors))

A high flip rate on a free model is not a reason to reject it. It is a reason to know which items are unstable and to route those to a human or a second pass.

Where free access and a free server actually fit

I use MonkeyCode's free model access for the candidate sweep and its free server option to keep the sweep running when my laptop sleeps. That is the honest role: the free tier buys me repeats, which is the only thing that made flip rate measurable at all. If inference cost per call were the constraint, I would have run one sample per item and shipped the noisy table.

Capabilities change, so treat availability as something to verify on the provider's current terms rather than a fixed promise. I deliberately do not quote quotas or model names here, because a number that was true when I wrote this is exactly the kind of stale figure that makes an eval post useless.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The tool is not the point. Swap it for anything and the scripts still work.

What broke, and what I would repeat

What broke:

1. Prompt examples copied from eval items. Fixed with the shingle gate, not with discipline.
2. Single-sample scoring. Fixed by repeats and flip rate.
3. A crash mid-sweep. The flush() per row and the call UUID saved me; without them I could not tell duplicates from retries.
4. Silently changing the prompt mid-sweep. Now the sweep writes the prompt hash into every row.

What I would repeat:

• Write the holdout after the prompt is frozen, never before.
• Keep the gate in CI so future me cannot "just quickly" copy an example into the eval.
• Compare format compliance and abstention before accuracy, because a model that returns prose cannot be parsed and a model that never abstains cannot be trusted.

A comparison table that survives price being zero

When every candidate is free, cost stops being the tiebreaker, so replace it with stability:

Limitations, and who should skip this

Sixty items is not a benchmark. The gate catches copy-paste leakage, not paraphrase leakage, so a human-written twin of an example will sail through. Flip rate with three repeats gives you a smell, not a confidence interval, and none of this replaces a real labelled sample of production traffic.

Skip this approach if you need contractual uptime or a data-processing agreement you have not read, or if your inputs cannot leave your boundary at all — a hosted free tier is the wrong shape for that, no matter how good the numbers look.

If you already have a large frozen eval set, you probably do not need the shingle gate; you need a better sampling policy. If you have no eval set yet, that is the actual first problem.

So: when you next see a headline jump on a narrow task, what would you check first — the model, or the set you scored it on? Run the gate on your own prompt before you answer; it takes about a minute and it will probably find something.