Halfway through a two-day batch run I noticed that one answer looked suspiciously like the answer to the item two rows above it, and I could not prove which input it came from. Has that ever happened to you — the job finished, the output looked plausible, and you had no way to defend a single row of it? These are my field notes from fixing that, with the harness I ended up keeping.
What I was actually trying to do
I had a flat list of short, independent text items, and I wanted one short answer per item, written to a ledger I could re-run safely. Sending one request per item is the obvious path, but it doubles the number of round trips, and every round trip is another chance to miss a wall-clock deadline. So I tried the cheaper-looking option: put two items in one prompt and ask for two answers back. Note that this is not a quota argument — I never measured a quota, and I am not claiming one exists. The motivation was purely fewer round trips and fewer half-finished batches.
For context, the driver ran on MonkeyCode's free server option, and the calls went through its free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I will keep the product out of the technical claims, because the failure below is a property of my prompt and my validator, not of any provider.
Day one: the harness I wrote
The whole thing is one file with no dependencies beyond the Python standard library, which was deliberate. If the harness can move between machines with scp and nothing else, I can debug it anywhere, including on a machine I do not own. Here is the shape of it.
# batch_harness.py — stdlib only; you supply call_model()
import hashlib
import json
from dataclasses import dataclass, field
def item_id(text: str) -> str:
return hashlib.sha256(text.strip().encode("utf-8")).hexdigest()[:12]
@dataclass
class Item:
text: str
id: str = field(init=False)
def __post_init__(self):
self.id = item_id(self.text)
def build_prompt(items):
blocks = "\n".join(f'<item id="{i.id}">{i.text}</item>' for i in items)
return (
"Answer each item separately. Return a JSON array only, no prose.\n"
'Each element must be {"id": "<id from the item tag>", "answer": "<answer>"}.\n'
"Copy ids exactly. Do not renumber, merge, or reorder the items.\n\n"
+ blocks
)
The important line is the id, and it is deliberately boring: a hash of the input text, truncated. Same input, same id, forever — so a second run of the same ledger is a no-op by construction. If the id were derived from the model's output, a reworded answer would look like a brand-new item, and the queue would quietly grow.
The validator, which is where the actual work lives
def validate(items, raw):
try:
payload = json.loads(raw)
except json.JSONDecodeError as exc:
return None, f"not_json: {exc.msg}"
if not isinstance(payload, list):
return None, "not_a_list"
rows = [r for r in payload if isinstance(r, dict)]
got = [str(r.get("id", "")).strip().lower() for r in rows]
want = [i.id for i in items]
if len(got) != len(set(got)):
return None, "duplicate_ids"
if set(got) != set(want):
missing = sorted(set(want) - set(got))
extra = sorted(set(got) - set(want))
return None, f"set_mismatch missing={missing} extra={extra}"
answers = {r["id"].strip().lower(): r.get("answer", "") for r in rows}
if any(not isinstance(a, str) or not a.strip() for a in answers.values()):
return None, "empty_answer"
return answers, "ok"
Two details I will defend. First, I lowercase and strip the id before comparing, because that transform is lossless — an uppercase hex string identifies exactly one item. Second, I refuse every lossy match: no positional mapping, no prefix matching, no edit distance. Fuzzy matching is precisely how you end up with two answers for one item and a warm feeling that everything is fine.
What broke on day two
Two failure modes showed up, and both were invisible to a try/except around the HTTP call.
-
Answers came back without ids. The response was a clean JSON array of
{"answer": ...}objects, in a plausible order. Nothing errored. A positional mapper would have accepted this and been right — probably. - Ids came back that I never sent. One batch returned eleven-character ids instead of twelve; another returned ids that were valid hex but belonged to nothing in my prompt. Set equality caught both instantly; a length check would not have.
Why is partial acceptance worse than a loud failure here? Because a batch that is 50% valid does not stop the run — it just moves two items into the wrong bucket, and you discover it when someone reads the output.
The fix: fail the batch, then fall back to one
def run_batch(items, call_model, attempts=2):
for attempt in range(1, attempts + 1):
raw = call_model(build_prompt(items))
answers, verdict = validate(items, raw)
if verdict == "ok":
return answers, {"attempts": attempt, "verdict": verdict, "fallback": False}
out = {}
for item in items:
raw = call_model(build_prompt([item]))
answers, verdict = validate([item], raw)
out[item.id] = answers[item.id] if verdict == "ok" else None
return out, {"verdict": "fallback_single", "fallback": True}
python batch_harness.py --input items.jsonl --batch-size 2 --ledger runs/ledger.jsonl
tail -n 3 runs/ledger.jsonl
{"batch": ["9f2c41ab07de", "41ab9f2c07de"], "attempt": 1, "verdict": "set_mismatch", "extra": ["7d10c3a1f0b2"], "fallback": true, "model_string": "<record what you actually got>"}
The ledger excerpt above is illustrative — the values are placeholders, and the deliverable is the columns, not the numbers. Record the model string you actually received, because free access is not the same thing as a pinned version, and a drifting model is a variable you want in the log.
The run sheet I would keep
| Column | Why it earns its place |
|---|---|
batch (sorted ids) |
Reproducing a failure needs the exact inputs, not the item count |
attempt |
Tells you whether retries are rare or structural |
verdict |
Distinguishes not_json from set_mismatch, which are different bugs |
extra / missing
|
The invented ids are the signal that the prompt drifted |
fallback |
A rising fallback rate means the batch size is too aggressive |
And a decision table for batch size, which is really a question about failure cost:
| Items per call | Use it when | Do not use it when |
|---|---|---|
| 1 | Items are long, or ordering is contractual | You are already comfortable with the fallback path |
| 2 | Items are short, independent, and id-stable | One bad item can poison its neighbour |
| 4+ | You have eyes on the ledger during the run | You cannot inspect the first hundred rows by hand |
What I would repeat, and what I would not
I would repeat the input-derived id and the strict set-equality check on day one, not day two; both are ten lines and they replace a whole class of late-night doubt. I would repeat keeping every model call behind one function, because it made the fallback trivial to write. I would not repeat starting at batch size two before watching a hundred single-item calls go through cleanly. The boring baseline is what tells you whether batching is safe for your particular items.
Who should not do this
- Anyone whose items reference each other, since batching changes what the model can see at once.
- Anyone under a hard latency or availability guarantee, because a free endpoint is a shared one and I make no promises about it.
- Anyone who cannot hand-inspect the first hundred outputs, since the validator checks shape and nothing else.
- Anyone paying per token on a long shared instruction, where duplicating it per item costs more than the round trips you save.
The harness does not know whether an answer is correct — it only knows that every id it sent came back once. That is a smaller promise than it sounds, and it is the one I can actually keep. If you run this drill on the free model access, start at batch size one and let the ledger tell you when two is safe.
Top comments (0)