Making LLM-as-Judge Scoring Reproducible and Trustworthy

I love using a language model to grade another model's output. A judge model can score "is this support reply empathetic and correct?" in a way that a regex never will. But the first time I wired one into CI, my test suite started flaking: the same input scored 0.8 one run and 0.55 the next, and a green build turned red for no reason I could point to. An eval you cannot trust to give the same answer twice is not a gate. It is a coin flip with extra latency.

This is a tutorial on making LLM-as-judge scoring reproducible. I will ground it in the llm-judge scorer from my eval tool, evalgate, but the techniques carry to any stack. There are five moves: a fixed rubric, structured output, temperature 0, a deterministic mock judge for tests, and a guard against judge drift.

1. Freeze the rubric into the prompt

The single biggest source of noise is a vague ask. "Is this a good answer?" invites the judge to invent fresh criteria every call. Instead, pass the exact rubric and pin the response shape. Here is how evalgate builds the judge prompt:

export function buildJudgePrompt(criteria: string, output: string, expected?: string): string {
  return [
    "You are a strict evaluator. Score how well the RESPONSE satisfies the CRITERIA.",
    expected ? `REFERENCE (an ideal answer):\n${expected}` : "",
    `CRITERIA:\n${criteria}`,
    `RESPONSE:\n${output}`,
    'Reply with ONLY compact JSON: {"score": <number between 0 and 1>, "reason": "<short>"}',
  ]
    .filter(Boolean)
    .join("\n\n");
}

Three things are doing the work here. The criteria are supplied by the caller, not improvised by the model, so the standard is constant across cases. An optional reference answer anchors the judge to a concrete target instead of its own taste. And the last line forces the output into a machine-readable contract. A judge that must return prose is a judge you cannot compare run to run.

2. Parse structured output defensively

Even with a clear instruction, models occasionally wrap JSON in commentary or a code fence. So do not trust JSON.parse on the raw reply. Extract the object, parse it, and degrade to a defined failure instead of throwing:

export function parseJudgeReply(text: string): { score: number; reason: string } {
  const match = text.match(/\{[\s\S]*\}/);
  if (match) {
    try {
      const obj = JSON.parse(match[0]);
      const score = Number(obj.score);
      if (!Number.isNaN(score)) {
        return { score, reason: String(obj.reason ?? "judge returned a score") };
      }
    } catch {
      // fall through to the failure path
    }
  }
  return { score: 0, reason: `could not parse judge reply: ${text.slice(0, 80)}` };
}

The point is that an unparseable reply is a scored event, not a crash. It gets a 0 and a reason that tells you exactly what came back. That reason string is worth its weight in gold when you are staring at a failed CI run at midnight.

3. Temperature 0

The judge is not a creative task. You want the most likely token every time, not variety. Set temperature to 0 on the judge call:

const reply = await ctx.provider.complete({
  model: (spec.model as string) || ctx.case.model || "",
  prompt: buildJudgePrompt(criteria, ctx.output, expected),
  temperature: 0,
});
const parsed = parseJudgeReply(reply.output);

Temperature 0 does not make a model perfectly deterministic (batching and hardware can still nudge results), but it removes the sampling variance that was turning my scores into a lottery. It is the cheapest reproducibility win available, and there is no reason to skip it for grading.

4. A deterministic mock judge for tests

Here is the part people forget. If your scorer's own unit tests call a real model, your tests are slow, cost money, and flake for the same reasons your evals did. You cannot test a nondeterministic thing with a nondeterministic thing.

The fix is a mock provider that computes a reproducible score with no network. In evalgate, when the provider is mock, the judge blends word overlap against the reference and the criteria instead of calling out:

if (ctx.provider.name === "mock") {
  const refScore = expected ? overlap(ctx.output, expected) : 0;
  const critScore = overlap(ctx.output, criteria);
  score = expected ? 0.7 * refScore + 0.3 * critScore : critScore;
  reason = `mock judge overlap score ${score.toFixed(3)}`;
}

The overlap here is a plain Jaccard set intersection over lowercased words, which is fully deterministic. Now the tests run offline and give the same answer every time:

it("scores high when output matches the reference", async () => {
  const r = await run(
    llmJudgeScorer,
    { type: "llm-judge", criteria: "should be correct", expected: "paris is the capital of france", threshold: 0.5 },
    ctx("paris is the capital of france"),
  );
  expect(r.passed).toBe(true);
});

The mock is not pretending to be smart. It is a stand-in that exercises the same code path (criteria required, reference fallback, threshold comparison) so you can prove the plumbing works without a live model in the loop.

5. Guard against judge drift

The last risk is slow: the provider silently ships a new model version and your grading standard shifts underneath you. Two habits keep this honest. First, pin the judge model explicitly rather than defaulting to "latest," so an upgrade is a deliberate diff you can review. Second, keep a small set of golden cases with known-correct scores and re-run them whenever you change the judge model or the rubric. If a case that used to score 0.9 now scores 0.6, that is drift, and you want to catch it in a review, not in production.

The honest caveat

An LLM judge is still an LLM. It can be biased toward longer answers, toward outputs that echo its own phrasing, or toward a particular position in a comparison. Temperature 0 and a fixed rubric make it reproducible, not correct. Reproducible means it gives the same wrong answer twice, which is exactly what lets you notice and fix the wrongness. Treat judge scores as a strong signal that needs periodic human spot-checks, never as ground truth. Calibrate the threshold against cases you have graded by hand.

If you want to see all of this wired together, including the mock provider and the scorer registry, the code is at github.com/AgentPostmortem/evalgate. Borrow the pattern, or just steal the deterministic mock idea for your own suite. Either way, your CI stops flipping coins.