Your team has a number. Maybe it is a pass rate on a held-out set, an LLM-judge score, or a dashboard green light. That number is not trust. Trust is whether the eval is tied to real failures, signed off by people who know the job, and wired into release policy. If the answers to the five questions below are vague, you are shipping on demos.
AI evaluation is the repeatable discipline that fills the gap when the same prompt can produce different plausible answers. An eval, in plain terms, is representative inputs, a definition of good, a scorer, and a bar you rerun when the prompt, model, or data changes. In a January 2025 episode on evaluation-driven development, Atindriyo Sanyal compared the moment to building traditional software in the 1980s, with no eval tooling around it yet. As a lead, you do not need to own the rubric. You need a one-page answer you can read in ten minutes.
What should I demand from whoever owns the eval suite?
Ask for a single written report, updated when the eval changes, not a live demo. The owner should be able to point to the test set, the labelling story, agreement metrics where humans judge quality, a ranked list of failure modes, and the exact rule that blocks a release. Hamel Husain, who teaches evals to engineers and product managers, warns that generic dashboards create a false sense of progress:
These generic metrics are most of the time not helpful at all. They're way too generic. They don't necessarily correlate with actual failures in your AI product, and they don't actually mean anything.
Hamel Husain, Parlance Labs
I'd start by refusing vanity metrics and asking for error analysis instead: read real outputs, label what went wrong, count failure types, fix the biggest buckets first. Husain estimates that most of the work of evals is that kind of looking at data.
Where did the test cases come from, and what fails most often?
Question one for your team: what are the most common ways this system fails, and how often does each happen? Question two: where did the test cases come from, and do they include real failures?
Representative inputs should look like production, including cases the system already failed on. Synthetic cases can fill gaps, but they should not replace documented production misses. Loïc Houssier, CTO of Superhuman, argues that product managers should name the dimensions an eval must cover. His example is a simple email question about time spent in Waymo last month: finding the right emails, interpreting "last month," reading durations, and summing them are separate dimensions to test.
Do not let a public benchmark substitute for this. Alex Ratner of Snorkel AI, whose company builds benchmarks, still says they cannot be the only tool. You need use-case-specific private tests as well.
Who labelled "good," and do the labellers agree?
Question three: who decided what a good answer is, and did someone who knows the business sign off?
For checkable behavior, code can score JSON shape, citation validity, or length. For judgment calls, teams use a second model or people on the cases that matter most. Either way, you should see who wrote the rubric and who approved it. Loïc Houssier puts the product role plainly:
I think it's the role of the PM to identify the dimensions
Loïc Houssier, Superhuman
When two people label the same outputs, percent agreement is not enough. Cohen's Kappa measures agreement beyond chance; Landis and Koch published widely cited bands from slight to almost perfect agreement, while calling those cutoffs arbitrary. Ask the eval owner for kappa (or an equivalent inter-rater method) on a sample of items your judge or human reviewers actually score. If agreement is weak, investigate the labels and scoring before you trust the headline metric.
When you discuss tradeoffs, separate false alarms from misses. Precision and recall spell out that tradeoff: high precision means fewer false positives, high recall means fewer missed real problems. Which side hurts your product more is a business call, not a math default.
Does every change rerun the eval, and what score blocks ship?
Question four: does the eval run on every change, and what score blocks a release? Question five: when something fails in production, how does it get into the test set?
Husain describes three levels of testing in his 2024 essay on evals: quick automated checks on every code change, human and model review on a cadence, and A/B tests after larger product moves. Each level costs more, so they run at different frequencies. Your job is to see the gate at level one and know the numeric or policy threshold that stops a merge.
Ratner names the wider problem many teams feel in production:
the fact that AI capabilities have been advancing so rapidly beyond our capability to measure them
Alex Ratner, Snorkel AI
That is why a static eval deck from last quarter is a risk. The loop from production failure to new test case should be explicit in the one-pager.
What does a one-page eval report look like?
Below is a sketch you can paste into a doc or ticket. It is not a vendor format; it is the minimum I'd accept before calling a feature "eval-backed."
# Eval report: [feature name] · [date]
Owner: [team/person]
## Scope
- Task: [one sentence]
- Dimensions (PM-signed): [bullets]
## Dataset
- N cases: [count] · Sources: [prod failures / support / synthetic]
- Last added prod failure: [date + ticket]
## Labels & scoring
- Rubric author + SME sign-off: [names]
- Scorers: [code checks / LLM judge / human sample]
- Inter-rater: κ = [value] on [n] items ([date])
## Failure taxonomy (from error analysis)
| Category | Share of failures | Top example id |
|----------|-------------------|----------------|
| ... | ... | ... |
## Release gate
- Runs on: [every prompt change / every deploy / weekly]
- Ship if: [e.g. pass rate ≥ X on set Y; category Z = 0 failures]
- Blocked releases since [date]: [count + reason]
## Production loop
- Path from incident → new test case: [link to process]
For more depth on how teams think about measurement and reliability, the best episodes on AI evaluation collection on Chain of Thought is a solid next read.
Checklist
- I have a one-page report with dataset provenance, including real past failures.
- A business-side owner signed the definition of good and the failure categories.
- Human or judge labels show inter-rater agreement on a documented sample.
- Precision and recall tradeoffs match what the product cannot afford to miss.
- Every change reruns the eval, and I know the exact rule that blocks release.
The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
Top comments (0)