The people who know your AI agent best may be the people least able to see all of its flaws.
Not because they are bad engineers.
Because they built it.
Years ago, when I was taking art classes, my teacher told me something I've never forgotten:
“Sara, you can't judge your own art.”
I remember thinking, of course I can. 😂
Then she explained.
After spending hours looking at the same piece, your eyes get filled with it. You stop seeing what is actually there. You see what you expect to see.
I've used that lesson everywhere since.
And I think AI agents have the same problem.
You designed the requirements.
You designed the system.
You know why every decision was made.
Then you design the evaluation and ask:
“Does my agent actually work?”
That's where the blind spot can appear.
Your evaluation may end up testing the system according to the same assumptions that created it.
The evaluator can inherit the system's assumptions
Consider a simple requirement:
“The agent should answer customer questions accurately.”
Seems reasonable.
So the team creates an evaluation set with questions that have clear intent and well-defined answers.
The agent performs beautifully.
94%.
Green dashboard. 🎉
But an external evaluator might ask a different question:
What happens when the customer's request has two plausible interpretations?
Now you have a different test:
“Can I change my billing address?”
Does the agent answer immediately?
Does it ask which account or address the customer means?
Does it make an assumption?
The original evaluation may have been technically correct.
It just never tested the ambiguity.
That is the blind spot.
Internal evaluation is still essential
This isn't an argument that internal teams shouldn't evaluate their own systems.
They absolutely should.
The people who built the system understand its requirements, architecture, constraints, tools, and intended behavior better than anyone.
That knowledge is extremely valuable when designing evaluations.
But it can also create an invisible constraint:
You know what the system is supposed to do, so you naturally tend to test within the boundaries you already understand.
An independent evaluator brings a different mental model.
Not necessarily better technical knowledge.
A different set of assumptions.
Someone who can ask:
- What did we assume here?
- What happens if the requirement is ambiguous?
- What happens at the edge?
- What did we forget to test?
- Which behaviors are we treating as acceptable without actually defining why?
- What if the system is doing exactly what we designed, but what we designed was wrong?
That last question is the uncomfortable one.
The dangerous evaluation is the one that confirms everything you already believe
An evaluation isn't only a collection of tests.
It is also a model of what “working” means.
If the same people define the requirements, design the system, choose the test cases, define the rubric, and interpret the results, there is a risk that the entire evaluation inherits the same assumptions.
Everything can look internally consistent.
And still be wrong.
This is why I think evaluation independence deserves more attention as AI agents become more capable.
You don't necessarily need an external evaluator for every test.
But you do need some mechanism that is independent of the assumptions being evaluated.
That could mean an external evaluator.
It could mean a separate team.
It could mean adversarial test design.
It could mean deliberately asking someone unfamiliar with the implementation to construct edge cases.
The mechanism can vary.
The principle doesn't:
The evaluation should be capable of challenging the assumptions behind the system, not merely confirming that the system behaves according to them.
Sometimes the hardest failure to find is the one everyone involved has learned not to see.
Internal testing is necessary.
But sometimes, you need someone who hasn't spent hours staring at the same painting.
Top comments (4)
The billing-address example nails it: the eval inherits the same assumptions that built the agent, so ambiguity never gets tested. What helped me was sourcing eval cases from real production transcripts rather than writing them myself, since the messy inputs I'd never think to author are where the agent quietly guesses. Do you have an outside reviewer write cases, or mine them from live traffic?
Exactly. I’d use both, but for different jobs. Real production transcripts are probably the best source for discovering cases I would never think to write myself, especially ambiguity and unexpected assumptions. An outside reviewer is useful for deliberately challenging the assumptions that still survive the production data. So I’d treat live traffic as the discovery layer, and independent evaluation as the challenge layer. The combination is much harder for the eval to become an echo of the system it is testing.
Ran into a version of this with a support agent eval: the test set was built from the same ticket taxonomy the routing logic used, so every question fit the buckets the system already expected. It scored high until we added cases where a ticket matched two categories or none, and logged which assumption the agent acted on. The cheapest independence mechanism I've found is writing test cases from user logs instead of the requirements doc. The doc encodes your assumptions; the logs don't. One question on the rubric side: if the external evaluator still scores against a rubric written by the builders, doesn't the blind spot just move there?
Yes. That’s exactly where I’d put the next layer of scrutiny. An external evaluator does not automatically make an evaluation independent if the rubric already encodes the builders’ assumptions.
I’d separate two things: evaluating against the stated requirements, and challenging whether those requirements are sufficient. The first can use the builder’s rubric. The second needs an independent challenge to the rubric itself, using production evidence, ambiguous cases, edge cases, and failure modes the original team did not anticipate.
So I think the real independence question isn’t only “Who evaluates the agent?” It’s also “Who gets to challenge what we decided counts as correct?”
Your ticket-taxonomy example is exactly the kind of thing that can expose that second layer.