I love the DSPy idea. You stop hand-editing prompts and let the compiler optimize them for you against a metric and a training set. I built a RAG pipeline that way, the compile score came out high, the hold-out looked fine, and I shipped it.
Live traffic quality was worse than what I had before. Nothing errored. The numbers I had been watching just did not match what users were getting.
The problem was not DSPy. It was what I was optimizing against and what I was measuring. Here is the short version of what I learned.
The metric you compile against is not your production rubric
DSPy optimizes your prompts against a metric you give the compiler. That metric has to be cheap, because the compiler scores thousands of trial prompts in a single pass. So in practice it is something thin: does the final answer contain the expected string, a single yes-or-no judge call, that kind of thing.
But your product is not judged on a thin metric. A real RAG answer needs to be grounded in what was retrieved, complete on multi-part questions, and willing to refuse when the answer is not there. None of that fits into the cheap check the compiler runs five thousand times.
So when the cheap compile metric and the real rubric disagree, the compiler happily overfits the cheap one. It gets very good at the thing you told it to measure, which was never quite the thing you actually cared about. That is exactly how a pipeline scores high at compile time and ships worse.
The fix is not a cleverer cheap metric. It is to keep the cheap metric where it belongs, inside the compile loop, and run the real, rich rubric separately, on your hold-out set and on live traffic.
Score the Signature, not the whole program
The second thing I got wrong: I scored the pipeline end to end, one number for the whole thing.
A DSPy program is a few modules chained together, each with its own little job. Mine had a retrieval step and an answer step. When the single end-to-end score dropped, it told me the program got worse and absolutely nothing about which step caused it. I was left guessing, and I guessed wrong for a while.
What actually works is scoring each module against its own job:
The retrieval step gets judged on whether it fetched the right material.
The answer step gets judged on whether it stayed grounded in that material and answered the whole question.
Now when the program regresses, the module scores point straight at the culprit. If retrieval tanks while the answer step holds, I know the compiler produced a bad retrieval prompt and I fix that one. The end-to-end number could never tell me that.
Every module can pass and the program still fail
Here is the sneaky one. Sometimes each module scores fine on its own and the program is still wrong, because the composition lost something between the steps. Retrieval fetched the right passages. The answer step reasoned fine over what it got. The final answer was still off.
So I also keep one check that looks at the whole run and asks, when the final answer is wrong, which step was the proximate cause. Run that across a handful of failing cases and you get an actual distribution instead of a hunch. If most of the failures trace back to retrieval, you fix retrieval first. If they trace back to the answer step despite it scoring well in isolation, that is your real weak link.
What I do now, in one line
Compile with the cheap metric, but never trust it as the verdict. Judge each module on its own job with the real rubric, keep one check for where the cascade breaks, and run all of it on a fresh hold-out and a slice of live traffic, not just on the set the compiler already saw. The moment I split the score up that way, the regression that had been invisible was obvious.
If you want the deeper version, with the exact per-module rubrics and how to wire the compile-versus-production comparison into CI, this piece goes through it step by step.
If you run DSPy in production, I am curious whether your compile scores and your live scores ever drifted apart. Mine did, quietly, and the gap was the whole lesson.
Top comments (2)
"The compiler overfits the cheap metric" is Goodhart stated in one sentence, and it's the failure mode nobody prices in when they adopt an optimizer. The cheap proxy has to be cheap because it runs thousands of times — so it's structurally guaranteed to be a worse target than your real rubric. The optimizer then does its job too well against the wrong thing.
Your per-Signature scoring point is the fix I'd underline hardest. An end-to-end number is a lagging indicator that tells you that something regressed and nothing about where. Scoring retrieval and answer separately turns "the program got worse" into "the compiler wrote a bad retrieval prompt," which is actionable.
The compositional-failure case (every module passes, program still fails) is the sneaky one — it usually means the modules are locally optimal but the interface between them lost information, e.g. retrieval returns the right doc but in a form the answer step under-weights. Did you end up adding an eval that scores the trajectory (what got passed between steps), not just the endpoints? That's the only thing that ever caught interface bugs for me.
BLEU was our mistake too, and it's basically the worst metric for this because it rewards string overlap without caring whether the answer's grounded in what was actually retrieved. The per-module scoring point is what we should've set up from the start, because when our end-to-end number dropped we assumed it was the answer step and it wasn't. Spent three days on the wrong part of the pipeline. Eventually we just scored retrieval separately on a small eval set and found the actual problem in about an hour.