Six months ago, we shipped our first LLM-powered feature the way we ship everything else: pull request, code review, green CI, deploy on a Friday afternoon (yes, I know).
By Monday, support tickets were stacking up. Nothing had crashed, no alerts had fired, and the service had returned 200s all weekend. The answers were quietly getting worse.
That weekend taught us something uncomfortable. Our DevOps setup was built to answer “is the service up?”, while AI features need it to answer “is the service right?” Those are very different questions.
This post covers the pipeline we built to answer the second one, what we got wrong along the way, and whether I think AI work really needs its own flavor of DevOps.
Why our existing pipeline wasn’t enough
Our team works as an AI development agency, which means we ship across several client codebases with different models, data, and risk tolerances. That variety exposed the gaps fast. Traditional DevOps quietly assumes a few things that stop being true:
- Deterministic behavior: Same input, same output. With LLMs, the same input can produce five slightly different answers, and “correct” is a spectrum.
- Code is the main artifact: In AI features, the prompt, model version, retrieval config, and underlying data matter as much as the code, and changing any of them changes behavior.
- Failures are loud: A broken endpoint pages someone. A degraded answer pages no one.
- Cost is predictable: With LLMs, a single prompt change can double your token spend overnight.
None of this means DevOps is wrong. It means the definition of done has to expand.
What we built
We ended up with five layers. Nothing exotic, mostly existing tools wired together with a different mindset.
1. Prompts and configs as versioned artifacts
Every prompt, model name, temperature, and retrieval setting lives in the repo as config, not as a string buried in a function. Each release gets a version tag, and every request logs which version served it.
# ai-config/support-bot.yaml
version: 2.4.1
model: primary-model-v3
temperature: 0.2
prompt_file: prompts/support_answer.md
retrieval:
top_k: 5
min_score: 0.72
fallback_model: small-model-v2
When quality drops, the first question is “what changed?”, and now we can actually answer it.
2. Eval gates in CI
This was the biggest shift. For each feature, we maintain a golden set of 150 to 300 real examples, scored by a mix of exact checks, rule-based checks, and a model-graded rubric. Every PR that touches a prompt, config, or retrieval code runs the suite.
# .github/workflows/ai-evals.yml (simplified)
- name: Run eval suite
run: python evals/run.py --suite support-bot --min-score 0.88 --max-regression 0.03
A PR fails if the overall score drops below the threshold, or if any category regresses by more than three points. It’s like unit tests for judgment. It’s imperfect, but it catches the dumb regressions before users do.
3. Shadow mode and staged rollouts
New versions first run in shadow mode: they receive real traffic and generate answers, but users only see the current version. We compare outputs offline, then roll out to 5%, 25%, and 100%, with automatic rollback if quality or cost metrics cross a line.
4. Observability beyond uptime
Alongside latency and error rate, we track:
- Cost per request and per user session
- Retrieval quality (did we fetch relevant context at all?)
- Refusal and fallback rates
- User signals: thumbs down, retries, and “talk to a human” clicks.
- Sampled traces reviewed weekly by a real person
The weekly human review is low-tech and probably the most valuable item on the list. Dashboards tell you that something drifted. Reading 30 real conversations tells you why.
5. Kill switches and fallbacks
Every AI feature has a feature flag that turns it off in seconds, plus a graceful fallback: a smaller model, a cached answer, or a plain non-AI path. We rehearse flipping the switch the same way you’d rehearse a failover.
What we got wrong
- We over-trusted model-graded evals: Our first rubric scored answers on “helpfulness,” and the grader loved long, confident responses. Scores climbed while users got more annoyed. We fixed it by adding rule-based checks and refreshing the golden set with real failures every month.
- We built too much before measuring: Our first version had a custom dashboard nobody opened. Now we start with logs and a spreadsheet, and only build tooling after we’ve felt the pain twice.
- We ignored data drift: Retrieval quality decayed as clients’ documents changed. Nothing in our code had changed, but the system got worse anyway.
A friend who works as an AI consultant put it bluntly when we showed her the first version: “You’ve built CI for code. Where’s CI for reality?” That comment led directly to scheduled eval runs, not just PR-triggered ones, and to the failure-driven golden set.
The pipeline’s first real save
Two months later, a client updated their pricing docs. Our scheduled eval run flagged a 9-point drop in the billing category within hours. The cause: retrieval was pulling the old and new pricing pages together, and the model blended them into a confident, wrong answer. We pinned the document version, re-indexed, and the score recovered the same day. Under our old setup, we’d have found out from an angry customer.
So, does AI development need its own DevOps?
My honest answer: it needs the same principles with a wider scope.
The core of DevOps (short feedback loops, automation, shared ownership, observability) still applies. What changes is what you version, what you test, and what you monitor:
| Traditional DevOps | AI-Flavored DevOps |
|---|---|
| Version the code | Version code, prompts, models, and data configs |
| Unit and integration tests | Evals with thresholds and regression checks |
| Monitor uptime and latency | Monitor quality, cost, and drift |
| Roll back a bad deploy | Roll back a bad deploy or bad model behavior |
| Postmortem on outages | Postmortem on quietly wrong answers |
Call it LLMOps, MLOps, or “DevOps done properly for probabilistic systems,” I don’t mind. The label matters less than the habit: never let “it returned a 200” be your definition of working.
If you’re starting from zero
You don’t need our whole stack. Start here:
- Put your prompts and model settings in version control.
- Build a 50-example eval set from real inputs, even if it’s just a spreadsheet.
- Add a feature flag so you can turn the feature off.
- Log cost per request from day one.
- Read 20 real outputs every week.
Over to you
Does your team treat AI features like regular services, or have you built something custom? I’m especially curious how others handle stale eval sets, and whether anyone has found a model-graded rubric they actually trust. Drop your setup in the comments; I’d love to compare notes.
Top comments (2)
The “200s all weekend while answers quietly got worse” line is exactly why uptime dashboards aren’t enough for LLM features.
Two things stood out:
The pricing-doc catch is the strongest part of the post. The code didn’t change, so a PR-only gate would never have seen it. Scheduled eval runs are what made the difference. It also shows that retrieval data is an input that needs versioning like code, which your “pin the document version” fix points to.
On your question about stale eval sets and trusting rubrics: the failure-driven golden set seems like the right approach. Two small additions that might help:
The weekly “read 20-30 real conversations” habit is underrated, and I’d bet most teams skip it. Thanks for being open about what went wrong, not just what worked.
You’re right about the pricing-doc catch. It’s the case that convinced the rest of the team, because nothing in the code had changed and only the scheduled run caught it. Treating retrieval data as a versioned input, like code, is probably the biggest mindset change we made.
The frozen anchor set is a good idea, and we don’t do it yet. Right now when a score moves after a monthly refresh, we can’t always tell whether the system got worse or the test set got harder. A small set that never changes would answer that. I’m adding it to our list.
On grader bias, we do a version of what you describe. We check the model grader against human labels on a small sample, and we’ve moved toward narrower pass/fail criteria (right doc cited, right price stated) over a single “helpfulness” score. I haven’t tried pairwise comparisons properly, so I’ll give them a go. If you’ve used them in production, I’d like to know how you handle position bias.
And yes, the weekly read-through is the least glamorous item on the list and the one I’d keep last.
Thanks for the thoughtful notes!