A normal unit test checks that a function returns an exact value. AI is non-deterministic: editing a system prompt, changing the temperature or pointing at a newer model can quietly reduce accuracy, add hallucinations or break the JSON your code depends on, and no exact-match test will notice. MLOps closes the gap by putting evaluation into the Git workflow.

In plain terms: A restaurant does not taste-test every dish by checking its weight. It has a tasting panel with a score sheet, and a new recipe only goes on the menu if the score does not drop.

1. The evaluation pipeline

Any pull request that touches prompt templates, RAG indexing or agent tools should trigger automatic checks before it can merge.

• Schema and lint tests. Assert that outputs follow the declared JSON schema and regex rules across a set of test inputs.
• Gold-standard suite. Run the pipeline against a version-controlled set of real edge cases and compare the pass rate with the last release.
• LLM-as-judge. Use a separate evaluator model to grade faithfulness, relevance and use of context, and flag a drop below the baseline.

Diagram: Any pull request that changes prompt templates, RAG indexing or agent tooling must pass automated checks before merging: schema and regex tests, a gold-standard test suite, and an LLM judge scoring faithfulness and relevance. See the animated version.

Diagram: Running a version-controlled gold-standard test suite on every change gives a pass-rate trend. A prompt change that drops the score below the agreed baseline is flagged before it ships. The numbers are illustrative. See the animated version.

2. GitOps and automated rollout

Release through GitOps, with a tool such as ArgoCD.

• Declarative infrastructure. Model settings, image tags and replica counts live as manifests in a Git repository. ArgoCD keeps the cluster in line with that repository, so a rollback is a revert of one commit.
• Canary rollouts. A new model version first receives a small share of real traffic, for example 5%, while error rate and latency are watched. If the numbers hold, the share grows step by step to 100%, and if a threshold is crossed the rollout reverts on its own. In the Argo ecosystem the canary itself is run by Argo Rollouts, which works alongside ArgoCD.

Diagram: In GitOps the model version, image tag and replica limits are declared in a Git repository, and ArgoCD keeps the cluster in sync with it, pulling drifted resources back. A rollback is a revert of one commit. See the animated version.

Diagram: A new model version receives a small share of traffic first, for example 5%, while error rate and latency are watched. If metrics stay healthy the share grows step by step to 100%, and if a threshold is exceeded the rollout reverts automatically. ArgoCD syncs the manifests, and Argo Rollouts performs the canary. See the animated version.

Below are the two main pieces. They assume a tests/evals/ folder with your gold cases, a helper called run_pipeline, and a judge model you provide. Treat the thresholds as starting points to tune.

# tests/evals/test_quality.py  (run by CI with: pytest tests/evals)
import json
from pipeline import run_pipeline          # your RAG or agent entry point
from judge import faithfulness_score       # calls an isolated judge model

BASELINE = 0.88                            # pass rate of the currently deployed version
CASES = json.load(open("tests/evals/gold_cases.json"))

def test_outputs_match_schema():
    for case in CASES:
        out = run_pipeline(case["input"])
        assert set(out) >= {"answer", "sources"}           # schema check

def test_quality_not_below_baseline():
    passed = sum(faithfulness_score(c, run_pipeline(c["input"])) >= 0.8 for c in CASES)
    assert passed / len(CASES) >= BASELINE - 0.02          # small tolerance for judge noise

# Argo Rollouts: a canary that shifts traffic in steps
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: model-api
spec:
  replicas: 4
  selector:
    matchLabels: { app: model-api }
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 10m }      # watch error rate and latency here
        - setWeight: 25
        - pause: { duration: 10m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100
  template:
    metadata:
      labels: { app: model-api }
    spec:
      containers:
        - name: model-api
          image: registry.example.com/model-api:1.5.0

For automatic rollback you attach an analysis that queries your metrics and fails the rollout when they cross a threshold. Automating testing and release turns AI deployments from manual experiments into repeatable, verifiable engineering.

Coming Up Next

Day 26: Zero-trust security for AI: sandboxing ephemeral execution and model egress.

MLOps #CICD #GitOps #ArgoCD #PlatformEngineering

Originally published at https://sureshpallapothu.in/blog/day-25-ai-cicd, where this post includes animated diagrams.