DEV Community

Shan Liu
Shan Liu

Posted on

I benchmarked 8 LLMs for a niche production app. The flagship cost 5.8x more - and lost.

My app generates personalized readings for BaZi — Chinese "Four Pillars" birth charts. Every reading is an LLM call, every call costs money, and the domain is full of trap terminology that models love to botch. So before launch I benchmarked every candidate model on my actual workload, and then built the routing layer around what the benchmark found.

The results generalize to any "LLM in a niche domain" app, so here they are — including the part where the most expensive model lost to one costing 5.8× less.

What "good" means in a niche domain

Generic benchmarks were useless to me. My acceptance criteria were:

  1. Domain accuracy: 甲 is Yang Wood. A model that renders it "Yin Wood" in English output is not 5% wrong, it's categorically wrong — the way a compiler that flips one bit is wrong.
  2. No invented jargon: the system has a closed vocabulary (the Ten Gods, fixed star names). A model that confidently introduces terms my engine never computed is a liability.
  3. Cost per reading, because a free tier exists and every free reading is marketing spend.

Running my own corpus through the candidates produced findings no leaderboard would have surfaced:

  • The flagship preview won't let you turn thinking off. Hybrid reasoning models "think" by default, and the preview snapshot rejects the flag outright (400 InvalidParameter: The value of the enable_thinking parameter is restricted to True), so you pay for the inner monologue whether you want it or not. On one streamed request that was 286 reasoning events before the first character of the actual answer: 2.1s to the first reasoning token, 10.5s to the first character a user can read. On list price the flagship already costs 5.8× the mid-tier model per call; the reasoning tokens bill on top of that, for prose I could not tell apart. Excluded.
  • Three mid-tier models flunked domain accuracy — English chart output with elements flipped (Yang Wood → "Yin Wood" class of errors). Excluded regardless of price.
  • The "character roleplay" fine-tunes hallucinated worst of all — invented relationships between the Ten Gods that don't exist in the system. The models optimized for persona were the least safe choice for a persona product. Excluded.
  • Two well-known open-weight models had quietly been delisted from the provider's international endpoint between planning and testing. A model choice is a dependency with an EOL you don't control.

What survived: a cheap-and-accurate small model for the free tier, and a mid-tier model for paid — with the surprise that the mid-tier's previous generation was equally accurate at lower cost, which is exactly what you want in a fallback.

Price is data: keep it next to the routing

The eval's outputs — which models are allowed, in what order, at what price — live in one file. A Route is a provider (endpoint + key) plus a model plus that model's list price:

const PRICE: Record<string, [number, number]> = {
  'small-fast':   [0.1, 0.4],   // USD per 1M tokens, in/out
  'mid-plus':     [0.4, 1.6],
  'mid-plus-prev':[0.5, 3.0],
  'flagship':     [2.5, 7.5],
}

const DEFAULT_CHAINS: Record<Tier, string[]> = {
  free: ['small-fast', 'legacy-plus'],
  paid: ['mid-plus', 'mid-plus-prev', 'flagship'],
}
Enter fullscreen mode Exit fullscreen mode

Each tier gets an ordered fallback chain: the head is the workhorse, the tail is who serves the request when the workhorse can't. If a backup API key is configured, the chain ends with the primary model on the backup account — because when your account balance dies, every model on it dies together, and only a different key helps.

"The model failed" is three different problems

The subtle part of fallback chains isn't trying the next model — it's knowing when the next model helps at all. Every failure gets classified into one of three moves:

function classify(e: unknown): 'retry' | 'next' | 'fatal' {
  if (e instanceof LLMHttpError) {
    const { status, body } = e
    if (status === 401 || status === 403) return 'fatal'   // new model won't fix your key
    if (status >= 500) return 'retry'                       // transient, same route
    if (status === 429)
      return /RateQuota|rate limit/i.test(body) ? 'retry' : 'next'
    if (status === 400 || status === 404)
      return /model|not.?found|InvalidParameter/i.test(body) ? 'next' : 'fatal'
    return 'next'
  }
  return 'retry'  // network-layer: ECONNRESET, DNS, timeout
}
Enter fullscreen mode Exit fullscreen mode

The one that bites people: 429 is two different errors wearing one status code. Rate-limit throttling is transient — back off and retry the same model. Quota/allocation exhaustion is not — retrying the same model just burns time; skip to the next route. You can only tell them apart by sniffing the response body, and the distinction is provider-specific. Learn your provider's error taxonomy; it's load-bearing.

When the whole chain is exhausted, the app returns placeholder text with an ok: false flag — and the flag exists because of a real trap: never persist a fallback stub. A paid user whose reading gets cached as "(placeholder)" sees that placeholder on every revisit, forever, and the system never retries because a cached reading exists. ok gates the database write; failures stay ephemeral and self-heal on the next request.

Cost telemetry that survives fallback

Every business action (one reading = up to 7 parallel calls) emits a usage event, and each call's cost is computed against the model that actually served it, not the one you intended:

const intended = primaryModel(tier)
const fellBack = served.some((m) => m !== intended)
capture('llm_usage', {
  kind, tier, model: servedModels, primary_model: intended,
  fell_back: fellBack, input_tokens, output_tokens, cost_usd,
})
Enter fullscreen mode Exit fullscreen mode

fell_back: true is the alert condition — it means your workhorse is degraded and your margins quietly changed. With this wiring, real numbers per call (~3.7k in / 0.4k out): $0.0021 on the paid-tier model, $0.0005 on the free-tier one — so a two-call free reading lands near $0.001. Those aren't estimates; they're what the meter read.

Takeaways

  1. Benchmark on your own corpus. Leaderboards can't see that your domain has a closed vocabulary, and "most capable" models can be your worst performers on it.
  2. enable_thinking (or your provider's equivalent) is the biggest single cost lever on hybrid reasoning models — and verify each snapshot actually honors it. A preview build that rejects the flag bills you for reasoning tokens on every call, on top of an already higher list price.
  3. Classify failures before you retry. Same-model retry, next-model failover, and give-up-now are different errors sharing status codes.
  4. End the chain with a different account, not a different model. Balance exhaustion kills models in bulk.
  5. Never persist fallback output. Gate the cache write on "this is real content."
  6. Emit cost per actual served model with a fell_back flag. Silent fallback is silent margin change.

The app all this serves is auspiceoracle.com — a bilingual BaZi calculator where a deterministic engine computes the chart and the LLM is only allowed to phrase it. That constraint is its own article (next in the series).

Top comments (3)

Collapse
 
max_quimby profile image
Max Quimby

The "a model choice is a dependency with an EOL you don't control" line is the part more teams need to internalize. We've been burned by exactly this — a provider retiring an endpoint mid-project, and suddenly your carefully-tuned prompts are throwing against a model that behaves differently. Building the routing layer around the benchmark is right, but the routing layer also needs a fallback tier for the day your primary vanishes.

Your Yang Wood → "Yin Wood" example is a great illustration of why generic evals mislead: in a closed-vocabulary domain that's not an 85%-vs-90% accuracy gap, it's a categorical correctness failure, and averaged benchmarks hide it completely. We got the most signal from a tiny hand-built adversarial set that specifically probes the trap terms — 30 examples that a model either gets right or is disqualified.

Curious how you handle the free-tier model drifting over time. A cheap small model that passes today can regress silently on a provider-side update — do you re-run the domain corpus on a schedule, or only when you see complaints?

Collapse
 
shanni profile image
Shan Liu

The fallback-tier point is right, and it's the part I under-built. My chain has ordered fallbacks, but they're wired for availability — quota exhaustion, endpoint errors, a model going away. A model that's still up and answering while quietly getting worse doesn't trip anything, which is exactly the failure you're describing, and I currently don't detect it.

So, honest answer to your question: closer to "only when I see complaints" than I'd like to admit — and complaints are a terrible detector here. This domain is closed-vocabulary, so a regression looks like one stem rendered as its neighbour: a user reads a perfectly plausible sentence and never files anything. The signal never arrives.

Your 30-example disqualifying trap set is the right shape and cheap enough that there's no excuse not to run it on a schedule. Pass/fail on trap terms, not an average — averaging is what let Yang Wood → Yin Wood hide in the first place. A single categorical miss should disqualify a model outright rather than shave a decimal off a score. Adding it to the deploy checklist plus a monthly cron; if the free-tier model ever regresses silently, that's now the tripwire.

Collapse
 
realmarcuschen profile image
Marcus Chen

Criterion 2 is the one I would build on hardest, because it is the only one of your three that does not need a model to grade it.

"No invented jargon" over a closed vocabulary is a set membership test. You already compute the Ten Gods and the fixed star names, so you have the allowed set in hand, and any term in the output that is not in it is a failure you can detect with a string check on every single production call, not just on your benchmark corpus. That gives you a continuous signal at 100 percent sampling for roughly zero cost, which is a very different instrument from a periodic re-benchmark.

The reason I would bother: it is the only check in your list that keeps working when the model quietly changes underneath you. Domain accuracy needs your labelled corpus, and that corpus ages. Vocabulary violations need nothing but the vocabulary.

One trap from doing this on a different closed-vocabulary domain. Watch how the check handles a correct term inside a wrong explanation, because a clean vocabulary score can sit on top of a badly wrong reading, and it will make you trust the output more than the evidence supports.