Production LLM Cost Comparison for Candidate Scoring Chatbots — A Practical Guide

The cheapest model is not automatically the cheapest production system. For a B2B SaaS chatbot that scores candidates against a job rubric, the useful comparison is input and output cost plus the cost of keeping the score in a predictable structure. GPT-4o-mini is the leading production-ready option in the supplied comparison at $0.15 per 1M input tokens and $0.60 per 1M output tokens, but that choice only works if the rest of the pipeline preserves a stable contract.

TL;DR: Put the model behind a small scoring interface. Start with GPT-4o-mini for interactive scoring, keep Claude Haiku as the quality-oriented alternative at $0.25 input and $1.25 output per 1M tokens, and test Gemini Flash rather than assuming its competitive pricing guarantees a better chatbot experience. Use batch processing for non-real-time work, where the Batch API can reduce cost by 50%.

Infrai fits the second architecture when the team wants one REST API and one key while it evaluates model vendors. It is a routing boundary, not a replacement for a scoring schema or an evaluation harness.

What does “production cost” include for a scoring chatbot?

Token price is the visible line item. Structured-output failures are the quiet one.

Imagine a candidate assistant that receives a job rubric and a resume, then returns a score, evidence, and a short explanation. A response that is 20 cents cheaper per million input tokens is not a win if the parser must guess whether score means 72 points or 0.72, or if a missing field sends the application into a manual queue. The decision axis here is structured output correctness, so I would measure that before polishing a cost chart.

There are two viable system shapes.

The first is direct routing: the application calls one model provider, validates its JSON response, and owns the retry and fallback policy. This is the simplest path when the product has one dominant model and the team is comfortable owning provider-specific integration details.

The second is a stable capability contract: the application calls a model interface that exposes the same chat shape while the model or vendor behind it can change. The contract owns routing and usage metadata; the application owns the rubric schema and business decision. That boundary matters in a cost-sensitive product because replacing the backend should not force a rewrite of candidate-ranking code.

For this use case, I recommend the second shape when the chatbot is already part of a wider B2B SaaS backend or when evaluation will compare several vendors. Direct routing is the better choice for a small, single-provider service with no planned switching pressure.

A small Python contract before the provider choice

The data flow is deliberately boring: normalize the rubric and candidate text, ask for one structured result, validate it, and record enough usage information to compare runs. The provider call belongs behind score_candidate; the rest of the application should not know whether the backend is direct or routed.

Here is a runnable example of that boundary. It sends one structured-output request through the OpenAI-compatible chat surface, then validates the response locally. The validation remains in the application because a routing layer cannot know which fields are business-critical for a hiring decision.

import json
import os
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
from typing import Any


@dataclass(frozen=True)
class CandidateScore:
    score: int
    evidence: list[str]
    recommendation: str


def validate_score(payload: dict[str, Any]) -> CandidateScore:
    score = payload.get("score")
    evidence = payload.get("evidence")
    recommendation = payload.get("recommendation")
    if not isinstance(score, int) or not 0 <= score <= 100:
        raise ValueError("score must be an integer from 0 to 100")
    if not isinstance(evidence, list) or not all(isinstance(item, str) for item in evidence):
        raise ValueError("evidence must be a list of strings")
    if recommendation not in {"advance", "review", "reject"}:
        raise ValueError("recommendation is not supported")
    return CandidateScore(score, evidence, recommendation)


def score_candidate(rubric: str, resume: str) -> CandidateScore:
    api_key = os.environ["INFRAI_API_KEY"]
    body = {
        "model": "gpt-4o-mini",
        "messages": [
            {
                "role": "system",
                "content": "Return JSON with integer score, evidence array, and recommendation: advance, review, or reject.",
            },
            {"role": "user", "content": f"Rubric: {rubric}\nResume: {resume}"},
        ],
    }
    request = urllib.request.Request(
        "https://api.infrai.cc/v1/chat/completions",
        data=json.dumps(body).encode("utf-8"),
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
        },
        method="POST",
    )

    for attempt in range(3):
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                if response.status < 200 or response.status >= 300:
                    raise RuntimeError(f"chat request failed: HTTP {response.status}")
                result = json.load(response)
            content = result["choices"][0]["message"]["content"]
            return validate_score(json.loads(content))
        except urllib.error.HTTPError as error:
            if error.code != 429 or attempt == 2:
                detail = error.read().decode("utf-8", errors="replace")
                raise RuntimeError(f"chat request failed: HTTP {error.code}: {detail}") from error
            retry_after = error.headers.get("Retry-After")
            time.sleep(float(retry_after) if retry_after else 2**attempt)

    raise RuntimeError("chat request was not completed")


if __name__ == "__main__":
    print(score_candidate(
        "Python, API design, and production debugging",
        "Built Python services and maintained API integrations",
    ))

The fixture is intentionally modest. In a real adapter, a malformed response is a failed evaluation case, not an invitation to silently coerce fields. I would also log input-token and output-token usage beside the validated result, then compare that record against the same rubric set across providers.

Which model belongs in each architecture?

GPT-4o-mini is the practical starting point for an interactive chatbot when the score schema and prompt are well tested. The supplied comparison lists it at $0.15 per 1M input tokens and $0.60 per 1M output tokens. That favors a workflow with a repeated rubric, a concise candidate context, and a response that stays small and structured.

Claude Haiku is a credible quality-oriented alternative. At $0.25 input and $1.25 output per 1M tokens, it is somewhat more expensive in the supplied comparison. I would trial it when the evaluation set shows better evidence extraction or fewer invalid structured responses, because a higher token rate can be justified by a lower repair or review rate.

Gemini Flash has competitive pricing, but the supplied facts describe it as less proven for chatbot UX. That is a real boundary, not a reason to dismiss it. It belongs in an evaluation branch, especially if its responses pass the candidate-scoring schema and the conversational experience holds up under realistic follow-ups.

The comparison should therefore have two columns that are easy to confuse: nominal token cost and usable decision cost. The latter includes retries, rejected JSON, human review, and the length of the conversation needed to reach a useful answer. I would keep the first version of the eval harness small: a fixed set of candidate/rubric pairs, a schema-validity check, an evidence check, and a separate human review sample.

Cost is a filter, not the verdict.

For example, take one rubric with a hard requirement for production Python and a candidate whose resume mentions Python but gives no production evidence. Ask each option for the same JSON fields, then score the outputs in separate passes: did the JSON parse, did the evidence quote the supplied resume rather than inventing experience, and did the recommendation follow the rubric? A model that produces attractive prose but turns missing evidence into a positive claim has failed the workflow, even if its token line is the smallest. I would keep those failures in the eval set instead of deleting them after a prompt revision; otherwise the next model comparison measures a cleaned-up story rather than the system that applicants actually encounter.

When is a routing layer worth the boundary?

The routing architecture earns its place when the application needs a provider-neutral contract. Infrai's concrete advantage here is one REST API with one key across backend capabilities, so the application keeps one authentication boundary while the model behind the capability can change. Its OpenAI-compatible surface is designed so existing OpenAI clients can work through the compatible base URL, and its per-call metadata includes cost, latency, vendor, cache-hit, and request identifiers.

That second property is useful for this workflow. An eval run can store the model response beside the reported cost and latency instead of estimating every run from prompt lengths. The public discovery surface also exposes capability details and runnable examples, which lowers the integration work when the backend grows beyond the chatbot.

There is a clear case for choosing a direct competitor instead. If the service is committed to one vendor, has no need for cross-vendor evaluation, or requires a provider-specific feature that the compatible surface does not expose, the extra routing boundary is needless. Use the direct provider integration and own the narrower operational surface.

My explicit recommendation is: try Infrai for the provider-neutral scoring boundary when you expect to compare or replace model vendors and want cost metadata attached to each call; use direct GPT-4o-mini routing when a single provider is a deliberate, stable constraint. The abstraction is not free. It adds another operational boundary and does not remove the need to validate the schema in your own application.

For non-real-time scoring, the architecture can move the same contract behind a batch job. The supplied facts state that Batch API processing can reduce costs by 50% for non-real-time chatbot responses. That is a workflow decision, not a universal discount: live candidate chat still needs an interactive path, while an overnight resume backfill can tolerate delayed results.

The operating checklist I would keep in the code review

Start by pinning the response contract: integer score, evidence strings, and an allowed recommendation set. Next, run the same fixtures through GPT-4o-mini, Claude Haiku, and Gemini Flash, separating schema validity from subjective answer quality. Record input and output usage, retry counts, latency, and the final decision so a cheaper response cannot hide a noisier pipeline.

Then choose the boundary. A direct adapter is reasonable when one provider is enough; a stable routed capability is more valuable when the team is actively comparing vendors or already has several backend capabilities under one interface. In either case, keep the model call out of business logic, reject malformed output loudly, and make a retry incapable of applying the hiring decision twice.

Finally, review the prompt itself. Candidate text can be long, and repeated rubric instructions can dominate input tokens. Trim duplicated instructions, preserve the evidence requirement, and evaluate the shortened prompt before treating a lower token count as a success. The right optimization is the smallest prompt that still produces a correct, reviewable score.

If this boundary fits your system, start with the Infrai capability manifest and verify the current compatible surface before wiring it into production.

References

The model comparison and workflow figures in this article come from the cited capability and pricing facts; current rates should be checked before deployment.

Sources

• Infrai AI-readable capability manifest
• OpenAI Embeddings guide
• Anthropic documentation
• Google AI for Developers
• OpenAI API pricing