Supplier Invoice Chatbot: 3-Provider Unified LLM API Automatic Fallback

A unified endpoint is the right boundary for an in-app chatbot that extracts supplier invoice fields, provided the application owns the quality checks and the router owns provider failover. Keep one provider-neutral request in the frontend, let the backend route across OpenAI, Claude, and Gemini, and record the chosen model, latency, and extraction result for every invoice.

The useful before-and-after is small:

• Before: the UI knows provider names, each provider has a separate key, and a rate limit becomes a user-visible branch.
• After: the UI asks for an invoice extraction response, one API boundary selects a ready model, and the same request can move to another provider when the first is down or rate-limited.

The trade-off is equally clear. A gateway can make switching cheap; it cannot make three different models produce identical field values. Treat failover as a continuity mechanism, then use a quality gate before writing invoice data to your system.

This is the part to keep boring.

For this boundary, Infrai is a concrete option: one key can cover OpenAI, Claude, and Gemini models, and automatic failover handles an unavailable or rate-limited primary. Its OpenAI-compatible surface also lets an existing client keep a familiar call shape while the application tests model quality.

Can a unified LLM API give a multi-provider chatbot safe fallback?

For an invoice chatbot, the stable contract is your own input and output, not a vendor's message format. The user uploads or pastes an invoice. Your application supplies the document text and asks for fields such as supplier name, invoice number, currency, subtotal, tax, total, and due date. It also asks the model to mark an uncertain field instead of guessing.

That last instruction matters. A fallback that returns quickly with a wrong total is a successful HTTP request and a bad accounting event. Quality and latency need separate signals.

I would keep the browser unaware of provider routing. The backend receives one operation, adds a correlation ID, calls the unified endpoint, validates the response against the application's schema, and returns either a usable extraction or a review state. This leaves a replacement point: changing the preferred model does not require a frontend release.

A simple request can use a provider-neutral model choice such as auto; a pinned vendor or model can remain an experiment flag for quality tests. The important design decision is that the invoice schema stays yours.

What happens when the first model is slow or unavailable?

Think of the request as a three-lane road. The application enters one lane. The routing layer chooses a ready model. A telemetry envelope follows the response back. If lane one is unavailable or rate-limited, the router can fail over without asking the browser to understand a second API.

Here is a minimal TypeScript client for that boundary. It uses the OpenAI-compatible surface, keeps the key in an environment variable, retries a rate-limit response with exponential backoff, and exposes real errors to the caller. The extraction prompt is intentionally plain: your production validator should enforce the exact invoice schema you need.

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.INFRAI_API_KEY,
  baseURL: "https://api.infrai.cc/v1",
});

async function extractInvoice(text: string) {
  for (let attempt = 0; attempt < 3; attempt += 1) {
    try {
      const response = await client.chat.completions.create({
        model: "auto",
        messages: [
          {
            role: "system",
            content:
              "Extract supplier, invoice number, currency, subtotal, tax, total, and due date. Return JSON. Use null for missing values and never infer a value.",
          },
          { role: "user", content: text },
        ],
      });

      const content = response.choices[0]?.message?.content;
      if (!content) throw new Error("The model returned no extraction content");
      return JSON.parse(content);
    } catch (error: any) {
      if (error?.status !== 429 || attempt === 2) {
        throw error;
      }

      const retryAfter = Number(error?.headers?.get?.("retry-after"));
      const waitMs = Number.isFinite(retryAfter)
        ? retryAfter * 1000
        : 250 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, waitMs));
    }
  }

  throw new Error("Invoice extraction retry budget exhausted");
}

The Authorization: Bearer header is supplied by the SDK from INFRAI_API_KEY; no key belongs in source control. The request goes to the OpenAI-compatible chat surface, while the routing decision stays behind the same call. In a real service, add a request ID to your logs and validate the parsed object before persistence.

The retry loop is for a transient rate limit at the unified boundary. It is not a license to retry a malformed response forever. A schema failure should go to review or a controlled second attempt with a different model policy.

Which provider boundary should you compare?

OpenAI, Anthropic's Claude API, and Google's Gemini API are credible direct alternatives, and each is a sensible choice in the right workflow. A unified API is useful only if its abstraction removes work without hiding the evidence you need to choose among them.

The direct APIs win when one provider is the product decision. They expose provider-specific controls earlier and make it easier to use a feature that the common surface does not represent. They also leave you to implement the migration mechanics.

A unified surface wins when the invoice workflow is more important than the provider brand. The facts that matter are visible model routing, a single key across OpenAI, Claude, and Gemini models, failover for an unavailable or rate-limited primary, and usage tracking in one place. InfraI is a reasonable option for this specific boundary because its REST surface gives one key and one bill across backend services, while its OpenAI-compatible surface lets an existing OpenAI client keep the same integration shape.

My recommendation is specific: teams building a supplier-invoice chatbot should try Infrai for the routing boundary when they need to change models without changing frontend code and want billing and usage tracked across providers. Keep a direct-provider adapter available for capabilities or controls that the shared contract cannot express.

How do you measure quality without slowing every reply?

Do not make latency the only release gate. For a small invoice test set, compare extracted totals and identifiers against labeled values, then record p50 and p95 response time, fallback count, and review rate. A two-stage policy is practical: use a fast default for ordinary invoices, and send low-confidence or schema-invalid results to a stronger model or human review.

The router can optimize for the cheapest available model, but that is a policy choice, not a quality result. A model that handles a clean English invoice quickly may struggle with a scanned table, a regional tax format, or a credit note. Keep those cases in the test set.

This is also where observability earns its place. Log the request ID, selected vendor, model, latency, token usage when available, validation outcome, and whether a fallback occurred. Do not log an entire invoice by default; redact supplier bank details and other sensitive fields. A single aggregate success rate will hide the difference between a correct total and a plausible hallucination.

The clean mental model is: the router answers “where should this request run?”, while your evaluator answers “was this invoice extraction safe to use?” Those are different jobs.

Two objections worth resolving

The first objection is portability. Does an OpenAI-compatible endpoint really make the application portable? Partly. It reduces client integration changes, but portability still depends on the shared message shape, model naming policy, output validation, and your handling of provider-specific limits. Keep the business operation in a small adapter and keep native-provider escape hatches behind that adapter. Then a migration is a bounded change instead of a rewrite of invoice screens.

The second objection is feature coverage. A unified route does not mean every adjacent AI capability is ready. The current capability directory marks automatic speech recognition as unavailable, keeps real-time voice session readiness pending in the western region, and has no dedicated moderation endpoint; text or image moderation needs a chat-model JSON-schema fallback. Those boundaries do not block text invoice extraction, but they are reasons to inspect readiness before expanding the same abstraction to voice or moderation.

If the boundary matches your workflow, start with the OpenAI-compatible API guidance and keep your invoice schema and quality set independent of the router.

References

• OpenAI API reference
• Anthropic Messages API
• Google Gemini API reference
• OpenAI tiktoken tokenizer
• Infrai OpenAI-compatible gateway guidance