Once you put a large language model behind a company API gateway, someone asks a simple question.
Who is paying for this?
The model bill arrives as one number. Finance sees one number. Every team in the company sees one endpoint. Nobody can tell you which team burned forty percent of the budget, which use case is growing, or which experiment someone forgot to shut off six weeks ago. That gap between one bill and one endpoint is where metering lives.
We built metering into our LLM gateway before we scaled usage, and it changed how teams behaved. Here is what we learned.
Why LLM metering is not like API metering
With a normal REST API, metering is easy. One request costs one request. You count requests per subscription key, and you are done.
LLM calls do not work that way. One request to a chat endpoint can cost five tokens or five thousand. A prompt injection test script someone ran in a loop overnight can spend more than an entire product team's monthly budget in eight hours. If you count requests, you learn nothing. The unit of cost is the token, and the token count lives in the response, not the request.
That detail shapes everything. Your metering code has to read the response body, extract the usage block, and attribute it before the data is gone.
{
"usage": {
"prompt_tokens": 1240,
"completion_tokens": 312,
"total_tokens": 1552
}
}
Azure OpenAI returns this on every response. If you do not capture it at the gateway, you have to reconstruct it later from logs, and that is a losing fight.
Capture it at the gateway, not in the app
Every team wanted to do metering in their own application code. We said no. The gateway is the one layer every call passes through, regardless of language, framework, or team. A metering implementation that depends on each team instrumenting their own service is an implementation that covers sixty percent of traffic on a good month.
We put a policy on the gateway's inbound path to stamp each request with metadata: the subscription the key belongs to, the product, the environment. On the outbound path, another policy parses the response, pulls the token counts, and ships an event to our analytics pipeline. The policy does not touch the request or response bodies in ways the caller can see. It observes and reports.
<policies>
<outbound>
<choose>
<when condition="@(context.Response.StatusCode == 200)">
<set-body>@{
// parse the usage block and forward a metering event
// key: subscription id, product, environment, model
// value: prompt_tokens, completion_tokens, latency
}</set-body>
</when>
</choose>
</outbound>
</policies>
This is pseudo-code, not a real policy file, because every gateway has its own syntax. The shape matters more than the syntax: stamp metadata on the way in, extract cost on the way out, send it somewhere durable.
Attribute everything, including the failures
The first version of our metering only counted successful calls. That lasted about two weeks. Someone ran a batch job against a deprecated model deployment that returned 400s all day. The calls were cheap individually, but there were hundreds of thousands of them, and the model provider still charged for the input tokens on each failed request.
Failed requests still consume input tokens. Rate-limited requests still consume them. Requests that hit a content filter still consume them. If you only meter the 200s, your bill and your dashboard will disagree, and finance will believe the bill.
Our outbound policy records every response code. Failures get their own bucket so teams can see them separately, but they are not excluded from the total. The first time a team saw how much they were spending on 400s, they fixed their client code that afternoon.
Give teams their own numbers, not the whole company
A company-wide dashboard that says the LLM bill is fifty thousand dollars a month helps nobody. A dashboard that says your team spent nine thousand, up forty percent from last month, on a specific model, changes behavior overnight.
We publish per-team views: tokens and estimated cost by team, by model, by day. The estimated cost part matters. Tokens are abstract. Dollars are not. We maintain a small price table per model and multiply. The estimate is not perfect, and we say that on the dashboard. It is close enough to drive decisions.
The behavioral change was immediate. Teams started batching requests. They cached repeated prompts. One team switched a summarization job from a large model to a smaller one after seeing the per-run cost. None of that happened when the bill was a single line item.
Budgets need teeth, but start with visibility
Once teams can see their spend, the next question is limits. We added budget alerts first, hard stops second. The sequence matters.
Alerts went to the team's channel when they crossed fifty, eighty, and one hundred percent of their monthly budget. Most teams self-corrected at fifty. For the ones that did not, a hard stop at the gateway rate-limits further calls until someone raises the budget.
We deliberately did not lead with hard stops. An engineer running a time-sensitive demo who hits a hard limit learns that the gateway is the enemy. An engineer who gets a warning at fifty percent learns to watch their budget. The hard stop exists for the runaway loops, not for normal work.
What we would do differently
Three things we would change if we started over.
First, we would meter streaming responses from day one. Token counts arrive at the end of a streamed response, and our first implementation missed them. Streaming became popular faster than we expected.
Second, we would standardize the metadata stamp earlier. We let teams add custom attributes organically, and we ended up with three different keys for the same project name. Normalizing that retroactively was tedious.
Third, we would keep the price table in version control with a review step. Model prices change. When they do, old estimates look wrong. We now treat the price table like code: a pull request, a reviewer, a changelog.
The real lesson
Metering is not a billing feature. It is a feedback loop. The moment teams could see what their calls cost, they started engineering like the cost mattered. Architecture improved. Waste fell. Nobody needed a policy document telling them to be careful with the expensive model, because the numbers told the story better than any document could.
If you are standing up an LLM gateway and nobody has asked about cost attribution yet, they will. Build the metering in before the bill becomes a conversation.
Top comments (13)
You are right on both counts. Pricing every input token at the full rate punishes the caching the loop is supposed to reward, so cached and fresh tokens need separate prices in the table. And per-task rollup by trace ID is the fix for fan-out: the question stops being who burned the tokens and becomes which feature paid for them.
The cache write point is a good catch. A cold cache that keeps rewriting looks cheap if every input token gets the same price, and that is exactly the inversion to avoid. On the seat-based coding agents, honest answer: that spend still sits outside the gateway view today, on a separate invoice, and folding it into the per-team number is the gap left to close.
Workflow identity. The human who launched the job gets the alert, but the quota itself keys to the workflow, because a headless job does not change behavior when a person sees a number. That split is what made the caps actually bite.
The "metering is a feedback loop, not a billing feature" line is the thesis I would carry everywhere. I have seen teams discover their spend weeks later in a finance report, by which point the only remaining option is blame; when the number sits right next to the code that generated it, the same team self-corrects before the bill lands. The streaming retrofit is the detail I would flag hardest — with SSE streams, a client disconnecting mid-response means the usage block never arrives, and that is exactly the spend your meter will miss. I would push the estimated-cost point one step further: engineers do not think in tokens, they think in dollars, so normalizing to cost at the gateway makes the feedback loop twice as strong. And keeping the price table in version control with a review step is the kind of unglamorous discipline that saves you from a silent margin leak when a provider changes pricing. One question: did you attribute spend by API key or by user identity behind the gateway, and which one held up better when a key inevitably got shared across teams?
We started on API key because it was the cheapest thing that worked. It fell apart the moment a key got passed around two teams and the bill landed on the wrong budget. Identity behind the gateway is what held up, since one human maps to one budget no matter which app they call through.
The "one human maps to one budget" line is the operational detail most metering posts skip. An API key fails on accounting before it fails on security: a key has no owner, so the moment it gets shared across two teams, nobody is accountable for the spend and nobody trusts the bill. I hit a smaller version of this when I wired per-user token budgets into one of my own tools — the fix was not a fancier dashboard, it was tagging every request to the requesting user instead of the key. Suddenly people could see their own burn, and wasteful calls dropped on their own. The part I would push on: did you land on per-user hard quotas, or attribution first and quotas later? Attribution feels like the feedback-loop side; quotas are the billing side. Curious which one changed behavior more for you.
Attribution first, quotas later, and it was not close. The moment each request carried a user identity and the burn showed up per person, the wasteful calls dropped on their own, exactly like you saw. Hard quotas only went in later for the few heavy agent workflows that treated attribution as informational.
The heavy agent workflows treating attribution as informational is the part that would worry me most, because that is where the cost concentrates. In my experience visibility only changes behavior for humans in the loop — anything headless keeps burning until a hard cap actually bites. The one middle ground I have seen work: per-workflow alerts routed to whoever launched the job, so the owner feels the page instead of a shared team channel getting spammed. For those agent workloads, did you gate the quotas by workflow identity or fall back to the human who triggered them?
The detail about failed 400s still billing input tokens is the one most teams find on the invoice rather than in testing. A retry loop around a malformed request can cost more than the successful traffic.
Odinaka's distinction between an API key and an identity is the part I would put first. One key shared by five workflows tells you the total and nothing else. On client builds we decide the unit of attribution before launch, usually the workflow or the customer, and tag every gateway call with it. That single decision answers the question in your title months later, when finance asks why one month doubled.
Wasim's view that per task caps should fail hard also matches what we have seen. A soft warning gets ignored at 2am; a hard stop gets a ticket and a fix.
How are you handling cached token pricing in the meter, as Alex raised? Do you record the discount per call or reconcile it later against the provider bill?
Extending the "price table in version control" lesson one step further: it is not just the prices that need versioning, it is the price structure. Your feedback loop trains teams to cache and batch, which is exactly the behavior you want to encourage. But cached input tokens are priced at a fraction of fresh ones, and the usage block reports them separately. If the gateway prices every input token at the full rate, the teams doing the most caching look like your biggest spenders, while the team re-sending the same system prompt on every call looks comparatively normal. The feedback loop inverts: it quietly punishes the caching it was meant to reward.
One related addition for the agent-heavy workflows robertpaschal flagged: attributing by user identity is necessary, but no longer sufficient when a single user action fans out into dozens of model calls across tools. Stamping a trace or session ID on the inbound path and rolling usage up per task, rather than per HTTP request, gives each team a number that actually answers the question in your title. Not "who burned these tokens," but "which feature paid for them."
cool piece, comments are nearly as good too!.
Alex has the cached-input point covered. I'd add that it cuts both ways if any of your teams go through Anthropic. Cache reads bill at a tenth of the input rate, but cache WRITES bill above it (1.25x, or 2x for the hour-long cache).
In my own coding-agent logs writes were 29% of the cost, more than three times the output! So a team whose cache keeps going cold looks cheaper than it really is if every input token gets the same price.
The other gap I'd look at is coding agents. Anyone on a Claude or Copilot seat never touches your gateway, so the dashboard shows a tidy API bill while the agent spend turns up on a different invoice. Did you bring those into the same per-team view, or keep them apart?
The failed-400s detail is the one that stings, since the provider still bills input tokens on calls that return nothing. Metering shows you the spend after the fact, though. A per-task cap on tool calls and tokens would stop a runaway loop before it becomes a line on the bill. When a team hits a cap in your setup, does the request fail hard or return a partial result?
Fail hard. A partial result poisons both the metering trail and the guardrail audit log, so a capped call is a clean stop, not a degraded answer.