Once you put a large language model behind a company API gateway, someone asks a simple question.
Who is paying for this?
The model bill arrives ...
For further actions, you may consider blocking this person and/or reporting abuse
Workflow identity. The human who launched the job gets the alert, but the quota itself keys to the workflow, because a headless job does not change behavior when a person sees a number. That split is what made the caps actually bite.
You are right on both counts. Pricing every input token at the full rate punishes the caching the loop is supposed to reward, so cached and fresh tokens need separate prices in the table. And per-task rollup by trace ID is the fix for fan-out: the question stops being who burned the tokens and becomes which feature paid for them.
The cache write point is a good catch. A cold cache that keeps rewriting looks cheap if every input token gets the same price, and that is exactly the inversion to avoid. On the seat-based coding agents, honest answer: that spend still sits outside the gateway view today, on a separate invoice, and folding it into the per-team number is the gap left to close.
Thanks, straight answer appreciated. It's the gap I keep running into too. Seat tools never touch a gateway, so per-team agent spend ends up being whatever the vendor's admin page shows you. Disclosure, it's the gap I'm building OmnisVigil to close, so I'll let you know if I crack it. Curious what you'd want in that per-team number first, cost per repo or cost per workflow?
We record it per call. The metering event carries cached and fresh token counts separately, each priced from the table, so the per-call cost already reflects the discount. The provider bill is a monthly reconciliation check, not the source of truth. If the meter only reconciled later, teams would see the wrong number all month and learn nothing from it.
The "metering is a feedback loop, not a billing feature" line is the thesis I would carry everywhere. I have seen teams discover their spend weeks later in a finance report, by which point the only remaining option is blame; when the number sits right next to the code that generated it, the same team self-corrects before the bill lands. The streaming retrofit is the detail I would flag hardest — with SSE streams, a client disconnecting mid-response means the usage block never arrives, and that is exactly the spend your meter will miss. I would push the estimated-cost point one step further: engineers do not think in tokens, they think in dollars, so normalizing to cost at the gateway makes the feedback loop twice as strong. And keeping the price table in version control with a review step is the kind of unglamorous discipline that saves you from a silent margin leak when a provider changes pricing. One question: did you attribute spend by API key or by user identity behind the gateway, and which one held up better when a key inevitably got shared across teams?
We started on API key because it was the cheapest thing that worked. It fell apart the moment a key got passed around two teams and the bill landed on the wrong budget. Identity behind the gateway is what held up, since one human maps to one budget no matter which app they call through.
The "one human maps to one budget" line is the operational detail most metering posts skip. An API key fails on accounting before it fails on security: a key has no owner, so the moment it gets shared across two teams, nobody is accountable for the spend and nobody trusts the bill. I hit a smaller version of this when I wired per-user token budgets into one of my own tools — the fix was not a fancier dashboard, it was tagging every request to the requesting user instead of the key. Suddenly people could see their own burn, and wasteful calls dropped on their own. The part I would push on: did you land on per-user hard quotas, or attribution first and quotas later? Attribution feels like the feedback-loop side; quotas are the billing side. Curious which one changed behavior more for you.
Attribution first, quotas later, and it was not close. The moment each request carried a user identity and the burn showed up per person, the wasteful calls dropped on their own, exactly like you saw. Hard quotas only went in later for the few heavy agent workflows that treated attribution as informational.
The heavy agent workflows treating attribution as informational is the part that would worry me most, because that is where the cost concentrates. In my experience visibility only changes behavior for humans in the loop — anything headless keeps burning until a hard cap actually bites. The one middle ground I have seen work: per-workflow alerts routed to whoever launched the job, so the owner feels the page instead of a shared team channel getting spammed. For those agent workloads, did you gate the quotas by workflow identity or fall back to the human who triggered them?
Extending the "price table in version control" lesson one step further: it is not just the prices that need versioning, it is the price structure. Your feedback loop trains teams to cache and batch, which is exactly the behavior you want to encourage. But cached input tokens are priced at a fraction of fresh ones, and the usage block reports them separately. If the gateway prices every input token at the full rate, the teams doing the most caching look like your biggest spenders, while the team re-sending the same system prompt on every call looks comparatively normal. The feedback loop inverts: it quietly punishes the caching it was meant to reward.
One related addition for the agent-heavy workflows robertpaschal flagged: attributing by user identity is necessary, but no longer sufficient when a single user action fans out into dozens of model calls across tools. Stamping a trace or session ID on the inbound path and rolling usage up per task, rather than per HTTP request, gives each team a number that actually answers the question in your title. Not "who burned these tokens," but "which feature paid for them."
cool piece, comments are nearly as good too!.
Alex has the cached-input point covered. I'd add that it cuts both ways if any of your teams go through Anthropic. Cache reads bill at a tenth of the input rate, but cache WRITES bill above it (1.25x, or 2x for the hour-long cache).
In my own coding-agent logs writes were 29% of the cost, more than three times the output! So a team whose cache keeps going cold looks cheaper than it really is if every input token gets the same price.
The other gap I'd look at is coding agents. Anyone on a Claude or Copilot seat never touches your gateway, so the dashboard shows a tidy API bill while the agent spend turns up on a different invoice. Did you bring those into the same per-team view, or keep them apart?
The detail about failed 400s still billing input tokens is the one most teams find on the invoice rather than in testing. A retry loop around a malformed request can cost more than the successful traffic.
Odinaka's distinction between an API key and an identity is the part I would put first. One key shared by five workflows tells you the total and nothing else. On client builds we decide the unit of attribution before launch, usually the workflow or the customer, and tag every gateway call with it. That single decision answers the question in your title months later, when finance asks why one month doubled.
Wasim's view that per task caps should fail hard also matches what we have seen. A soft warning gets ignored at 2am; a hard stop gets a ticket and a fix.
How are you handling cached token pricing in the meter, as Alex raised? Do you record the discount per call or reconcile it later against the provider bill?
The failed-400s detail is the one that stings, since the provider still bills input tokens on calls that return nothing. Metering shows you the spend after the fact, though. A per-task cap on tool calls and tokens would stop a runaway loop before it becomes a line on the bill. When a team hits a cap in your setup, does the request fail hard or return a partial result?
Fail hard. A partial result poisons both the metering trail and the guardrail audit log, so a capped call is a clean stop, not a degraded answer.