DEV Community

Cover image for How to Reconcile Metered Usage Billing When Events Arrive Late, Twice, or Never
Shola Jegede
Shola Jegede Subscriber

Posted on

How to Reconcile Metered Usage Billing When Events Arrive Late, Twice, or Never

A short link app has a simple job: count the clicks and bill the owner for them. The app counts each click in its own database. The billing provider keeps a second count, called a meter, and the invoice comes from the meter. Both counts start equal, but nothing keeps them equal.

A click can reach the database after the total was sent. A retry can send the same usage twice. A report can fail on a timeout and never be sent again. Each case leaves the meter wrong, and the customer sees the error only as an invoice that is too high or too low.

The hard question is this: how do you prove that the meter matches what really happened, and fix it when it does not?

This article answers that question with a small system I call Notch. Notch is a short link and QR code app that bills per click. It keeps a ledger of every click, sends totals to the meter, and runs a reconciler that compares the two, repairs a difference, and keeps the proof. The billing provider in my test is Kinde, a platform for authentication and billing, but the design needs only two things from a provider: a way to set a total, and a way to read it back.

Why the meter and the ledger drift apart

A ledger is the list of events your app wrote down, one row for each click. A meter is the provider's running total for a billing cycle, which is one month of usage. The ledger is the record, and the meter is a copy that you keep up to date. Three things break that copy.

Late, twice and never: what happens to the meter with and without a ledger

Usage arrives late. The app sends a total at 10:00. A click is recorded at 10:01. Until the next report, the meter is one click behind. If the cycle closes before that report, the meter stays wrong, and a closed cycle is the one you can no longer change.

Usage arrives twice. A retry runs after a timeout, or a second job writes to the same meter. If the write adds usage, the second write adds it again. The customer pays for clicks that happened once.

Usage never arrives. A report fails on a timeout or a rate limit, and nothing sends it again. The meter stays low. Nothing fails loudly, so nobody sees it.

All three look the same from the provider's side. The meter holds a number, and the provider cannot tell you whether the number is right. Only the app has the record to check it against, which is why the first rule of the design is that the ledger is the record.

What the provider lets you do

A design has to match what the provider actually accepts, so I read the Kinde documentation and then tested each behavior on my own account before I relied on it.

Kinde's usage endpoint takes a meter_type_code. The value absolute sets the running total to the number you send, and it is the default. The value delta adds the number to the running total. That difference decides the whole design. An absolute write is safe to repeat, because the second write sets the same total. A delta write is not, because a retry adds the usage again, and I confirmed this by writing the same delta twice and reading the total move twice.

Stripe takes the opposite route. Each Stripe meter event has an identifier that you can set, and Stripe uses it to prevent a second count of the same event. Kinde's documentation shows no such key, so with Kinde the safe write is the absolute one.

The write is only half of the API. To compare the meter with the ledger, the app must read the total back. Kinde's read endpoint returns the current cycle by default, and all_cycles=true returns every record. The read needs the read:meter_usage scope, which is separate from the write scope.

Three more facts shaped the design:

  • A meter_usage_timestamp is stored on the record, but it does not select a billing cycle or change the total. A write that I backdated became its own record, and the live total did not move.
  • Each write is a record. After about 1,200 writes, the total still read correctly.
  • Kinde bills recurring metered charges in arrears, and it starts each cycle on the customer's sign-up date, not on the first of the month.

The ledger is the record

Every click becomes one row in the ledger, and the row has a key that the app makes before it writes anything. The redirect to the destination goes out first. The write happens after the response, and it carries the same key on every try. A second try with the same key adds nothing, so a retry cannot count a click twice.

A click that a person or a bot should not have caused is not deleted. The ledger is append only, so a correction is a new row with units: -1, written into the cycle of the click it corrects. The total is the sum of all rows, and the history stays readable.

The four parts: ledger, reporter, reconciler and evidence

One more detail keeps the ledger fast. If every click updated one counter row, many clicks at once would fight over that row. Notch splits each counter over 64 rows and adds them on read. Each click picks one shard at random, so two clicks rarely touch the same row. The click and its counter update commit in one transaction, and the same transaction marks the cycle as changed.

That mark is what connects the ledger to the meter. A cycle is marked once, and more clicks while it is marked add no more writes. The mark says one thing: the meter may be behind, so send the total again.

The reporter sends totals, not clicks

Every minute the reporter takes up to 100 marked cycles, deletes the marks, and reads the ledger total for each cycle. It sends each total to the meter with an absolute write, 10 at a time.

Sending totals fixes two of the three failures at once. A repeated report is harmless, because it sets the same number. A late click is also harmless, because it marks the cycle again and the next report carries the new total.

The third failure, the report that never arrives, needs one more step. If a write fails or gets a 429, the reporter marks the cycle again after the loop ends. The next minute tries again. Kinde documents a limit on concurrent requests and a RateLimit-Reset header on a 429, which gives the seconds until the limit resets. I never saw a 429 in my tests, so that path is covered by tests with a fake meter, not by a real rejection.

If a cycle has already closed, the reporter does not write. A closed cycle goes to a review list, because a write to a closed cycle has an effect that I could not confirm.

The reconciler proves the copy

The reporter keeps the meter close. It cannot prove the meter is right, because it only knows what it sent. The reconciler does the proof, and it runs nightly and on demand.

For each cycle, the reconciler reads every ledger row and adds them up. It checks that sum against the running counters, and it rebuilds the counters if they differ. It then compares three numbers: the ledger total, the last total the app sent, and the total the provider holds.

How the reconciler decides what to do with three numbers

The comparison is a short pure function, so a test can check each case alone:

export function decide(e: Evidence): Decision {
  if (e.provider === null) return { verdict: "unreadable", repair: null, /* ... */ };
  if (e.provider === e.ledger) return { verdict: "match", repair: null, /* ... */ };
  if (e.closed) return { verdict: "closed_cycle_drift", repair: null, /* ... */ };
  return {
    verdict: e.provider > e.ledger ? "over_billed" : "under_billed",
    repair: e.ledger, // the ledger is the record
    /* ... */
  };
}
Enter fullscreen mode Exit fullscreen mode

The function never trusts the meter over the ledger. When the two differ in an open cycle, the repair is always the ledger total. The cause text depends on the evidence: if the last report failed, the reconciler says so, and if the meter moved without a report from the app, it says that a report was counted twice or something else wrote to the meter.

A repair is not finished when the write returns 200. The reconciler reads the meter back and checks that it now equals the ledger. Only then does it record "repaired and confirmed". If the read back differs, the run ends with an error.

Every run stores evidence, which is one row with the three numbers, the verdict, the cause, and the read back. A run on the same numbers adds no new row, so running the reconciler twice leaves the record unchanged. That makes the evidence cheap to keep and useful to show: the answer to "was this customer billed correctly in October" is a row, not a memory.

The meter's calendar is not yours

One mistake would corrupt every number above. A cycle named by calendar month would put a click on 20 October into 2026-10 while the provider counts it in a cycle that started on 20 September. On 1 November the app would then write a small October-only total over the meter's real cycle.

Kinde starts cycles on the customer's sign-up date. Notch therefore takes an agreement start when you connect an account, and it names each cycle by the month it starts in. Take an agreement start of 20 October at 15:29 UTC. A click on 19 November belongs to the cycle that began on 20 October, and a click on 21 November belongs to the next one.

Two rules follow. The start must fall on the 1st to the 28th of a month, because a start on the 31st has no clear meaning in a short month. The start is set before the first click and refused afterward, because changing it later would move old clicks into different cycles. Kinde's FAQ says that only monthly cycles are supported, so Notch handles monthly cycles only.

What happened with live clicks

I ran the system against a real Kinde account in two ways. First, I sent 5,000 clicks to a short link at 100 at a time, from a laptop dev server. All 5,000 were recorded. The server handled about 97 redirects a second, with a median of 455 ms, a 95th percentile of 916 ms and a 99th percentile of 1,633 ms. The reconciler then found a match at 6,048 in the ledger, the sent log and the meter, with 0 corrections.

That first run proves that the loop works when nothing is wrong. The second run shows what happens when something is. After a later run of 2,100 clicks, the ledger held 8,457 clicks, and the meter held 8,031. The meter was 426 clicks behind. The reconciler marked the cycle as under-billed, wrote 8,457 with an absolute write, read it back, and stored the evidence. The next run matched at 8,457 and added no new row.

Three reconciliation runs: a match, an under-billed repair, and a match again

Those are the numbers from the runs I watched:

  • 5,000 of 5,000 clicks were recorded at 100 at a time.
  • The 2,100 click run reached about 65 redirects a second, with a median of 274 ms and a 99th percentile of 705 ms.
  • The reconciler repaired a gap of 426 clicks and confirmed the repair by reading the meter back.

The cause text that the reconciler stored reads "The ledger changed after the last report. A late click or a correction arrived." That is the late case from the start of this post, found and closed with evidence, and nobody had to notice it.

Limits

This system has edges, and I want to name them.

  • The reconciler proves the copy, not the capture. A click that never reached the ledger is invisible to it. The load run counts what was sent against what was recorded, and that is a separate check.
  • Only the current cycle can be read. Notch reads the current cycle from Kinde. A closed cycle that drifts is flagged for a person, and the reconciler does not repair it.
  • The cycle rollover is untested. My test agreement starts a new cycle on 21 November. Until that date passes, I have not seen what Kinde does at the exact moment of the change.
  • I have not seen a real 429. The 1,200 writes that I sent at about 180 a second all returned 200. The retry path rests on Kinde's documentation and on tests.
  • I do not know what a past-month write does to an invoice. Kinde's documentation does not say. Stripe's documentation says that a late event may be accepted without changing an invoice that is already final. Check your own provider before you rely on a late write.
  • The load numbers come from a laptop. They show that the design holds under load, not what a production host will do.
  • One provider was tested. The design needs a set operation and a read, but I only ran it against Kinde.

The rule that holds it together

The ledger is the fact, and the meter is a copy of it. Once you treat them that way, a late click becomes a new mark, a repeated report becomes the same number, and a failed report becomes a retry. Each case is the same case: the copy is behind, so send the total again and check it.

To run it yourself, clone the repository and run npm test for the 58 tests. To measure your own deployment, run node scripts/load.mjs <short code> --clicks 5000 --concurrency 100. Then compare the ledger with the meter in the reconciliation panel, and look for a gap that you did not cause. Do let me know what you find in the comments.

FAQ

Why send a total instead of each click?
A total is safe to repeat and safe to send late. A per-click write needs a unique key at the provider for every click. Where a provider has no such key, a repeated write counts twice.

What if the provider only supports delta writes?
Keep the ledger and send each change with an idempotency key if the provider accepts one. Stripe does, through the meter event identifier. If the provider has neither an absolute write nor a key, you can only detect a drift and not repair it safely.

Why not trust the meter after a repair?
A write that returns 200 says that the provider accepted the request. It does not say what total the provider now holds. Reading the value back is the only proof.

Does the cycle anchor matter if I bill on the first of the month?
No. With no anchor, a cycle is a UTC calendar month, which matches a provider that starts every cycle on the first. The anchor matters when the provider starts cycles on the customer's sign-up date.

How often should the reconciler run?
Run it nightly, plus on demand after an incident. It reads every ledger row, so run it in pages. Notch does 50 accounts a page, 5 at a time.

Sources

Top comments (0)