Stay, Harden, or Pay: A Free-Tier Checklist for AI Features

Last month a user told me my summarizer "just spun forever."

My dashboard was green. My success rate read 94%. So I did the thing I always do: I blamed the prompt.

The prompt was fine. My accounting was not. Twelve requests had no outcome row at all — no ok, no error, nothing. They timed out and vanished.

That is the trap with free tiers. Nothing throws. Everything just gets quiet.

So I wrote a checklist with three possible answers instead of two. Here it is, with the script I actually run.

The three answers, and why two are cheap

Every free-tier review ends in one of these:

1. STAY — the evidence supports the free path for this exact workload.
2. HARDEN — you do not have enough evidence, or too many outcomes are unknown. Do not change what real users see.
3. PAY — the evidence says the free path cannot keep the promise you made.

Only one of those needs a credit card. The other two need logging you probably already have.

Notice what is missing: "try harder" and "wait and see." Both are HARDEN in disguise, and HARDEN has an exit condition.

The number almost nobody logs

An error is an outcome. A timeout with no record is not an outcome.

Ask yourself: if my worker dies mid-call, does any row get written? For me the answer was no. My success rate was computed over the requests that bothered to report back.

That is survivorship bias with a JSON file. So the checklist puts an unknown rate ahead of success rate. If you cannot account for a request, you cannot count it as a win.

Step 1 — Pick a log format and actually write it

One JSON object per line. No nesting, no schema migration, no database.

{"task":"sum-0421","outcome":"ok","latency_ms":5400,"tier":"free","server":"free"}
{"task":"sum-0422","outcome":"error","latency_ms":9100,"tier":"free","server":"free"}

Rules I hold myself to:

• Write the row after the call resolves, including timeouts and cancellations, with outcome: "unknown".
• Log server and tier on every row. You will want to compare later.
• Never log prompt text or user input. Task IDs only.

Step 2 — The checker

Save this as tier_check.py. It is about 55 lines and has no dependencies.

#!/usr/bin/env python3
"""tier_check.py - STAY / HARDEN / PAY for one AI workload. Unknowns fail closed."""
import json
import sys

BUDGET_P95_MS = 8000
MIN_SUCCESS = 0.90
MAX_UNKNOWN = 0.02
MIN_SAMPLE = 50


def load(path):
    rows = []
    with open(path, encoding="utf-8") as fh:
        for i, line in enumerate(fh, 1):
            line = line.strip()
            if not line:
                continue
            try:
                rows.append(json.loads(line))
            except json.JSONDecodeError:
                # a corrupt line is an unknown outcome, not a skipped line
                rows.append({"outcome": "unknown", "why": f"line {i} unparsable"})
    return rows


def summarize(rows):
    n = len(rows)
    ok = sum(1 for r in rows if r.get("outcome") == "ok")
    unknown = sum(1 for r in rows if r.get("outcome") not in ("ok", "error"))
    lat = sorted(r["latency_ms"] for r in rows if isinstance(r.get("latency_ms"), int))
    p95 = lat[-(-len(lat) * 95 // 100) - 1] if lat else None
    return {
        "n": n,
        "success_rate": round(ok / n, 3) if n else 0.0,
        "unknown_rate": round(unknown / n, 3) if n else 1.0,
        "p95_ms": p95,
    }


def verdict(s):
    if s["n"] < MIN_SAMPLE:
        return "HARDEN", f"n={s['n']} < {MIN_SAMPLE}"
    if s["unknown_rate"] > MAX_UNKNOWN:
        return "HARDEN", f"unknown_rate={s['unknown_rate']:.1%} > {MAX_UNKNOWN:.0%}"
    if s["success_rate"] < MIN_SUCCESS:
        return "PAY", f"success_rate={s['success_rate']:.1%} < {MIN_SUCCESS:.0%}"
    if s["p95_ms"] is None:
        return "HARDEN", "no latency samples"
    if s["p95_ms"] > BUDGET_P95_MS:
        return "PAY", f"p95={s['p95_ms']}ms > {BUDGET_P95_MS}ms"
    return "STAY", "all checks passed"


if __name__ == "__main__":
    path = sys.argv[1] if len(sys.argv) > 1 else "runs.jsonl"
    s = summarize(load(path))
    decision, why = verdict(s)
    print(json.dumps({**s, "decision": decision, "why": why}, indent=2))
    sys.exit({"STAY": 0, "HARDEN": 1, "PAY": 2}[decision])

The check order matters. Sample size first, then unknowns, then success, then latency. The first failing check decides, and ties go to the safer answer. Unknown data never becomes STAY.

Step 3 — Your thresholds, written down before you look

Write these in the repo before running anything. Otherwise you will move the line to fit the result.

Mine are tuned to a chat feature with a patient user. A background job can take 10x the latency.

Step 4 — Run it against a clean log

python tier_check.py runs.ok.jsonl; echo "exit=$?"

{"n": 120, "success_rate": 0.967, "unknown_rate": 0.008, "p95_ms": 6100,
 "decision": "STAY", "why": "all checks passed"}

exit=0. Fine. Ship the feature, keep the log running.

The failure fixture is the point

A checker that only prints STAY is decoration. So keep a second file in the repo: a log where the numbers look good and are wrong.

python tier_check.py fixtures/runs.quiet-fail.jsonl; echo "exit=$?"

{"n": 120, "success_rate": 0.9, "unknown_rate": 0.092, "p95_ms": 6400,
 "decision": "HARDEN", "why": "unknown_rate=9.2% > 2%"}

exit=1. Success rate is exactly at the floor. Latency is fine. Eleven requests simply never reported.

That fixture is 120 lines I can point at in review. A test log beats a paragraph of opinion.

Then wire it in, so it runs without me:

python tier_check.py runs.jsonl || { echo "tier check did not STAY"; exit 1; }

Note that exit code 2 is also a failure. PAY is not a pass — it means stop and change something before this reaches users.

Where the free tier fits

I built this against MonkeyCode's free model access and its free server option, because a free tier is exactly where quiet failures hide: shared resources, no invoice to make you look, nothing to page you at 2am. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The product is not the method, though. Any runner can fill runs.jsonl. The workflow survives a change of provider, which is how I want it. Check the project's site for the current free quota rather than trusting a number in a blog post, including this one.

Limitations, plainly

• 100 calls on one workload prove nothing about another. A summarizer and a code generator have different tails.
• Free-server latency is a shared-resource measurement. It is not a capacity promise, and it will move.
• The thresholds are opinions encoded as integers. Change yours, but change them before you look at results.
• This catches silence, not quality. A fast, wrong answer scores ok.

Who should not use this

If your task is irreversible or high-stakes, this checklist is far too weak. If you have a contractual latency SLA, a 50-call sample is not evidence. And if you cannot write an outcome row per request at all, fix that first — the script will just tell you what you already fear.

My abandonment criteria

I gave this three evenings, roughly four hours. If two HARDEN cycles over two weeks do not drop the unknown rate, I stop trying to rescue the free path for that workload. Either I pay, or I cut the feature.

Both are acceptable endings. A feature running on unexplained silence is not.

How many of your last 100 AI calls have no recorded outcome? That single number decided more for me than latency ever did, and I would like to know what yours looks like.