Give Your AI Feature a Kill Switch, a Budget, and a Paper Trail

The second-most dangerous demo is "watch the agent do everything." The most dangerous is shipping that demo to real users on Monday.

I build web and mobile products with Node, React, and Flutter, and I speak about AI in community spaces. The pattern I keep seeing: teams add a chat box or an "AI assistant" tab, wire an API key, and hope prompt vibes will behave in production. Hope is not an architecture.

This post is a practical starter kit for fullstack builders: three controls every AI feature should have before you invite non-technical users — a kill switch, a budget, and a paper trail. No cloud-vendor lock-in assumed. You can implement these with boring Node middleware and a small React admin panel.

Why demos lie

Figure: gate every model call — kill switch, budget, paper trail.

In a Meetup demo:

• You are the only user
• You type polite, on-distribution prompts
• You ignore the bill
• You can Ctrl+C when it loops
• Nobody pastes a 40-page PDF of junk to "summarize"

In production:

• Users paste secrets by accident
• Prompts go adversarial or just confused
• A retry storm turns into a surprise invoice
• An agent tool call hits the wrong environment
• You need to explain what happened three days later

Governance sounds enterprise. The core is simpler: assume the model will be wrong, expensive, or both — and design the blast radius.

Control 1: Kill switch (feature flags that actually stop spend)

A kill switch is not a blog toggle in Notion. It is a runtime gate checked before every model call and every tool invocation.

Minimal shape:

// lib/aiFlags.js
export async function assertAiEnabled(featureKey) {
  const row = await db.flags.get(featureKey);
  if (!row?.enabled) {
    const err = new Error("AI_FEATURE_DISABLED");
    err.code = "AI_FEATURE_DISABLED";
    throw err;
  }
}

Call it at the edge of your AI service:

export async function runAssistant({ userId, featureKey, input }) {
  await assertAiEnabled(featureKey);
  // ... budget check, then model call
}

Wire a React admin (or even a secured POST /admin/flags) so you can disable meetup-copilot without redeploying. When something smells wrong at 1 a.m., you want a button — not a pipeline.

Mobile note: Flutter clients should treat AI_FEATURE_DISABLED as a first-class UI state ("Assistant temporarily off"), not a generic 500. Users forgive maintenance; they do not forgive spinning forever.

Bonus: scoped kills

Prefer per-feature flags over a global "AI off":

• chat.summarize
• chat.reply-suggest
• agent.tools.github (keep this off until you are truly ready)

That way a broken experimental agent does not take down a simple title-rewriter that still helps organizers.

Control 2: Budget (tokens, calls, and blast radius)

If you cannot answer "what is the max this feature can cost today?", you do not have a product — you have an open tab.

Soft limits per user / per org

export async function assertWithinBudget({ tenantId, featureKey }) {
  const usage = await db.usage.today(tenantId, featureKey);
  const limit = await db.limits.get(tenantId, featureKey);

  if (usage.calls >= limit.maxCalls || usage.tokens >= limit.maxTokens) {
    const err = new Error("AI_BUDGET_EXCEEDED");
    err.code = "AI_BUDGET_EXCEEDED";
    throw err;
  }
}

Practical defaults for an early community product:

• Cap calls per user per day
• Cap max input characters (reject the 2MB paste early)
• Cap max tool iterations for agents (e.g. 5 steps, then stop with a clear message)
• Separate "playground" keys from "production" keys

Hard limits at the provider

Also set provider-side spending caps where they exist. Application budgets fail closed when Redis hiccups; provider caps fail closed when your app has a bug. Use both.

Agents need step budgets, not vibes

If you are building anything agent-like (tool calling, multi-step plans):

• Maximum steps
• Maximum wall-clock time
• Allowlist of tools (read calendar ✗ delete calendar until proven)
• Human confirmation for irreversible actions

Pausing / resuming agents is a hot topic for a reason: long-running autonomy without checkpoints is how you get surprise side effects. Even a simple job table with status: running | paused | done | failed beats an unbounded while-loop in memory.

Control 3: Paper trail (logs you can defend)

When someone asks "why did the assistant tell my users X?", screenshots are not enough.

Log structured events:

await db.aiEvents.insert({
  id: crypto.randomUUID(),
  tenantId,
  userId,
  featureKey,
  model,
  promptVersion,      // e.g. "rsvp-helper@3"
  inputHash,          // hash raw input; store raw only if policy allows
  toolCalls: [...],   // names + args summaries
  outputHash,
  tokensIn,
  tokensOut,
  latencyMs,
  createdAt: new Date().toISOString(),
});

Privacy-aware defaults

• Prefer hashes + short redacted previews over full prompts for sensitive products
• Never log API keys, auth headers, or payment fields
• Define retention (30/90 days) and stick to it
• Make "export my AI history / delete my AI history" a real endpoint if you serve EU users or simply respect people

Prompt versioning is underrated

Store prompts in git (prompts/rsvp-helper.v3.md) and send promptVersion with every call. When behavior changes, you can diff. "The model got dumber" is often "we edited the prompt on Tuesday and forgot."

A thin React admin that earns its keep

You do not need a full observability suite on day one. A single authenticated page with:

• Feature flags table + toggle
• Today's spend / call counts per feature
• Last 50 AI events (status, latency, error code)
• Big red "disable all AI features" button

…will save you more pain than another model provider abstraction.

How this maps to a real feature

Suppose you add "Suggest event description" for organizers:

1. Flag: events.ai.suggest-description (default off in prod until soak test)
2. Budget: 20 suggestions / organizer / day, 1k max input chars
3. Paper trail: prompt version + hashes + tokens
4. UX: show the suggestion as a diff the human must accept — never auto-publish
5. Kill: if suggestions degrade, flip the flag; the rest of the app keeps working

That is product engineering. The model is a dependency, not the product.

Takeaways

1. Kill switches belong in the request path, not in a document.
2. Budgets need daily caps, input caps, and agent step caps — plus provider hard limits.
3. Paper trails need structured events, prompt versions, and privacy rules.
4. Human confirmation for irreversible actions is not optional.
5. Ship AI as a flaggable slice of a working app, not as the app's only spine.

If you are adding your first assistant to a Node/React or Flutter app, start with these three controls before you polish the avatar animation. Cool UI with no blast-radius design is how demos become incidents.

What control do you wish you had added earlier — flag, budget, or audit log? I am curious what broke first for other builders.