Logging AI tool usage without logging sensitive data

When a team adds an AI feature, the first debugging instinct is to log everything: the full prompt, the full response, the user's message. It works. It also means your logs now hold a copy of every customer email, contract and support ticket that went through the feature, sitting in a system with weaker access controls than your database and a much longer memory.

You need logs. You need them to debug bad answers, answer audit questions and keep costs under control. You don't need them to be a second copy of your customers' data.

Here's a practical middle path.

Decide what the logs are for

Before writing any code, list the questions your logs must answer. For most small teams it's these:

1. Debugging: which prompt version and model produced this answer, and did the call fail?
2. Audit: who (which user or service) used the AI feature, when, and on what record?
3. Cost: how many tokens did we use, by feature and by customer?
4. Safety: did a call get blocked, filtered or flagged?

Notice that none of these needs the full text of the prompt or the response. That's the key idea: log metadata by default, and content only by exception.

The default log record

This is the record to write for every call:

import hmac
import json
import logging
import os
import time
import uuid

log = logging.getLogger("ai_usage")


FP_KEY = os.environ["AI_LOG_FP_KEY"].encode()   # random secret, not in code


def fingerprint(text: str) -> str:
    """Keyed, non-reversible fingerprint so identical inputs can be grouped."""
    return hmac.new(FP_KEY, text.encode("utf-8"), "sha256").hexdigest()[:16]


def log_ai_call(*, feature, prompt_name, prompt_version, model, actor_id,
                record_id, prompt_text, response_text, usage, status, started):
    # started: value of time.monotonic() taken before the call
    usage = usage or {}                      # failed calls often have no usage data
    call_id = str(uuid.uuid4())
    log.info(json.dumps({
        "event": "ai_call",
        "call_id": call_id,
        "feature": feature,                  # e.g. "ticket_summary"
        "prompt": prompt_name,
        "prompt_version": prompt_version,
        "model": model,
        "actor_id": actor_id,                # internal user or service id, not an email
        "record_id": record_id,              # e.g. ticket id
        "input_chars": len(prompt_text),
        "input_fp": fingerprint(prompt_text),
        "output_chars": len(response_text or ""),
        "input_tokens": usage.get("input_tokens"),
        "output_tokens": usage.get("output_tokens"),
        "status": status,                    # "ok", "error", "blocked"
        "latency_ms": int((time.monotonic() - started) * 1000),
    }))
    return call_id

With this, you can trace any answer to a prompt version and a record, add up costs by feature, and see failure rates, all without storing a word of the customer's text.

A note on the fingerprint: a plain hash of short or predictable text (an email address, a yes/no answer) can be reversed by guessing, so the fingerprint is keyed with a secret (AI_LOG_FP_KEY), so it can't be checked by guessing without the key. The fingerprint is for grouping long, varied inputs ("were these two calls the same request?"), not for hiding identifiers. Identifiers like emails shouldn't go into it in the first place.

When you do need content: redact first

Sometimes you need to see what was sent, for example while fixing a specific bug. When that happens, log a redacted sample, not the raw text, and only when a flag is switched on.

A simple redaction pass catches the most common sensitive patterns:

import re

PATTERNS = [
    (re.compile(r"\b(?:sk|pk|api|key|token|ghp|xox[bp])[-_][A-Za-z0-9_-]{12,}\b", re.I), "[SECRET]"),
    (re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+"), "[EMAIL]"),
    (re.compile(r"\b(?:\d[ -]?){13,19}\b"), "[CARD]"),
    (re.compile(r"(?<![\w.])\+?\d(?:[\s()-]{0,2}\d){9,14}(?![\w.])"), "[PHONE]"),
]


def redact(text: str, limit: int = 500) -> str:
    for pattern, label in PATTERNS:
        text = pattern.sub(label, text)
    return text[:limit]

Order matters: run the most specific patterns first. Secrets go before emails, cards and phone numbers, or a phone pattern can eat the digits in the middle of a key and leave the rest in the log.

import os

DEBUG_CONTENT = os.getenv("AI_LOG_CONTENT") == "1"

call_id = log_ai_call(...)  # same call as above

if DEBUG_CONTENT:
    log.info(json.dumps({
        "event": "ai_call_content",
        "call_id": call_id,
        "prompt_sample": redact(prompt_text),
        "response_sample": redact(response_text or ""),
    }))

Be honest about the limits. Regular expressions catch emails, phone numbers, card-like numbers and obvious keys. They won't catch names, many provider key formats (cloud access keys, bearer tokens), addresses, medical details or anything written in free text. That's why content logging stays off by default, is switched on for a short time for a specific problem, and is truncated. If your feature handles health, financial or legal information, don't log content at all; reproduce the problem with test data instead.

Keep content logs somewhere separate, for less time

Metadata and content have different risks, so treat them differently:

Most log platforms let you route by a field (here, event) and set a separate retention period per stream. Use that, so content samples expire automatically and nobody has to remember to delete them.

Don't forget the other copies

Your own logs are only one place prompts end up:

• The AI provider. Check their data retention and training settings for your account, and turn off anything you don't need. Write down what you chose and why.
• Error trackers. Exception reports often capture local variables, including the prompt. Configure them to scrub request bodies and the variables that hold prompt text. Provider error messages sometimes echo part of the prompt too, so log the exception type (type(exc).__name__) and status code, not str(exc).
• Analytics and tracing tools. If you trace requests end to end, make sure spans don't attach the full prompt as an attribute.
• Chat and tickets. When someone pastes a "bad answer" into a team chat to discuss it, that's a copy too. Share the call_id instead, and look the details up where access is controlled.

What auditors usually ask

If you work with larger clients, sooner or later someone will ask how you control AI use. The metadata log answers most of it:

• Which features use AI, and which models? (feature, model)
• Who used them, on which records? (actor_id, record_id)
• Can you show a change history for prompts? (prompt_version, plus your repository history)
• How long do you keep AI-related data, and who can see it? (your retention table above)

Being able to answer those from a dashboard, rather than by searching raw text, is a good sign you've got the balance right.

A short checklist

1. Write a metadata record for every AI call: feature, prompt version, model, actor, record, tokens, status, latency.
2. Never put emails or other identifiers into actor_id or the fingerprint.
3. Keep content logging off by default; when it's on, redact and truncate.
4. Send content samples to a separate stream with short retention and narrow access.
5. Review provider retention settings, error-tracker scrubbing and tracing attributes.
6. Share call IDs in chat, not prompts.

Good AI logs tell you what happened without telling everyone what your customers said.

Ahmed Vaseer is Founder & CEO - Ingenious Concepts, an AI-first technology and digital growth agency. 80+ projects, ISO 27001 certified via Global CB.