I Was Running 3 AI Coding Agents Locally and Had No Idea What They Were Breaking

Like many developers over the past few months, my daily workflow quietly shifted from typing code manually to orchestrating AI coding agents directly from the command line.

On any given Tuesday, I might have Claude Code refactoring an API endpoint in one terminal pane, Kimi Code writing benchmark scripts in another, and Google Antigravity (Agy CLI) running multi-agent tasks in an isolated git worktree.

They were getting work done. But after a couple of weeks, I had that familiar uneasy feeling every engineer gets when running services without an APM:

• What did that 20-minute session actually cost me?
• Did prompt caching save me money, or did it miss and re-bill 150,000 input tokens on every turn?
• Did that test runner bash command actually pass, or did the agent quietly fail with exit code 1 and rewrite the code 4 times before I noticed?
• Why is this agent editing the exact same file 14 times in a row?

None of these CLIs talk to each other. None of them give you an aggregated dashboard. And worst of all, checking external cloud provider billing pages only shows aggregate monthly dollar amounts with a 6-hour delay—useless for real-time debugging.

So I dug into what was actually happening on my machine, and what I found led me to build GroundControlAI: a zero-overhead, real-time observability daemon and cyber-cockpit for local AI agents.

Here is what I learned from reverse-engineering the local log streams, and how you can track your own agents.

1. The Goldmine Already Sitting on Your Disk

The first surprise was that these agent tools are already recording rich telemetry locally. They just dump it into scattered, format-incompatible log files:

• Claude Code (~/.claude/projects/**/*.jsonl): Logs exact prompt tokens, completion tokens, cache-creation tokens, cache-read tokens, reasoning tokens, and tool invocations (Bash, Edit, Write).
• Kimi Code (~/.kimi-code/sessions/**/wire.jsonl): Records precise Time-to-First-Token (TTFT) latency in milliseconds, stream decode durations, background task exit codes, and process IDs.
• Agy CLI (~/.gemini/antigravity-cli/brain/**/*.jsonl): Records multi-agent hierarchy graphs, planner deliberation thinking blocks, and subagent delegation handoffs.

The data was already there. It just needed an ingestion engine that didn't murder my CPU with heavy polling loops.

2. Three Surprises When You Actually Measure Agent Activity

Once I started aggregating this stream into a central SQLite database, three patterns immediately jumped out.

Surprise A: Prompt Caching Is Either a 90% Discount or a Silent Budget Killer

Modern models (like Claude 3.5 Sonnet) offer huge discounts for cached prompt tokens—down from $3.00/1M to $0.30/1M (a 90% discount).

When an agent stays within the cache lifetime window, multi-turn coding is remarkably cheap. But the moment you trigger a cache miss (e.g., passing a slightly modified system prompt or letting the 5-minute TTL expire), that 180,000-token repo context is re-billed at full price.

Seeing a live Cache Savings % counter on the screen completely changed how I interact with agents. I stopped killing and restarting sessions unnecessarily because I could literally see the cache hits saving $15–$25 per working session.

Surprise B: The "Deliberating..." Lie (Silent Exit Code 1 Loops)

Have you ever seen an agent show Thinking... or Deliberating... for two minutes straight?

Often, it's not thinking. It ran a bash command like pytest tests/, hit an ImportError (exit code 1), tried to fix it with an invalid argument, hit exit code 2, and is quietly spiraling in a retry loop.

By capturing tool_executions with their raw commands and exit codes, GroundControlAI exposes a real-time SRE stream. If you see a cluster of red [EXIT 1] pills, you know immediately that the agent is stuck in a circular loop and you can intervene before it burns 200,000 tokens on hallucinated fixes.

14:32:01  ▶ TOOL: [Bash] `pytest tests/test_normalizer.py`    [STATUS: EXIT 1]
14:32:05  ▶ TOOL: [Edit] `normalizer.py` (Line 14-38)         [STATUS: SUCCESS]
14:32:08  ▶ TOOL: [Bash] `pytest tests/test_normalizer.py`    [STATUS: EXIT 0]

Surprise C: File Hotspots & Circular Churn

When human developers edit code, they touch a file 1 or 2 times per task. When an agent edits the same file 12 times in 10 minutes, that file is a hotspot indicating brittle logic, failing test assertions, or conflicting instructions.

Tracking file operations (read, edit, write) per repository reveals instant code smells.

3. The Architecture: 0.00% Idle CPU with Linux Inotify

A telemetry tool that burns 15% CPU polling log files defeats the purpose of running lightweight local tools.

GroundControlAI uses the Linux kernel's inotify subsystem via Python's watchdog. Instead of scanning files or parsing entire 50MB .jsonl transcripts on every disk write, the collector daemon maintains an in-memory dictionary of byte offsets (f.tell()):

# collector.py (simplified concept)
file_offsets = {}

def process_file_incrementally(file_path):
    last_offset = file_offsets.get(file_path, 0)

    with open(file_path, "r", encoding="utf-8") as f:
        f.seek(last_offset)
        new_lines = f.readlines()
        file_offsets[file_path] = f.tell()  # Save offset for next turn

    for line in new_lines:
        parse_and_insert_turn(line)

Because it only seeks and reads the newly appended bytes, turn ingestion takes under 1 millisecond and idles at 0.00% CPU.

The data lands in an SQLite database running in WAL (Write-Ahead Logging) mode, allowing the collector daemon to write continuously while the FastAPI web server reads concurrently without lock contention.

[Agent Log Files]
(Claude, Kimi, Agy)
       │ (inotify kernel events)
       ▼
[Collector Daemon] ──(f.tell() incremental bytes)
       │
       ├──> [Path Normalizer] (auto-resolves worktrees & mono-repos)
       ├──> [Pricing Engine]  (calculates tokens & 90% cache discounts)
       │
       ▼
[SQLite DB (WAL Mode)]
       │ (live concurrent read)
       ▼
[FastAPI Cyber-Cockpit] ──> [http://localhost:8080]

(You can explore the full interactive architecture diagram online at *varunrai.github.io/GroundControlAI*).

4. Solving the "Worktree Sprawl" Problem

One subtle issue with modern agents (especially Google Antigravity / Agy CLI) is that they spawn isolated git worktrees for tasks (~/.ao/data/worktrees/my-repo/subagent-1).

If your dashboard keys projects by raw working directory, your single repository ends up split into 15 disjointed "projects" in the UI.

To solve this without hardcoded directory names, GroundControlAI includes a dynamic path normalizer. It inspects the directory structure:

1. If .git is a worktree file, it reads the gitdir: <path> pointer and resolves the main parent repository directly.
2. If it's a subpackage (apps/web, infrastructure/terraform), it rolls it up to the canonical project root.

The result is a clean, unified view per repository regardless of how many worktrees or subagents were spawned.

5. Running It Locally in 60 Seconds

GroundControlAI is fully open-source (MIT License) and packaged into a Docker Compose stack that mounts your local agent directories read-only:

Step 1: Clone the Repo

git clone https://github.com/varunrai/GroundControlAI.git
cd GroundControlAI

Step 2: Spin Up the Stack

docker compose up -d

Step 3: Open the Cockpit

Navigate to http://localhost:8080 in your browser.

The collector immediately scans existing logs to hydrate your historical stats, then transitions into live kernel-watching mode for any new agent turns.

Wrapping Up

AI coding assistants are no longer just fancy autocomplete—they are autonomous junior engineers running terminal commands, editing files, and making API calls. They deserve the same observability standards we apply to production backends.

If you're running Claude Code, Kimi Code, or Agy CLI, give it a spin:

• ⭐️ GitHub Repo: github.com/varunrai/GroundControlAI
• 🗺️ Live Interactive Architecture: varunrai.github.io/GroundControlAI

PRs and new agent log parser contributions are very welcome!