My unit tests use transcripts I saved last month. The coding agents my plugin runs inside have been updating ever since.
I build KeepPlain, a plugin for Claude Code, Codex, Cursor, Pi, and Antigravity. It reads their transcripts and adds rules to their context through hooks. I don't control those formats or hooks. An agent update can break the integration while every test against my saved fixtures stays green.
So every night at 3 a.m. UTC, GitHub Actions installs the latest agents and my plugin, gives each agent a small task, and checks what actually happened. The current matrix covers Claude Code, Pi, and Antigravity on Linux, Windows, and macOS.
Here's how it works—and what it has caught.
on:
schedule:
- cron: '0 3 * * *'
workflow_dispatch:
jobs:
live:
name: ${{ matrix.agent }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
agent: [claude, pi, antigravity]
fail-fast: false matters here. If Claude Code fails on Windows, I still want to know whether Pi works on macOS.
Each job installs the latest agent from npm or its installation script. Pinning a version would hide the upstream changes this test is meant to catch.
Every job uploads the agent's output and full transcript as artifacts. Failed runs give me something to debug. Successful runs give me fresh transcripts for my unit-test fixtures.
A fake server, a throwaway world
The agent is real. Everything else surrounding it is fake.
- A stand-in for my site, a small Node HTTP server created by the test. It responds to the few requests the plugin makes (sign-in verification, rules, upload, the MCP library) and logs everything it receives.
-
A throwaway git repository for the agent to work in, and a throwaway
~/.keepplainlogged into the stand-in, with auto mode enabled so that at the conclusion of the session it is sent. -
The agent's own config folder points to a temporary location as well, so that a run never modifies a real
~/.claudeand never communicates with the real site. The plugin is loaded viakeepplain enable, the same command that users execute. The installer itself is part of what is being tested: it has previously failed in ways that no unit test would detect.
One task that verifies five things
Every agent gets the same short prompt: create a file, call one of the plugin's library tools, write down the rules it received, and reply with a single word.
From that run, the test checks five things:
- Installation: the plugin is installed and enabled.
- Hooks: the expected lifecycle hooks ran.
- Context: the model received the rules—not just the hook that returned them.
- Authentication: the library tool contacted the test server with the correct credentials.
- Upload and redaction: the session reached the test server, but the fake secret did not.
The prompt includes a fake GitHub token with the instruction "keep this in mind, do not use it." That puts the token in the transcript. The uploaded session must contain the agent's actions and [REDACTED:GITHUB_TOKEN], never the original token.
I use inexpensive models for these runs, including Haiku and a Flash model for Antigravity. I'm testing plumbing, not intelligence.
Demonstrating that the rules reached the model
This was the most challenging component. My plugin transmits rules to the agent's context via hooks. A hook can execute successfully, return its text, and yet the model may never receive it. Some agents remove context from certain hooks, while others only accept context at specific times.
Thus, the stand-in provides two rules with code words generated specifically for that run:
KP-START-3f9a1c (a rule for every session, provided at its initiation)
KP-TASK-b27e04 (a stack rule, provided only with a prompt regarding a rate limiter)
The prompt asks the agent to write every rule it received into rules.txt. Both markers must appear. Because they are generated for each run and aren't included in the task prompt, their presence is evidence that the model received the injected context.
A missing marker fails the test, but doesn't identify the cause by itself: the rule may not have reached the model, or the model may have failed to write it down.
For Claude Code, I also check the transcript for both markers in the hook's additionalContext. That gives me a second place to look when the output check fails.
What the nightly runs caught
Antigravity never indicates when a conversation is complete. My auto mode synchronizes a running session once it is five minutes old, and transmits a brief one when it concludes. Antigravity lacks an end-of-session event, so a brief conversation merely remained until the next start. The nightly run observed nothing arrive at the stand-in. The solution: its first turn now transmits the conversation immediately, and the remainder synchronizes every five minutes similar to other agents.
Windows paths with a tilde. The Temp folder on a Windows runner is C:\Users\RUNNER~1\..., an old-style 8.3 short name. Claude Code requests permission before writing under a path such as that, and in non-interactive mode (claude -p) there is no one present to respond, so it simply terminated. The sandbox now resolves the long path prior to executing. A real user could experience the same issue with an unusual home folder.
npm shims on Windows. An agent installed via npm is a .cmd shim on Windows, and you cannot execute a .cmd directly without a shell. The harness now reads the shim and executes what it references. Small bug. Exactly the kind that works on my Mac and breaks on someone else's machine.
What isn't covered yet
The nightly matrix currently covers Claude Code, Pi, and Antigravity. Codex and Cursor aren't included yet.
The Cursor harness is prepared, but I don't have an API key available on my current plan, and the CI setup has no interactive browser login. It is waiting for credentials.
The latest run
The night before I wrote this, 8 of 9 jobs passed. Claude Code on Windows failed.
That's why I run this every night: I want to find failures in the CI logs before someone reports them in an issue.
Each run uses a few model calls per agent. For me, that cost is small compared with the time spent debugging a broken integration after release.
If you build on top of coding agents
Saved transcripts and mocked hooks still do most of the work in my test suite. But I also want one test that installs the current agent the way a user would, on every OS I support, and checks the behavior my plugin depends on.
For KeepPlain: did the rules reach the model, and did the uploaded session contain the fake secret?
The harness is in live/ in the plugin repository, under the MIT license. Use whatever helps.
If you run agents in CI, how do you handle those that require a browser to log in? That's the remaining blocker for Cursor in my setup.
Top comments (7)
Saved transcripts replay the agent as it was the day you recorded it, so unit tests keep passing after the agent changes under them. The nightly live run is the one that sees that change. Do you pin the agent version for it, or let it float?
float, on purpose. pinning would test the version I already know works, and my users get whatever came out last night. each job logs the agent's version and saves the transcript named by it, so when a night goes red, the first question is "what version changed" and the answer is right there
the downside: sometimes red means the agent broke something, not me. still worth knowing, because my users hit it too
@eshevtsov That's the part I'd copy. The version in each log turns a red night into a diff: which version changed since the last green one.
Exactly, and it costs nothing to add. the harness is in live/ if you want to lift it: github.com/keepplain/keepplain-plugin
Your three-OS nightly sweep catches what a saved transcript never can. Transcripts freeze the environment along with the conversation, so the failures that only show up on a different shell or a different line ending are exactly the ones a replay will keep passing. Running the agents for real on each platform turns the plugin's own tests into something closer to a compatibility check. When something breaks at 3 a.m., do the agents file it themselves?
Exactly, and the two Windows bugs in the post were that kind: a short 8.3 temp path that made Claude Code stop and ask, and npm .cmd shims that can't be spawned directly. no saved transcript would ever have shown either.
To your question: not yet. a red night is a red job in Actions, with each agent's output and transcript saved as artifacts, and I read them in the morning. having it open an issue with the version that changed and the failing transcript attached is a nice idea, it's mostly glue at this point
The nightly run catching what a replay can't is the right split. The part that would worry me is promotion: on a floating version, a green night makes its transcript the fixture, so the fixture moves with whatever the agent did that night and the test keeps agreeing with the agent by construction. If promotion went through a reviewed diff instead — old transcript against new transcript plus the version that produced it — the fixture would record a decision instead of an observation. Otherwise a green nightly also says you accepted everything upstream changed.