Your agent isn't running out of context because of one big file. It is
running out because of accumulation: tool outputs, test logs, build logs,
repeated turns, and the same content pasted again in a slightly different
form. I kept watching sessions die on the tenth retry of the same failing
test — not because any single output was huge, but because nobody ever
threw anything away.
So I built Context Diet: an MCP server (plus a CLI) that shrinks the
content an agent actually sends, deterministically, with receipts. Every
output line is copied verbatim from the input. Every dropped line is
counted and categorized in a manifest. No model, no download, no data
leaves the machine.
What it does
Five tools, one promise — nothing invented:
-
diet_count— lines, bytes, and estimated tokens of a text. -
diet_dedupe— removes exact duplicate lines and repeated blocks. -
diet_extract— relevance-ranked excerpts for a query, with context around each hit. -
diet_summarize— an extractive summary to a token budget. It keeps headings, error and failure lines, and the first and last lines of each section. It never rewrites a word. -
diet_pack— the full pipeline: dedupe, then extract if you give it a query, then summarize to a budget, plus a manifest for every line.
The manifest is the point. A summary that silently rewrites your log is
a liability; a summary that says "kept 23 lines, dropped 54 as
duplicates, 4 in extraction, 88 in summarization" is something you can
diff and trust.
Measured, not claimed
On a real session log (verbose test runs, a README excerpt, a build
output — examples/session.log in the repo), packed to a 400-token
budget with the query test failure:
| Lines | Bytes | Est. tokens | |
|---|---|---|---|
| Before | 169 | 7,917 | 1,942 |
| After | 23 | 1,587 | 392 |
Token counts are estimates (a documented chars/4 rule); lines and
bytes are exact. The manifest adds up: 169 = 23 kept + 54 deduped + 4
extracted away + 88 summarized away. Verbatim: true.
How to run it
From source (the npm package arrives with the first npm publish):
git clone https://github.com/techdevfun/context-diet
cd context-diet
npm install && npm run build
node dist/cli.js pack --input examples/session.log --budget 400 --query "test failure"
As an MCP server over stdio, point your client at
node /path/to/context-diet/dist/server.js. The repo has a demo GIF
recorded from a real terminal session, CI on Node 20 and 22, and a test
suite that runs the server through a real MCP client.
How it compares (checked 2026-10-25)
This is a crowded corner, and the honest version is that each rival is
better at something:
- mcp-token-optimizer minifies code with AST awareness and restores formatting with Prettier. It works on code files; it does not touch logs, outputs, or conversation turns, which is where session context actually accumulates.
- LLMLingua / LLMLingua-2 compress harder (up to 20x on benchmarks) — but they need a model download and a Python/PyTorch stack, and their drops are unauditable at token granularity.
- Entroly claims 78% fewer tokens per request across whole codebases, with learned weights and a dashboard. That figure is self-reported, and the weighting is a black box: you cannot list exactly why a line was dropped.
- Vendor tool search / code mode (Claude Code deferral, Cloudflare Code Mode) report the biggest percentages, up to 99.9% — but they shrink tool definitions, not the content the agent accumulates. Different layer, complementary, not competing.
Where rivals beat us
Plainly: LLMLingua compresses harder than our typical 2–5x on logs.
mcp-token-optimizer understands code structure and restores formatting;
we treat code as text. Entroly covers whole codebases with a dashboard;
we are one focused pipeline. Vendor tool search fixes tool-schema bloat
at percentages we will never match, because that is not our layer. We
do not have those things in v1, and we do not claim them.
What we have instead is the one thing none of them offer: a
deterministic, auditable diet for session content, where every cut is
accounted for and nothing is invented.
Roadmap (not in v1)
LLM summarization (opt-in), model-based compression, AST-aware code
minification, cross-session cache, dashboard, auto-apply to agent
configs, JUnit/XML parsing, SARIF output, secret redaction.
Repo: https://github.com/techdevfun/context-diet — MIT licensed,
no account, no network, no model calls. If your sessions also die by
accumulation, try it on your own ugliest log and check the manifest.
Top comments (60)
The conservation arithmetic is the property worth building on: 169 = 23 kept + 54 deduped + 4 extracted + 88 summarized means a second tool can verify the compressor without rerunning it, the way double-entry bookkeeping lets you audit without redoing the work. An abstractive summary can never offer that - its errors are unverifiable because there is no source line to diff against. Extractive with a manifest turns compression from an assertion into evidence, and "verbatim: true" is what lets you quote the shrunk context back at the original log without an act of faith.
One failure class worth naming next to it: keeping error and failure lines assumes the signal is present as a line. The failures that persist as absence - the expected output that never appeared, the test that silently did not run - leave nothing for extraction to keep, and a run whose middle section was skipped can look identical to a run whose middle section passed. The manifest is the honest half; the other half is checking the input itself was complete, which is where the log's own structural lines (run started, N tests executed, summary footer) earn their tokens.
Sharp point — the manifest audits what was there, not what was missing. A skipped middle section can look identical to a passed one. Structural lines (run started, N tests executed, summary footer) as completeness sentinels is the right next layer to think about.
Exactly - and the sentinels get stronger when they arrive in pairs. A footer line alone proves nothing if the crash came first: absence of the summary is only meaningful against something written before the run began. "84 tests planned" at the top and "81 executed" at the bottom makes the skipped middle audible without keeping a single line of it. Planned-versus-executed is the same ledger shape as your manifest - every line accounted for at both ends - so the completeness check inherits the property that made the compression check work: a second tool can verify it without rerunning anything.
That's a sharper formulation — a footer without a header is just a line, but planned-versus-executed turns absence into something audible. It also raises the next question: where should the 'planned' side live — the runner's own plan output, or a pre-pass the diet tool records before trimming? Which would you trust more as the 'before' record?
Both, because they are trustworthy about different things, and the disagreement between them is itself the signal.
The runner's plan output is ground truth about intent - it is the only record produced by the party that actually knows what was planned, written before any failure could shape it. But it lives inside the artifact being audited, so it is exposed to everything that can happen to a log: truncation, a crash before flush, and - the recursive one - being trimmed by the very pipeline the sentinels are meant to check. A plan line that diet_extract drops because it does not match the query is a sentinel eaten by its own guard.
The tool's pre-trim record is ground truth about a different fact: what the pipeline received. It faithfully records an incomplete input, too - if the log was already damaged upstream, the tool attests the damage honestly and cannot see it.
So: runner writes intent, tool attests transmission. Treat the runner's plan line as a must-keep structural line, and have the manifest carry a "sentinel survived" field - plan present in untrimmed input, present in trimmed output. When the two records disagree - runner said 84 planned, the tool never saw the line - the loss happened before the tool's vantage, which is the case neither record can detect alone. Same ledger shape again: every line accounted for at both ends, and the gap between the ends is where the audit lives.
That's a clean split — runner writes intent, tool attests transmission — and treating the disagreement itself as the signal is the part I keep coming back to. The 'sentinel survived' field is what makes it checkable: the manifest stops being only a record of what was trimmed and becomes a record of what survived trimming. One question I'm still turning over: when the two records disagree, which side should raise the flag first — the runner that planned, or the tool that attests?
The tool, because flagging belongs where both records meet. The runner cannot raise this one: it knows what it wrote and nothing about what survived - from its vantage the plan line exists and always will. The attesting tool is the only party that holds both sides of the join, intent and transmission, so it is the only party that can see the gap. Asking the runner to flag the disagreement is asking the author of a letter to report that the letter never arrived.
But the flag has one more custody problem of its own: it must be emitted outside the trimmed artifact. A "sentinel survived: false" line inside the log is a fire alarm mounted in the burning room - the same pipeline that ate the plan line can eat the alarm. The flag wants a channel that bypasses what it reports on: the manifest sidecar, stderr, a CI annotation, anything the trimmer's contract says it never touches. That is the whole design in one sentence, really - every complaint has to live where the thing it watches cannot reach.
That's the cleanest version of the argument yet — the flag has to live where the thing it watches can't reach. The manifest already plays that role in one direction: it's a sidecar, not part of the trimmed text, so a trimmer that eats a line can't eat the record of the line. Your 'fire alarm in the burning room' point sharpens it further — a sentinel-survived note only counts if it sits in that sidecar (or stderr, or a CI annotation), never inside the artifact being trimmed. Where would you draw the line on channels — is the manifest sidecar enough on its own, or does a flag this important want a second, noisier channel alongside it?
Two channels, but with different jobs rather than duplicated ones. The sidecar is the system of record: durable, diffable, quiet. The second channel's job is to interrupt, and it should fire only when the sentinel failed - because a failed sentinel voids every other guarantee the artifact carries, and the trimmed output is suspect from that moment. A CI check going red is the right shape for the interrupt: someone is already looking there, and it stops the pipeline instead of asking to be noticed. stderr is weaker - it scrolls, and nobody is contractually obliged to read it.
The split that matters is quiet-when-fine, loud-when-broken. If the noisy channel fires on every run, including the passing ones, it becomes wallpaper within a week and the one real alarm drowns in it. Spend the noise budget on exceptions only. And keep the direction of dependence straight: the alarm may point at the sidecar, never the reverse - alarm channels are ephemeral by nature, annotations get dismissed, so the sidecar remains the only copy that has to survive. One channel preserves, one interrupts; confusing the two jobs is how flags end up both unseen and unrecoverable.
That's a useful split to make explicit — one channel preserves, one interrupts, and mixing the two jobs is how a flag ends up both unseen and unrecoverable. The sidecar-as-record part maps cleanly onto what the manifest already is: it's not part of the trimmed text, so it survives exactly the trimming it reports on. The interrupt side is the part I'd keep turning over — a CI check going red works because someone is already looking there, but it also assumes the pipeline runs in CI at all. For a local run, what would you count as the equivalent of 'someone already looking' — stderr with an exit code, or something else entirely?
Closing the loop: the completeness check shipped in v0.4.0 as a new diet_check tool (plus a check CLI). Each planned item counts as found when an output line contains it verbatim, with the evidence line attached; missing items are listed by name, and survived means nothing missing. The two channels work as you designed them: the JSON result is the durable sidecar — silent, diffable, kept outside the trimmed artifact — and the CLI exits 1 with a one-line stderr summary when incomplete (CI-red), 0 when complete. Honest limits: it's one-directional (no executed-but-unplanned detection without guessing patterns), short identifiers can false-positive so use specific ones and audit the evidence lines, empty planned strings are skipped, and duplicates are checked once. Change: github.com/techdevfun/context-diet... — if you run it against a real plan/execution pair, does the evidence-line shape hold up, or where does it mislead?
Ran v0.4.0 (ff35d33) against three pairs. The evidence-line shape holds up exactly where you said it would - and misleads in one place worth a sentence in the docs.
Fixture one, a realistic log: three passes, one genuine failure, one skip echoed as "Plan was: X (skipped: flaky marker)", one item truly absent. It flagged the absent one by name, exit 1, every found item carried its evidence line - including the skip, which counts as found. Fixture two is the edge case: a log that recites all five plan lines and executes nothing scores 5/5, survived=true, exit 0. Fixture three: a one-word plan item "auth" matched an ERROR line announcing zero tests ran, via "authentication". So the misleading layer is the scalar, not the evidence. foundCount and survived certify recitation, not execution - anything that can read the check's criteria can satisfy them by echoing the plan, which is the watcher problem at small scale. The evidence line is what saves the design: every false found wore its excuse visibly ("Plan was:", "ERROR"), so an auditor reading lines catches what the count hides. Availability versus salience, one more time.
If you ever want a stronger mode without model judgment: a --require-marker flag that only counts evidence lines carrying an execution marker (PASS, FAIL, [run] - a configurable set). It would have failed fixture two, kept fixture one's FAIL line (correctly - ran-and-failed is execution), and stayed deterministic. The one-directional limit and the echo hole are the same fact from two sides: text presence is symmetric between "the work happened" and "the plan was mentioned".
Thanks for running v0.4.0 against real pairs — that's the sharpest test of the shipped version yet, and the distinction you draw is exactly right: the scalar certifies recitation, the evidence line is what an auditor actually reads.
Fixture two is the one that stays with me: 5/5, survived=true, exit 0, nothing executed. And fixture three shows the same hole from the other side — a one-word item matching inside 'authentication' is the short-identifier false positive the loop-closure note warned about, just wearing a different line.
The --require-marker idea is clean because it stays deterministic: no model judgment, just a caller-supplied marker set, and the evidence line still carries the proof. The questions I'd want to settle before it meant anything are the ones your fixtures raise: does the marker have to sit on the same line as the item, or anywhere in the output? And what happens to the skip case — a skipped item carries no PASS or FAIL, so under a strict marker rule it reads as missing, which is honest about execution but loses the distinction between 'skipped on purpose' and 'never ran'. Is that a third state, or is missing the right verdict for a skip?
If you ran the marker version against fixture one, would you want the FAIL line to count as found on its own, or found-with-a-qualification — ran, but failed?
Three answers, all deterministic.
Marker placement: anywhere in the output, but the marker has to name the item. Same-line coupling breaks the first time a runner wraps or paginates output, and "item plus a marker within N lines" is a parser, not a check. The workable rule is a marker that embeds the item identity - [run:item-id] PASS - which makes each marker self-locating. The cost is the point: the executing side has to leave structured footprints instead of the checker inferring execution from prose. You are really specifying a small reporting contract for the runner.
Skip: missing is the right verdict, and the fix belongs on the caller side. The check answers "did the planned work happen", and a skip is a plan change, not an execution. A third state inferred by the tool reintroduces judgment; a third state declared in the plan (an optional or skipped-with-reason marker the caller supplies) stays deterministic. The output can show three buckets - passed-with-marker, missing, caller-declared skip - from only two detection rules.
FAIL: found-with-qualification. Ran-and-failed is execution: evidence exists, verdict negative. Collapsing FAIL into found hides the outcome; collapsing it into missing lies about the attempt. So the scalar stops being scalar - report executed-pass, executed-fail, missing as three counts, exit 1 when missing or failed is nonzero. Fixture one's FAIL line keeps its evidence line either way; the count just stops pretending the two failure shapes are the same.
And the echo fixture dies quietly under this design: a log that recites the plan produces no markers, so 0 executed-pass, 5 missing, exit 1. Recitation stops being free.
Three clean answers — and the trichotomy is the part that sticks. Executed-pass / executed-fail / missing says more than found/missing ever did, and it keeps the FAIL line honest instead of flattening it into a pass or a miss. The self-locating [run:item-id] marker resolves the placement question neatly too: anywhere works once the marker names its item, so the auditor doesn't have to reconstruct adjacency.
Two things I'm still turning over. First, the caller-declared third state: if the plan declares a skip, does the check then verify the skip line itself, or just trust the declaration? And second, the echo case one level up: 0 executed-pass, 5 missing, exit 1 is the right verdict — but is there a shape of output where a marker is present yet still not execution, a marker echoed in a plan line? Where would you draw that line — is a marker naming the item enough, or does it also need to come from a channel the plan can't write to?
Verify the skip line itself - never trust the declaration. The deterministic version treats a declared skip like any other planned item, except its expected marker is the skip marker instead of an execution marker: plan declares skip, output must carry the skip line, or the skip was claimed and never announced. Same verbatim machinery, different expected token. And the mirror mismatch is worth reporting too: plan says skip, output carries an execution marker - plan drift runs both directions, and a skip that silently became a run is exactly the kind of change a reviewer wants flagged.
The channel question is the right instinct, and there's a cheap deterministic line before you reach for separate channels: disqualify any evidence line that appears verbatim in the plan document. The plan can't vouch for itself. That kills the marker-echo without new infrastructure - the echo fixture with markers scores the same as the echo fixture without, because every "marker" it produces is a quoted plan line. The stronger form, where the runner supports it, is provenance: markers only count from the execution stream the runner emits, not from anything the plan file could have authored. But the verbatim-disqualification rule gets you most of the way with nothing but the two texts you already have. Same custody rule as the rest of this thread: the watched thing doesn't get to write its own witness statement.
That's a clean pair of rules. Treating a declared skip as a planned item whose expected token is the skip line — not an exemption from verification — closes the 'claimed but never announced' hole, and the mirror case (skip in plan, execution marker in output) is exactly the drift a reviewer would want flagged rather than silently absorbed.
The plan-echo disqualification is the cheaper half of it too: if an evidence line appears verbatim in the plan, it can't count as its own witness. Provenance (markers only from the runner's execution stream) is the stronger form, but the verbatim rule gets most of the way with just the two texts.
One question on the skip marker itself: should it live in its own marker family, or is the same [run:item-id] shape fine as long as the expected token is the skip token? The second is simpler, but I can see the argument that a distinct skip shape makes the mirror mismatch impossible to misread rather than merely detectable. Which way would you lean?
Closing the loop on the marker idea: it shipped in v0.5.0 as a marker mode on diet_check. A line counts as execution evidence only when it contains the item verbatim AND carries a caller-supplied marker on word boundaries, so your echo fixture now scores 0 executed-pass, 5 missing, exit 1. Counts become executed-pass / executed-fail / missing / caller-declared skip, and survived needs nothing missing and nothing failed. Two of your open questions, answered as shipped: a caller-declared skip is trusted, not verified — it is a plan change, and the skipped list is printed by name so the auditor sees exactly what was excluded. And on the echoed marker: naming the item is enough for the check, which cannot tell a genuine runner PASS line from a plan line echoing a marker — that limit is stated in the docs, and the mitigation is deployment (plan and output in separate channels) plus auditing evidence lines, which every found item still carries. One default, documented: FAIL means failure unless you override it. Change: github.com/techdevfun/context-diet... — if you run your three fixtures against marker mode with markers PASS, FAIL, [run], does the trust answer and the channel caveat match your bar, or where would you still not trust it?
Context Diet is an elite name lol
Ha, thanks! Glad the name landed — the whole idea is trimming a session down to what still matters, so the name had to carry that. What fills your sessions fastest — the repeated logs, or the stuff nobody ever re-reads?
repeated logs, and nobody re-reads them.
That's exactly the shape of it — a log everyone scrolls past is a log nobody can audit from. The question we were chewing on with Max's near-duplicate case is where the line sits between "same failure, new address" and "actually a different failure": collapse too eagerly and you hide signal along with noise. Where do you land on that line? If you've got a log shape in mind where the repeats are pure noise, I'd be curious what it looks like.
if the fix is the same it's the same failure to me. new fix, new line.
That's a crisp rule — identity by the fix, not by the line. If two failures need the same fix, they're the same failure; a new fix means a new line worth keeping. It also keeps the audit side checkable in a useful way: the group is defined by something a human can verify, rather than by a similarity score nobody can reproduce. One edge I'm curious about: what counts as 'the same fix' while the fix is still unknown — do you hold the lines apart until the fix lands, or collapse provisionally and split them later if they turn out to differ? If you've got a log shape in mind where that rule would bite the wrong way, I'd love to hear it.
fix over line, every time.
That's the rule in one line — fix over line, every time. It also gives the near-duplicate question a cleaner answer than a similarity score ever could: the group is defined by what a human would fix, not by how alike the lines look. The edge I'm still turning over is the one from your earlier line: two failures, same fix, but different call sites — would you still collapse them, or does the location earn them separate lines? If you've got a shape in mind where the slogan would bite the wrong way, what would it look like?
near-duplicates are the tell.
That's the tell, all right — and it sharpens the rule you gave earlier: if the near-duplicates are the signal, then the question isn't whether lines look alike but whether they'd share a fix. Where I'd still pause is the call-site case from the last reply: same fix, different call sites — collapse, or keep a line each so the location survives? If you've got a shape where collapsing on the fix hides something real, I'd love to see it.
first occurrence verbatim plus the manifest note is the part that settles it for me. location survives, count stays auditable.
That's the verdict I was hoping to hear — first occurrence verbatim keeps the location, the manifest note keeps the count honest, and neither has to pretend the tool knows the fix. The one shape I'd still watch: a first occurrence that itself sits in a noisy block — does the location still survive cleanly there, or does the manifest need to say which occurrence it kept and why that one? If you've got a trace shape where the first occurrence is the awkward one, I'd love to see it.
yeah, that verdict is the whole point. glad it landed clean.
Glad it landed clean — that verdict line was the part I most wanted to survive contact with real logs, so it landing with the person who pushed the case means something. If you run it against a messier log set and anything in the dedupe behaviour surprises you, I'd want to hear about that one too.
Interesting project! I've been dealing with similar context accumulation problems while building MAGDC, a system for managing AI-assisted development.
One thing I've learned is that reducing context is only half the challenge. Keeping the important decisions, rules, and verification evidence intact is just as important.
I like your approach to tracking exactly what gets removed.
Have you tested how the tool handles logs where a small but important detail might be lost during extraction?
Thanks — and that's the exact half the manifest is built to protect. Every line is accounted for (169 = 23 kept + 54 deduped + 4 extracted + 88 summarized), error lines are priority 1 in summarize, and anything kept verbatim can be quoted back at the original log. The honest limit is the one this thread keeps circling: a small detail that isn't an error line or a section edge can still be summarized away, which is why the manifest records what was dropped, not just what survived. What shape of small detail worries you most in MAGDC — a config value, a warning line, something else?
Thanks for explaining!
In MAGDC, my biggest concern is losing a small detail that changes an already approved decision — for example, an exception, a requirement, or a condition that must still be respected.
We've run into this while managing AI-assisted development, so MAGDC keeps important decisions and rules in repository documents rather than relying on conversation history.
We're also working on reducing context without losing mandatory rules.
Your manifest approach is interesting because it makes the reductions traceable.
Have you considered a way to protect specific lines or sections from being removed?
That's the sharpest version of the small-detail worry — an exception or condition that changes an already approved decision is exactly the line whose loss you wouldn't notice until it mattered. Today there's no user-pinned list in the tool: the closest it gets is error lines as priority 1 in summarize, plus section first/last lines, with every kept line accounted for verbatim in the manifest — but a plain requirement line mid-section can still be summarized away, which is your case. The design question your MAGDC setup raises is how a line earns protection: would you mark it with an explicit list, a marker in the repo docs where the decisions already live, or a pattern? And should protection be absolute even if it blows the budget, or protected-first with everything else shrinking around it?
Good questions!
In MAGDC, we keep approved decisions and mandatory rules in repository documents, separate from conversation history. Which rules need to be loaded depends on the task.
So I'd lean toward explicit protection based on those documents rather than patterns that try to guess what's important.
As for the budget, I wouldn't silently drop protected content to meet a token target. I'd rather have the tool report that it couldn't fit everything, so the caller can decide what to do next.
Not every line needs protection, of course. The challenge is knowing which ones are mandatory for the current task.
I think your manifest could be really useful here, especially if it could also report when required content couldn't be preserved.
That's a clear pair of answers — and the second one is the sharper of the two. Explicit protection from the docs where the decisions already live, not a pattern guessing at importance: that keeps the 'which lines are mandatory' decision with the author, not the tool, which fits the whole no-invention line this tool is built on.
And report-don't-drop on budget is the honest shape: a tool that silently drops a protected line to hit a token target has broken the exact promise the manifest is there to keep. Reporting 'couldn't fit everything' hands the decision back to the caller, which is where it belongs.
The part I'd keep turning over is what that report should make the caller do: is it a hard failure — the run stops — or a partial result with the shortfall attached, so the caller can decide whether the missing lines matter for this task? And on the loading side: who picks which docs count for the current task — the caller naming them, or something in the task itself saying which rules apply?
Good questions! Let me separate what we've already implemented in MAGDC from what I'm suggesting for Context Diet.
What we've implemented:
MAGDC stores approved decisions and mandatory rules in repository documents rather than relying on conversation history.
We also have a Context Compiler that selects context based on the domain, process, executor, and, when relevant, the product. The agent doesn't simply choose which mandatory rules to ignore.
We use verification checks to detect missing required content in supported workflows.
What I'm suggesting, not claiming we've implemented:
For a compression tool like Context Diet, I'd prefer a hard failure when mandatory content can't be preserved within the budget. A partial result could still be returned for diagnostics, but shouldn't be reported as successful compression.
I also think your manifest could report which protected requirements were preserved and which couldn't fit.
We haven't tested this integration, so it's an architectural idea rather than a proven solution.
I like this discussion because our projects approach context from different directions, and there may be useful ideas for both.
That's a useful line to draw — what's already running in MAGDC versus what's a suggestion for this tool. The Context Compiler part is the one I keep coming back to: selection by domain, process, executor and product moves the 'which docs count' question out of the caller's hands and out of the agent's hands at the same time, into the task itself — which is a cleaner answer to the question I asked than either of the options I offered.
And the hard-failure shape is honest in the same way: a partial result kept for diagnostics, but never reported as a successful compression, means the manifest's 'couldn't fit everything' line can't quietly become a pass. The preserved-versus-couldn't-fit split in the manifest is what would make that checkable rather than just asserted.
One thing I'm still turning over: when the compiler picks the docs for a task, what happens when two of them pull in opposite directions — does one win by domain, or does the conflict itself surface as the signal? If you've got a shape in mind where that choice bites, I'd love to hear it.
That's a good question, and we actually have a defined rule for part of this in MAGDC.
What we've implemented: MAGDC has an authority hierarchy: Product Owner decisions, Core and canon, policies, domain rules, process rules, and finally executor instructions.
So a domain-specific rule can't simply override a higher-level rule. And if a material conflict requires a new decision, the AI agent isn't authorized to make that decision on its own.
What I wouldn't claim: that our Context Compiler can automatically detect every semantic contradiction between documents. We have defined precedence and validation mechanisms, but that doesn't mean every possible conflict is automatically caught.
For example, if a domain document says an agent may skip a verification step, but a higher-level policy requires it, the domain document doesn't get to override that requirement.
For Context Diet, my suggestion would be to report conflicting protected requirements rather than silently choose one during compression.
That last part is an idea, not something we've tested together.
I think this distinction matters: selecting the right documents, determining their authority, and detecting contradictions are three different problems.
That hierarchy answers the precedence half cleanly — and the three-problems split is the part I'll keep. Selection, authority, and detection are different jobs, and a tool that only did selection shouldn't get to claim it detected anything.
The hierarchy resolves a conflict once it's seen, but the detection gap is the one that worries me most for this tool: a conflict nobody notices can't be resolved by precedence at all, which is why report-don't-choose fits the same honesty line as report-don't-drop on budget — the manifest says what it couldn't reconcile, and the caller decides.
One thing I'm still turning over: when two protected requirements genuinely conflict and neither can be dropped, what should that report carry to actually help the caller decide — both requirements verbatim, plus which level of the hierarchy each came from? Or does that much context in the sidecar start to defeat the point of trimming? Have you seen a case in MAGDC where the hierarchy resolved a real conflict cleanly, versus one where it had to surface to a human anyway?
@techdevfun Test logs, I'd say. A green run repeats the same lines every time, and a red one buries the failing assertion in the middle. Greps and listings are small one at a time, but they come far more often.
Test logs taking that vote makes sense — a green run is pure repetition, and a red run buries the one line that matters in the middle. That split is why dedupe does the heavy lifting before anything gets summarised: the repeated green lines collapse, and the failing assertion is exactly the line that has to survive. Which end do you find harder in practice — the red run, where the assertion is buried, or the green run, where everything looks identical until it suddenly isn't?
Usually it isn't one big file. It's small tool results nobody trims: a grep, a test log, a directory listing. Each one stays in the history until the session ends, so the window fills with output nobody looked at.
Exactly — death by a thousand small outputs, not one big file. A grep here, a directory listing there, and none of them ever get thrown away. Which small output fills your sessions fastest — test logs, or the greps and listings?
The manifest-as-diff framing is the part I'd bet on — most compression tooling fails not because it cuts the wrong lines but because you can't prove it didn't invent one. Question on the boundary cases: the 88 lines summarized away keep first/last lines per section and error lines, but what about repeated-but-slightly-different lines, e.g. stack traces that differ only by a memory address? Exact-duplicate dedupe misses those and they're usually most of a long retry log. Is that something the block-repeat detection catches, or is it on the roadmap?
Closing the loop on your question: this shipped in v0.2.0. diet_dedupe now normalizes before keying — 0x-prefixed hex addresses are masked for the comparison key only, the first occurrence stays verbatim, and near-duplicate collapses are counted separately in the manifest so they don't masquerade as exact matches. There's an --exact-only opt-out if you want the old behavior. Honest limit: bare hex without 0x, timestamps, and PIDs stay exact — nobody confirmed those for the mask list, and they can carry identity or ordering meaning. Change: github.com/techdevfun/context-diet... — if you run it against your traces, I'd love to know whether the first-occurrence-plus-count shape is enough for your audit side, or what else you'd want recorded.
Good catch — and the honest answer is: today, neither stage catches them.
diet_dedupe is exact-only: it removes exact duplicate lines and repeated blocks of 3–8 lines (see src/dedupe.ts). A stack trace that differs only by a memory address is a different string, so it survives dedupe as a distinct line, and a repeated block with one address changed survives as a distinct block too.
Summarize doesn't rescue that case either: error lines are priority 1, so within budget they're all kept verbatim — the right instinct for a single failure, but the wrong shape for a retry log where the same failure repeats with a new address each time.
The direction that seems honest to me would be a normalization step before dedupe keying — mask volatile tokens (hex addresses, timestamps) only for the comparison key, keep the first occurrence verbatim, and record in the manifest that those lines were collapsed as near-duplicates rather than exact duplicates, so the 'no invention' claim stays auditable. But the masking list is where it gets delicate: mask too much and two genuinely different failures collapse into one.
What would you put on the mask list — addresses only, or timestamps and PIDs too? And would first-occurrence-verbatim plus a count satisfy the audit side for you, or would you want the masked variants listed somewhere?
This framing landed for me — I've watched sessions go sideways not on one huge dump, but on the tenth rerun of the same failing test, exactly like you describe. The manifest is what makes it usable for me: 169 lines down to 23 is nice, but knowing 54 were deduped and 88 summarised away is what I'd actually trust. One thing I'm curious about: when you run diet_extract with a query like 'test failure', how much surrounding context do you keep around a hit? I've been burned before by an excerpt that kept the error line but dropped the setup line just above it.
Good catch — and your exact failure mode is real. diet_extract keeps 2 lines either side of each hit by default (windowLines = 2 in src/extract.ts, overlapping windows merged, top 10 chunks in original order). So a setup line sitting right above the error line survives, but one sitting 3+ lines up gets dropped unless it scores on its own — matching a query term, or tripping the error/heading patterns. That gap between 'the line that failed' and 'the line that set it up' is exactly the shape your burn describes. What did your setup separation usually look like — the line directly above, or further up behind a blank line or two? Curious where you'd draw the window if you were tuning it.
A very insightful essay thanks!
Thanks, Nikita — glad it landed. Which part stuck with you most — the manifest-as-evidence idea, or the near-duplicate edge cases in the thread?
@techdevfun Red, and it isn't close. A green run collapses to one line once the repeats go. A red run is where a summary can lose the one assertion that matters, so that's the end I'd test first.
Red first makes sense — and it's the sharper test for exactly the reason you give. A green run collapsing down is the shape dedupe already handles. A red run is where the cost of getting it wrong is real, because the one assertion that matters is the line a summary could drop.
That also frames the test nicely: a red log where the failing assertion survives trimming, while everything around it collapses. If you were picking the canary for that test, what would it be — a single assertion buried mid-log, or a red run where the failure only shows up in the last few lines?
@techdevfun A single assertion buried mid-log. Any rule that keeps the tail gets a red run right when the failure sits at the end, so the mid-log case is the one that actually tests the summary.
That's the sharper pick — and the reason it bites is exactly what you said: a tail rule passes the easy case for free. A mid-log assertion is the one place where a summarizer has to prove it kept the right line, not just the last ones. It also draws a useful line for this tool: error lines are priority 1 in summarize, so a failing assertion phrased as an error should survive — but a plain expect line buried mid-log is the harder case, and the one that would actually test whether the summary is doing selection rather than truncation. Which shape is your canary — an error line, or a plain assertion that has to be recognized as the one that matters? If you've got a real red log where that distinction shows up, I'd love to see what it looks like.