I benchmarked two agent token-compression tools on real data — the higher ratio was a trap

TL;DR — I measured two "compress your agent's tool output" tools on my own data. headroom wins on JSON (44% smaller, lossless at every array size I tested — I couldn't trigger its documented sampling fallback, though it exists) and keeps more detail on logs and grep too. RTK posts the biggest ratios on grep and logs — but they're lossy: it drops the specific line the raw number hides. Where RTK wins cleanly is git, find, and status. Running both together saved the most tokens and tied RTK for the worst parity — you pay more compute for the same error rate. The answer isn't "pick one," it's route by content type. Numbers, harness, and the places I under-tested are all below.

If you run a coding agent — Claude Code, Cursor, something in-house — your token bill isn't mostly your prompts. It's tool output: kubectl ... -o json, verbose logs, git diff, grep dumps that scroll for pages. So there's a niche of tools that compress that output before it reaches the model. Two lead it:

• RTK — a per-command CLI proxy. It rewrites git status into rtk git status and emits a compact, hand-tuned version of each command's output.
• headroom — a content-aware compressor (library / proxy / MCP). It sniffs the content type — JSON, logs, code — and crushes each one differently.

Both promise same answers, far fewer tokens. I don't trust that from a vendor benchmark, because it's always measured on the vendor's favorite input. So I measured it on mine, with two questions in order: how much smaller? then do the answers survive? Most write-ups stop at the first. The second is the one that changed my mind.

The setup

Real tool output from a live homelab session: a Kubernetes pod list (-o json), a metasearch JSON API response, a systemd journal, git diff, git status, git log, a recursive grep, a directory listing, and a source-file read. Every arm is counted with the same tokenizer (tiktoken o200k_base) — compression ratios are stable across encodings, so the exact choice doesn't move the comparison.

One thing to pin down up front: I tested one-shot, non-reversible compression — headroom's library compress(), RTK's CLI output — scoring each answer against the compressed text alone. I did not enable headroom's CCR mode (more on that at the end), which is exactly the feature aimed at the parity question. Read the parity numbers as "no retrieval available," which is headroom's most conservative setting.

Stage 1 — how much smaller?

Weighted by token mass, headroom looks like a runaway winner:

But that total is a lie of averages — one big JSON blob is doing most of the work. Broken out by content type (splitting logs from the rest of the command output, because they behave differently):

* RTK isn't incapable of JSON — it ships structured handling for several commands. Its kubectl filter targets the default tabular output; my -o json variant bypassed it, so the filter never fired. Read this 0% as a fixture fact, not a tool limitation. † RTK passthrough on the file read. ‡ headroom has an AST-aware CodeCompressor — but it covers Python, JS/TS, Go, Rust, Java, C/C++, and Perl, not HCL (I checked the source). My one file was Terraform, so passthrough was expected: the 0% says nothing about code compression, because I never fed it a language it handles.

The real story: RTK owns structured command output (git, find, status — a hand-written summarizer per command); headroom owns data-JSON (reshaping the array into a schema header plus rows, where RTK's filter wasn't in the path) and also compresses logs and grep well. They win on different content. Stop here and you'd conclude "run both, bank 60%." Don't stop here.

Stage 2 — do the answers survive?

A compression ratio is vanity. What matters is whether the answer to your question is still in the compressed text. I first tried grading answers with a cheap model, but exact-token grading off a small model was too noisy to trust — so I measured the stricter, fully deterministic thing: is the ground-truth answer still present in the compressed output? If it isn't, the model provably can't answer; if it is, it at least can (presence is the ceiling on parity, not a guarantee — a value can survive but be reshaped into a form the model misreads). Nine needles across the fixtures:

(Running both — RTK then headroom — also retains 6/9, identical to RTK: RTK compresses first and throws the detail away before headroom ever sees it.)

Three findings:

headroom retained 8 of 9 — it keeps the JSON values, the error line, the grep match, the changed filename. The one thing it dropped was a single INFO-level log line. On JSON it was lossless in every test — and I pushed on that, because "lossless" is exactly the claim I'd distrust. headroom's own docs describe a sampling fallback for large arrays: keep ~30% from the head, 15% from the tail, 55% by importance score (plus errors and anomalies), and drop the rest when the budget is tight. So I went looking for it — synthetic arrays from 25 up to 8,000 items under a deliberately tight token budget. It never fired. Reshape alone — dropping the repeated keys and braces — got even the 8,000-item array under budget every time, so every item survived. That's a stronger result than "my fixture was small": I tried to reach the documented lossy path and couldn't. But it is there by design, so price the risk and verify at your array sizes and budgets.

RTK retained 6 of 9 — and its misses are exactly its high ratios. It passes JSON through, so it keeps those. But its 90–100% wins on grep and logs come from throwing detail away: the grep match and both log-line details are simply gone. Great when you want the gist; a silent failure when you needed that line. A high compression ratio is not a feature. It's a risk you have to price.

Stacking both is no better than RTK, and worse than headroom alone (6 vs 8). RTK compresses first, at command time; by the time headroom sees the output, the detail is gone. Best-ratio was not best-parity.

The decision: route, don't replace

"Replace RTK entirely" loses — but narrowly. Headroom wins or ties on JSON, logs, and grep, which is most of the surface area; the case for keeping RTK really rests on three git-family commands — git, find, status — where its compact per-command summaries do work headroom doesn't touch, plus the plain convenience of running as a command-time hook. "Run both everywhere" loses the other way — it costs answers on detail-sensitive output. The setup that holds up is a router:

• data-JSON (kubectl, cloud APIs, Terraform state) → headroom. Big, and lossless at the sizes I checked.
• grep and logs → headroom if you need the detail — it compresses them ~88–91% and kept more of my needles than RTK did — or RTK if you only want the gist (higher ratio, but it drops the specifics; this is exactly where a big number is a trap).
• git (diff / status / log), find, ls → RTK. Compact, structured, and it kept every needle I threw at it.
• detail-sensitive (a specific log line) → leave it raw, unless you turn on retrieval (next section).
• never stack both on the same detail-sensitive stream.

Shipping the JSON win without a proxy

headroom's parity-safe advantage in my tests was JSON — so I didn't need its proxy. I wrote a tiny filter, hrjson, that is JSON-only by construction:

kubectl get pods -A -o json | hrjson

Four gates keep it honest: input must parse as JSON or it returns byte-for-byte unchanged; the compressor must pick a JSON-aware transform (never the lossy text path); it must actually save tokens; any error passes through. A command-rewrite hook then appends | hrjson to anything emitting JSON and hands everything else to RTK. On the real fixtures: kubectl −44%, API JSON −27%, logs untouched, needles intact.

Where I under-tested (be skeptical here)

Fair criticism I want to surface rather than bury:

• I ran headroom non-reversibly. Its actual answer to "compression dropped the one line I needed" is CCR: compressed content goes to a store, and the model gets a headroom_retrieve tool to pull the original on demand. My one INFO-log miss and my "keep detail-sensitive raw" rule might both soften with retrieval enabled. I measured headroom's most conservative mode; the reversible mode is a fairer test I haven't run.
• The 0% ratio cells are fixture facts, not tool verdicts. RTK's kubectl filter targets tabular output — I confirmed live that rtk kubectl crushes the tabular form ~96% but passes -o json through untouched. And headroom's AST code compressor doesn't cover HCL at all, so passthrough on my Terraform file was expected — not a borderline miss. Neither 0% says "this tool can't do that."
• Small sample, deterministic grading. 9 needles, scored by answer-token retention rather than an LLM (I dropped the model grader as too noisy). Directional, not a paper — but with no model in the loop, the retention numbers reproduce exactly.

What I'd tell you to take from this

• Benchmark on your own data — a vendor's ratio is measured on a vendor's input.
• Measure parity, not just ratio — the highest ratio preserved the fewest answers.
• Name your mode — reversible vs one-shot changes the parity conclusion entirely.
• Compression tools are routers, not replacements — different content, different tool, sometimes no tool.
• Lossy summarization is fine until it isn't — know which outputs are "gist is enough" versus "I need the exact line," and never compress the second kind blind.