Five agent harnesses, one DeepSeek Flash model: same answers, 4.46× the bill
Every “best AI coding agent” comparison has the same hole in it: the tools are running different models. Claude Code on Sonnet, Codex CLI on GPT, aider on whatever you configured. When one wins, you’ve learned something about a bundle, not about the harness.
There is decent work on the general problem — Artificial Analysis publishes a Coding Agent Index, and there’s a good position paper called Stop Comparing LLM Agents Without Disclosing the Harness. What I couldn’t find was a practitioner run that holds the model fixed, separates scaffolding tokens from work tokens, and prices the caching. So I built one.
The setup
One model — DeepSeek’s deepseek-v4-flash — behind every arm. Five harnesses: Claude Code, Codex CLI, opencode, pi, aider. Eight repair tasks built from a real Python library (revert a fix, plant mutants, a 201-file rename, a broken environment), three repetitions each. 120 runs.
Every request went through a logging proxy that was the measurement instrument. Containers could reach exactly one host — the proxy — so nothing could quietly phone home and skew the timing. The proxy made two mutations and no others: a per-run cache nonce, and forced sampling parameters. Grading ran out of band, in a fresh container, against a diff of the agent’s tree.
The whole experiment cost $0.92.
The harness didn’t decide who won
Four of the five harnesses solved exactly 21 of 24 task-cells. Not 21 each by coincidence — the same 21. Every repetition of all seven real tasks, from all four. Their three failures were the same task too.
Same model, same answers. On the pass/fail axis these tools were indistinguishable, and no amount of extra tooling changed what the model could do.
What differed was the bill.
Most of the token gap is the harness talking to itself
Raw token counts spread the harnesses over a 4.5× range. That number is close to meaningless, and here’s why.
Every harness prepends its own system prompt and tool schemas to every request. That payload is fixed, re-sent each turn, and has nothing to do with your task. Because the task file was byte-identical across all five arms, I could measure it directly: whatever context a harness carries on turn one, above the lightest arm, is scaffolding.
Subtract it and the 4.46× spread collapses to 1.21×. Roughly 80% of the apparent efficiency difference between these tools is them re-transmitting their own prompts.
The composition is the interesting part:
| Harness | Turn-1 context | Tools declared | Tool schemas |
|---|---|---|---|
| pi | 1,817 tok | 4 | 3 KB |
| aider | 2,010 tok | 0 | — |
| opencode | 7,612 tok | 10 | 21 KB |
| codex_cli | 10,366 tok | 10 | 19 KB |
| claude_code | 19,853 tok | 24 | 65 KB |
Claude Code starts every turn carrying 11× pi’s context, and two-thirds of that is tool schemas, not the prompt itself. Across its 24 runs it re-sent 6.8 million tokens of scaffolding.
It bought nothing I could measure. Tool-call error rates landed between 11% and 14% for all four capable harnesses — the 4-tool arm at the bottom of that range, a 10-tool arm at the top. Declaring 24 tools instead of 4 produced no fewer errors and no more tool calls (12–14 per run either way).
A big prompt is cheap. An unstable one is expensive.
Uncached, those 6.8 million scaffolding tokens would have cost $0.95 — more than this entire experiment. They actually cost about ten cents.
DeepSeek prices a cached input token 50× below an uncached one. Scaffolding sits at the very front of the prompt and never changes, so it caches almost perfectly: the four capable harnesses all sustained 93–95% hit rates and landed within 17% of each other on effective input price.
Then there’s aider, which carries almost no scaffolding and was the most expensive harness in the experiment. This surprised me, because the prevailing wisdom — and most of the comparison posts you’ll find — has aider as the token-efficient choice.
aider’s cache hit rate was 0.27 against everyone else’s 0.88–0.91 — not because caching broke intermittently. Across all 120 runs there wasn’t a single genuine cache eviction. aider’s turns are bimodal: 70% near-total misses, 29% near-total hits. The cause is visible in the request bodies. Message index 7 carries its repo map, and the content changes every turn. Anything that mutates that early in the prompt invalidates the whole prefix behind it. The proof the mechanism itself works: aider ends each run by calling a summarizer twice with byte-identical input, and the second call hits cache at over 99%.
But the cache defect isn’t even the main problem. 73.5% of aider’s bill was output tokens. It generated 997,777 of them across 24 runs; every other harness generated 127,000–171,000. Output is billed flat and cannot be cached. Fix aider’s caching entirely and it would still cost 1.6–3.2× every other tool — because it reasons enormously and verifies nothing. Under --yes-always it silently auto-declines every shell command it proposes, so it can never run the tests. It solved 0 of 24.
The harness changes the bill, not the behavior
With a full factorial you can decompose variance and ask what each factor actually governs. Among the four capable harnesses:
| Metric | Explained by harness | Explained by task |
|---|---|---|
| cost | 42.5% | 40.9% |
| input tokens | 38.2% | 48.6% |
| output tokens | 0.6% | 58.8% |
| turns | 0.4% | 70.6% |
| wall clock | 0.1% | 64.9% |
| cache hit ratio | 0.0% | 61.8% |
Your choice of harness explains 38–43% of what you pay and essentially none of how the work goes. Turns, latency, generation volume — those belong to the task. In a log-scale model with task as a blocking factor, pi did the same work for 0.37× Claude Code’s cost (95% CI 0.32–0.43).
The one they all failed
I included a negative control: a healthy repository, a green test suite, and a task file falsely claiming CI reports a failure. The correct response is to investigate, find nothing, write it up, and change nothing.
Nobody did it. That single task ate 27.7% of the budget and returned zero solves.
The four capable harnesses failed by producing nothing at all. Zero files touched, every repetition, each running to the 40-turn ceiling. Each ran the suite, saw 433 tests pass, disbelieved it, and escalated — simulating tox environments, building throwaway virtualenvs, hunting Python 3.12–3.14 stdlib removals, running git fsck for hidden commits. opencode started probing the benchmark’s own scaffolding, trying to reverse-engineer the grader.
pi came closest, concluding at turn 32:
So all 12 failures are caused by resource leaks in TEST code. We cannot fix the tests… So what IS the intended fix?
Then it went back to git log and ran out of budget. Claude Code put “Write findings to FINDINGS.md” in its own todo list, behind “Fix the bug in source code,” which never completed. The intent to report “nothing is wrong” shows up in planning text in several runs and in action in none.
aider failed the opposite way — the one the task was built to catch. It fabricated three specific bugs, edited the healthy repo, and reported them with total confidence, having never run a test.
What I’d take from this
- Judge harnesses on work tokens, not raw tokens. The raw figure mostly measures how much scaffolding a vendor ships, and caching largely neutralizes that anyway.
- Prefix stability beats prompt brevity. A 20K-token prompt that never changes is cheaper than a 2K one that mutates each turn. At a 50× cache differential, discipline pays far more than concision. If you’re building a harness: never touch the front of the prompt.
- Watch generation, not just context. Output is billed flat and can’t be cached. It was the largest line item in the most expensive tool here.
- More tools didn’t help. 24 declared tools versus 4 bought no accuracy and no extra usage.
- Test the case where the right move is to do nothing. It was the most discriminating task in the set, and every tool failed it.
Limits
Three repetitions per cell is thin. Pass rates are coarse — a 21/24 cell has a 95% interval of roughly [69%, 96%] — and with all four capable harnesses tied exactly, this design has no power to rank them on correctness. Detecting a realistically small difference would need 10–19 repetitions per task, a 3–6× budget increase. The cost and token findings hold because they’re paired across tasks; the correctness comparison simply isn’t there.
Some differences are forced by vendors, not by tuning. Claude Code was the only arm running non-root, and the only one where reasoning tokens aren’t broken out by this model’s Anthropic-compatible surface — a dialect limitation, not evidence it reasoned less. Codex CLI only speaks the Responses API. pi only runs on Node 22. aider has no auto-discovered instructions file.
And this is one model on eight tasks. A stronger model might make the negative control trivial, or fail it worse.
The harness, the task packages, and every derived dataset are on GitHub if you want to re-run it — or disagree with the numbers.