The Harness Is 2.3% of Agent Cost. Keep Reading.

The model explains 92.8% of what a coding agent costs. The harness explains 2.3%. That’s the line people will screenshot from Runta’s new benchmark, and the conclusion they’ll draw is “the wrapper doesn’t matter, just pick a model.”

That conclusion is wrong, and the same post says why two paragraphs later.

What Runta measured

Runta and Baseten published FrontierHarness Eval v1.1 on Nov 17. A harness is the program around the model: Codex CLI, OpenCode, Pi, Cline CLI and so on. It decides the tools, the prompt and when the agent stops.

The setup, in their numbers:

  • 22 harness configurations x 5 open models x 62 terminal tasks x 4 attempts
  • 27,280 runs and 83B tokens, 76 times the size of v1.0
  • Every harness on default settings at maximum reasoning effort
  • All five models (GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse, Kimi-K3) served by Baseten

Runta sells the runtime each trial ran in, and Baseten was the only provider. Every figure below is theirs, not an independent audit.

Where the 92.8% comes from

Runta split the spread of cost per task across all 110 harness and model pairs into three parts:

Part Share What it means
Model 92.8% Differences between the five models
Harness 2.3% The part of a harness’s effect that’s the same with every model
Pair 4.9% The rest: how a harness behaves with one specific model (plus noise)

The model wins because list prices for these five models differ by 20 times or more. Any comparison across models is mostly a comparison of price lists.

The 2.3% only counts the part of a harness’s effect that’s identical for every model. The pair term is twice as big. With the model fixed, 68% of the harness variation depends on which model you paired it with.

Fix the model and the harness moves the bill 1.8x to 5x

This is the number I’d put on a slide instead:

Same model, cheapest harness vs most expensive harness (cost per task)

   lowest model:  1.8x
   highest model: 5.0x   (DeepSeek: Cline CLI vs OpenCode v1, +2.0 points)

   shuffled trials, same model:  1.2x to 1.8x  ("chance")

Runta shuffled trials between harnesses 300 times to see how big the gap gets from noise alone. All five real ratios beat every shuffle (p = 0.003). The spread is real.

And it doesn’t buy you much. With four of the five models, the cheapest harness per task was at most 0.4 points behind the most expensive one. Rank correlation between cost and pass rate ran from -0.39 to +0.22. Paying more for the harness didn’t reliably pass more tasks.

The cheapest harness changes when the model does

  • Codex CLI was 1st on cost per task with GLM-5.3 and 16th with Kimi-K3.
  • DeepSeek Harness: Creator went from 1st with Muse to 21st with GLM Flash.
  • OpenCode v1 was 1st with DeepSeek and 13th with Muse.
  • The median harness moved 11 of 22 places between models.

Runta checked that this isn’t rerun noise. Split the attempts in half and rank by cost: the two halves agree at 0.71 with the same model, and only 0.26 across different models.

Each model had a different best-value harness: Codex CLI with GLM-5.3, Pi with GLM Flash, ZCode with DeepSeek, Oh My Pi with Muse, Alma with Kimi-K3. Only the first one is solid. Codex CLI stayed on top with GLM-5.3 in 99.6% of resampled runs. The other four won between 25% and 53% of the time, so treat them as “one of several close options.”

If you benchmarked your harness on last quarter’s model, that result is stale. The easy path is to swap the model in a config line and keep the harness because it “already works.” This data says rerun the comparison.

Steps drive cost, not prompt size

A lean harness with a small prompt and few tools sounds like the cheap option. In this data it isn’t.

Runta compared four things against cost per pass (cost per task divided by pass rate): the size of the first request, the number of tools, the cache hit rate and the number of steps. Only steps tracked cost (rank correlation +0.51, p = 0.015). The other three were between -0.03 and +0.02.

The reason is boring arithmetic. Every step re-sends the whole context. A trial used 2.0 to 4.8 million input tokens on average, and the biggest first request was 21,365 tokens, under 1% of that. mini-SWE-agent has the smallest first request at 882 tokens, took 38 steps per pass, and landed 17th of 22.

The cost breakdown fits:

cached input   42%  ######################
output         33%  #################
other input    25%  #############

Shaving your system prompt is cheap to do and saves almost nothing. Cutting a step saves the whole context, again. I covered the cache side in Budget AI by Cost per Task, Not Cost per Token. Most harnesses already read 91% or more of their input from cache, so it no longer separates them.

Two thirds of the money goes to runs that failed

64.5% of the list-price cost went to trials that didn’t pass: 52.7% to failures, 11.8% to timeouts.

Worse, among the five best-value harnesses, 68% (Pi) to 82% (Codex CLI) of failed runs ended with a confident final answer, not at the time limit. Runta’s example is Codex CLI reporting “validated, including cancellation-cleanup tests” while the verifier shows the exact test it claimed to have passed failing.

An agent that fails loudly is cheap. An agent that says “done” on a broken result costs you the run, then a human’s time finding out. Runta suggests a check before the harness stops could save part of that spend. They didn’t test it, and neither have I, but it’s the first thing I’d try: run the task’s own tests before accepting “done,” and count those catches as their own metric.

The fine print

  • Open models only, one provider. Closed models and other providers’ cache pricing could reorder all of this. Runta says closed models are next.
  • Each pass rate has a 95% interval of about plus or minus 10 points, so a gap of a few points is a tie.
  • Audits flagged 216 trials for reward hacking, 163 of them online lookups. Those scored zero.

What to take away

  • Don’t read 92.8% as “the harness doesn’t matter.” It means the price list matters most across models. With your model fixed, the harness swings cost up to 5x.
  • Re-test the harness every time you change the model. Rankings moved 11 of 22 places at the median.
  • Measure cost per pass, and count steps per task. Steps predicted cost here; prompt size and tool count didn’t.
  • Log failed runs separately. They were 64.5% of spend, and most of them claimed success.
  • Use your own tasks. Runta’s own advice is a private set of your tasks, big enough to see past the plus or minus 10 points of noise.

results matching ""

    No results matching ""