FrontierHarness Eval v1.1
Choosing the right agent harness is as critical as the model itself, with costs per task varying by up to five times.
Deep dive
- Benchmarked 22 harnesses across 5 models: GLM-5.3, GLM Flash, DeepSeek V4.1 Flash, Muse, and Kimi-K3.
- Model choice drives 92.8% of cost variation, but harness selection dictates the remaining spread.
- Harnesses like Codex CLI were 1st for cost with GLM-5.3 but plummeted to 16th with Kimi-K3.
- High-cost harnesses do not correlate with higher pass rates, suggesting inefficiency is systemic.
- Most cost is consumed by failed trials, with 60%-82% of failed runs ending in a 'confident' wrong answer rather than a timeout.
- Steps per pass is the best predictor of cost, as each step re-sends the entire context.
Decoder
- Harness: The software infrastructure that manages the agent's interaction loop, tool calls, and environment state.
- Reward Hacking: When an agent learns to exploit the testing criteria or environment to achieve a 'pass' signal without actually solving the underlying logic.
Original article
TL;DR
We partnered with Baseten to build the largest-scale harness leaderboard using 83B tokens across 27,280 agent runs. We wanted to answer two questions: when you switch your model, should you consider switching the harness too? What is the best harness for each model?
Previously, our v1.0 FrontierHarness leaderboard ranked harnesses on a single model, Kimi K3.
V1.1 expands the benchmark to 22 harnesses × 5 models × 62 tasks × 4 attempts, 76× the size of v1.0.
Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience.
Each model had a different best-value (pass rate vs. cost) harness: Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse and Alma with Kimi-K3. Cost ranks moved far between models.
By cost per task, the median harness moved 11 of 22 places, and Codex CLI went from 1st with GLM-5.3 to 16th with Kimi-K3.
Result
Figure 1. Cost per task and pass rate of each harness
Each point is one harness: further left is a lower average cost per task, and higher is a higher pass rate. The orange line joins the harnesses that no other harness beats in both cost and pass rate. Select a model to filter all figures of this post.
Over all models, four harnesses are on the line: OpenCode v2 (lowest cost per task, $0.561), Pi, Codex CLI and ZCode (highest pass rate, 61.9%). The others cost up to 1.6 times as much and passed 50.6% to 60.2%.
Each card names the best-value harness for one model: the lowest cost per pass, unless another harness costs at most 30% more per pass and passes at least 5 points more often. This rule picked ZCode over OpenCode v1 with DeepSeek (25% more cost, 7.7 more points) and Alma over Oh My Pi with Kimi-K3 (9% more cost, 6.5 more points).
Why We Built FrontierHarness Eval v1.1
V1.0 ran nine harnesses with one model. Their pass rates were within 17 points of each other, but the median cost per pass differed by 17 times. We thought that this came mostly from the pair of harness and model: a harness that is cheap with one model can be expensive with another.
V1.1 runs each harness with five models.
What We Tested
Harnesses. 18 harnesses. The four DeepSeek Harness profiles have different tool sets, and OpenCode ran at two versions. Each counts separately: 22 configurations.
Models. Baseten served all five: GLM-5.3 (zai-org/GLM-5.3), GLM Flash (zai-org/GLM-5.3-Flash), DeepSeek (deepseek-ai/DeepSeek-V4.1-Flash-1017), Muse (meta-models/muse-spark-open) and Kimi-K3 (moonshotai/Kimi-K3).
Tasks. 62 terminal tasks. The time limits are those of the tasks, from 15 minutes to 8 hours.
Setup and Scoring
Configuration and Runtime
We ran the Harbor evaluation with each harness’s default settings, because we think this is closest to how people use a harness every day. Each harness used the same version and settings with all models, at the maximum reasoning effort.
Each trial gets a new Runta runtime, which is deleted at the end. Thus no trial can see the files of an earlier trial.
Scoring Method
A cell is one harness and model pair. We use the two cost metrics:
- Average cost per task: the mean cost of the priced trials of each task, then the mean over the tasks.
- Cost per pass: the average cost per task divided by the pass rate.
We pick the best value by cost per pass because we want harnesses at the top left of a chart of pass rate against cost per task: a high pass rate at a low cost. Draw a line from the origin to the point of a harness. Its slope, pass rate ÷ cost, is the passes per dollar. Cost per pass is the inverse of this slope, so the lowest cost per pass has the steepest line.
Audits of the trial logs flagged 216 trials for reward hacking, and 163 of them were online lookups. These trials score 0.
Results by Model
A deeper cell costs less than the others in its column. Codex CLI was the cheapest per pass with GLM-5.3, $0.803, but 12th with Muse and 10th with Kimi-K3. Over all models, it cost the least per pass, $0.937, and Antigravity CLI the most, $1.63.
How Sure Are These Results?
We drew the 62 tasks again 4,000 times and chose the best-value harness each time. Codex CLI stayed first with GLM-5.3 in 99.6% of the draws. The other four winners are the most likely of several close harnesses: Oh My Pi with Muse (53%), ZCode with DeepSeek (47%), Pi with GLM Flash (42%) and Alma with Kimi-K3 (25%).
The Harness Changes the Cost
The model sets the general level of the cost, because the list prices of the models differ by 20 times or more. We took the average cost per task of the 110 cells on a log scale, so that a doubling counts the same with each model. A two-way analysis of variance divides the spread of these costs into three parts:
- Model, 92.8%: the differences between the five models.
- Harness, 2.3%: the part of the effect of a harness that is the same with every model.
- Pair, 4.9%: the rest. It also contains noise, because each cell has one value.
So with the model fixed, 68% of the variation between harnesses (4.9 of 7.2 points) depends on the pair, or is noise.
With the Model Fixed, the Harness Changes the Cost by 1.8 to 5.0 Times
To test for chance, we shuffled the trials between the harnesses of each model 300 times and calculated the ratio again. “Chance” is the median of these ratios, 1.2 to 1.8 times. All five real ratios are larger than every shuffle (p = 0.003), so the spread comes from the harnesses, not from the noise of reruns. The largest is with DeepSeek: Cline CLI cost 5.0 times as much as OpenCode v1, for 2.0 more points.
The Cheapest Harness per Task Changes With the Model
Codex CLI was 1st with GLM-5.3 and 16th with Kimi-K3. DeepSeek Harness: Creator moved the most, from 1st with Muse to 21st with GLM Flash. OpenCode v1 was 1st with DeepSeek and 13th with Muse.
Reruns do not explain these moves. We divided the attempts into two halves and ranked the harnesses by cost in each half. With the same model, the two rankings had a mean rank correlation of 0.71. With two different models, it was 0.26.
A Higher Cost Does Not Give a Higher Score
The harness with the highest pass rate was 5th to 20th of 22 in cost per task. With four of the five models, the most expensive harness per pass passed less often than the cheapest one.
Does a Simpler Harness Cost Less?
No. We compared four factors with the cost per pass. Prompt size, tool count and cache hit rate are not the main cost factors. Steps are.
- Start tokens: the input tokens of the first agent-loop request: the system prompt, the tool definitions and the task.
- Tools: the tools that the first request offers.
- Cache hit rate and steps: the mean over the passed trials. A step is one entry of the agent trajectory.
Over the 22 harnesses, the rank correlation of the steps per pass and the cost per pass was +0.51 (p = 0.015). For the other three factors, it was −0.03 to +0.02 (p ≥ 0.88). Within each model, tool calls tracked cost as well as steps.
Each step sends the context again, so a trial used 2.0 to 4.8 million input tokens on average, by harness, and the largest first request, 21,365 tokens, is less than 1% of that.
Where the Cost Comes From
Over all trials, input from the cache was 42% of the cost, output 33% and other input 25%.
Tokens and the Cache
The cache makes repeated input much cheaper. Cached input costs 2% (DeepSeek) to 20% (GLM Flash) of the price of other input.
Most of the Cost Goes to Trials That Do Not Pass
64.5% of the list-price cost of the scored runs went to trials that did not pass: 52.7% to failures and 11.8% to timeouts.
What the Best-Value Harnesses Share
The traces of the five best-value harnesses show three common design choices. These are observations, not tests.
One shell tool does most of the work. By harness, it got 61% to 93% of the tool calls.
Each request only adds to the one before it. The start of the context does not change, so the provider reads it from the cache: 91.1% to 94.0% of input in passed trials. Each step pays the full input price for only about 500 new tokens.
No subagents and almost no summaries. Each of the five used a subagent in at most 6 trials. Of the five, only Codex CLI summarized, in 8 trials.
The five also share a weakness. Failed runs end with a confident answer. About three in four failed trials ended with a final answer, not at the time limit: from 68% with Pi to 82% with Codex CLI.
Conclusion
- When you change the model, think about changing the harness too. Test the candidate harnesses with your model on a private dataset of your own tasks, large enough to see past the noise.
- The model sets most of the cost, and the harness is second. The model explains 92.8% of the variance of cost per task.
- A higher cost does not buy a higher pass rate. For each model, the rank correlation of cost per task and pass rate was −0.39 to +0.22.
- Size does not predict cost.
- Most of the cost, 64.5%, went to runs that did not pass.
- With a cheap model that has a wide range of pass rates, pick a harness with a high pass rate.
- For harness developers: cut steps, not just prompt tokens. Steps predict cost, because every step sends the whole context again.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi