We benchmarked coding agents on our own internal tasks at Databricks and learned a lot!
Databricks' internal benchmark of coding agents reveals that smaller, open-source models like GLM 5.2 now rival top-tier commercial offerings in real-world engineering tasks.
Deep dive
- Databricks created custom internal benchmarks to avoid the 'over-tuning' common in public datasets like SWE-Bench.
- GLM 5.2 emerged as a high-performing open-source agent model across diverse languages.
- Simple, lightweight harnesses often outperform vendor-provided harnesses by minimizing unnecessary tokens passed to the LLM.
- Model selection should be based on task-specific cost-per-task rather than per-token pricing.
- The team developed 'Omnigent,' a meta-harness for composing and swapping agents, to manage complex multi-step workflows.
Decoder
- Harness: A software layer that manages the interface between a developer's codebase and an LLM, handling task orchestration, context management, and output parsing.
- SWE-Bench: A common benchmark used to evaluate how well AI models solve GitHub issues by writing and testing code.
- Meta-harness: A higher-level abstraction that allows developers to chain or switch between different individual agent harnesses within a single workflow.
Original article
We benchmarked coding agents on our own internal tasks at Databricks and learned a lot!
There are many surprising opportunities to lower cost and increase quality, and many models including open source ones are truly competitive now.
Why did we build an internal coding benchmark? Public benchmarks like SWE-Bench are often over-tuned for, so we took real tasks our engineers did and curated test suites for them to see which agents can solve them end-to-end.
These are the results on OUR sample of OUR codebase, so they are not meant to be comprehensive, but we think many companies can do a similar internal benchmark. Several findings immediately stood out:
1) Many models are now competitive at the top tier, including open source.
2) GLM 5.2 in particular was a major step forward in open source coding agent performance, even in our own codebase that is VERY different from SWE-Bench and TerminalBench (lots of Scala, Go, Rust, Java, TypeScript, Protobuf, Jsonnet, etc).
3) Harnesses make a huge difference in cost-performance. The very simple Pi harness got the same success rate as harnesses from the LLM vendors with Opus and GPT 5.5, but at 2x less cost! Seems to be mainly due to smaller inputs to the LLM.
4) Cheaper per-token does not imply cheaper per-task. For example, Sonnet 5 costs less per token than Opus 4.8 but used more tokens, resulting in higher cost and lower quality.
Read more in the blog how we built the benchmark and what we're doing with the findings. This is partly why we built Omnigent as a "meta-harness" to let developers switch and compose agents, and Unity AI Gateway to analyze and gate LLM usage centrally.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi