DEVOURED
Model-Harness-Fit
Research shows frontier AI models are post-trained for specific tool harnesses, with Claude Opus 4.6 scoring 4.5% higher with the right framework and harness choice alone moving Cursor from top 30 to top 5 in benchmarks.
What: Bustamante's analysis of Codex CLI, Claude Code, and GitHub Copilot CLI reveals that AI labs optimize their models for specific tool harnesses, embedding preferences for particular tool names, schemas, citation formats, memory patterns, and editing styles directly into model weights during post-training.
Why it matters: This explains why the same model performs dramatically differently across coding tools and suggests that choosing the right harness-model pairing is as critical as model selection itself, with mismatches wasting reasoning tokens on unnecessary adaptations.
Takeaway: Developers building AI-powered tools should design harnesses that align with their target model's training preferences, or select models specifically trained for their existing infrastructure to maximize performance.
Deep dive
- Terminal-Bench 2.0 data demonstrates measurable impact: Claude Opus 4.6 achieved 79.8% accuracy with ForgeCode versus 75.3% with Capy, a 4.5 percentage point difference from harness choice alone
- Cursor's jump from top 30 to top 5 ranking was achieved solely by changing the harness, without any model improvements
- OpenAI models default to patch-based file editing approaches while Anthropic models prefer string replacement methods
- Harness mismatches force models to spend reasoning tokens adapting their output format instead of solving the actual problem
- Post-training against specific harnesses embeds tool names, schemas, citation tag formats, memory rituals, and system prompt structures directly into model weights
- This suggests frontier labs are optimizing models for their own tooling ecosystems, creating lock-in effects
- The research covers three major CLI tools: Codex CLI, Claude Code, and GitHub Copilot CLI
- Harness design choices include how tools are invoked, how context is structured, and how outputs are formatted
- The findings challenge the assumption that models are general-purpose and perform consistently across different integration layers
- For developers, this means harness selection is a first-order concern, not just an implementation detail
Decoder
- Harness: The integration layer or framework that wraps an AI model, defining how it receives inputs, formats outputs, and interacts with tools
- Post-training: Additional training applied to base models to optimize them for specific use cases, tools, or formats after initial pre-training
- Terminal-Bench: A benchmark for evaluating AI models' performance on command-line and terminal-based coding tasks
- ForgeCode/Capy: Different harness frameworks used in the benchmark comparisons
- Reasoning tokens: The computational budget models spend on internal processing and problem-solving, which can be wasted on format adaptation instead of the core task
- Citation tags: Specific markup formats models use to reference sources or indicate tool usage in their outputs
- Memory rituals: Patterns for how models maintain and reference context across interactions
Original article
JavaScript is not available.
We've detected that JavaScript is disabled in this browser. Please enable JavaScript or switch to a supported browser to continue using x.com. You can see a list of supported browsers in our Help Center.
Explore all 52 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi