Back to Devoured - August 24, 2026
DEVOURED
The Scaffolding Matters More Than the Interface

The Scaffolding Matters More Than the Interface

Data aiagents Arxiv
A study of seven agent frameworks reveals that the underlying scaffolding, not the tool interface, determines AI agent cost and efficiency.
What: Researchers Marc Alier Forment et al. tested seven agent frameworks against a standardized software task. They found that CLI-based tools were 5x to 28x cheaper than those using the Model Context Protocol (MCP), suggesting that the 'harness' architecture has a much higher impact on performance and cost than the communication interface.
Why it matters: The industry is currently obsessed with standardizing agent interfaces like MCP, but this research highlights that engineering optimizations within the agent's logic are significantly more critical for cost-effectiveness.
Decoder
  • MCP: Model Context Protocol; an open standard for connecting AI assistants to systems like databases or code repositories.
  • Scaffolding: The structural logic and orchestration code that manages how an AI agent breaks down tasks, calls tools, and processes feedback.
Original article

Title: The Scaffolding Matters More Than the Interface: A Controlled Comparison of MCP and CLI Tool Use Across Seven Agent Scaffoldings, Five Language Models, and One Software Task

How much an AI coding agent costs to run can depend more on the agent scaffolding that drives it than on the interface through which it reaches its tools. We set out to measure the cost of tool use over the Model Context Protocol (MCP) against tool use over an ordinary command-line interface (CLI), a difference on which published estimates disagree by more than an order of magnitude while resting on practitioner reports that cannot be reproduced. We ran one fixed software task -- six operations against a private online git repository -- across seven agent scaffoldings and five language models, and we verified completion by inspecting the repository state rather than trusting the agent's self-report.

The dominant effect was the scaffolding. Two of the seven ship no MCP support at all; they completed every run using only the CLI, which shows that MCP is unnecessary for this class of work, and they were 5.0x to 28x cheaper than the five scaffoldings that do support MCP, comparing CLI runs alone with no MCP server attached anywhere. The effect was largest for a small 27-billion-parameter model running locally, whose cost varied 139x across scaffoldings while it completed the task under all of them.

The comparison we set out to make proved unstable: thirteen strictly paired MCP-to-CLI ratios span 0.43x to 29x, with outliers on both sides. The two interfaces separate on the cost of failure, where 12.9 per cent of the money spent on MCP runs bought no completed work against 2.2 per cent on CLI runs, but not on its frequency: failures were equally common in both, in the original runs and in their repetitions alike. Agents frequently ignored the interface they were assigned, so comparisons that do not verify actual behaviour measure an unknown mixture. The harness, the task, the verification and the complete dataset are released as open source.

Explore all 66 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi