DEVOURED
Stripe evals: Measuring real-world coding agent performance
Stripe released an evaluation framework revealing that while AI coding agents often produce working code, their success rate drops sharply under strict quality criteria.
What: Stripe benchmarked 14 coding agent pairs across 69 integration scenarios. While agents improved with a dedicated 'Stripe agent plugin'—reducing web searches by 96%—the strict pass rate averaged only 29.4%.
Why it matters: The gap between 'working code' and 'production-grade integration' remains the primary bottleneck for autonomous coding agents, necessitating domain-specific plugins and rigorous eval harnesses.
Takeaway: Developers building with agents should use the released Forge eval harness to test integration quality against Stripe's rubrics.
Deep dive
- Tested 14 coding model/harness pairs including Claude Code, Codex, and OpenCode.
- Created 69 scenarios covering one-time payments, subscriptions, and Connect platforms.
- Introduced a 'Stripe agent plugin' providing MCP-based tools and documentation access.
- Found the plugin reduced API parameter hallucinations from 7% to 1.3%.
- Discovered that while agents perform well on basic tasks, they struggle with 'last-mile' production requirements like error handling and idempotency.
- The 'strict pass rate' across all models dropped from a loose 78.1% to a stringent 29.4%.
Decoder
- MCP (Model Context Protocol): An open standard that allows AI models to connect securely to local data sources and tools without needing custom integrations for every platform.
- Harness: A testing framework that automates the execution of code within a simulated environment.
- Idempotency: The property of an API operation where making the same request multiple times produces the same result as a single request, crucial for payment systems.
Original article
Full article content is not available for inline reading.
Explore all 56 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi