Back to Devoured - November 20, 2026
DEVOURED
Stripe evals: Measuring real-world coding agent performance

Stripe evals: Measuring real-world coding agent performance

AI Stripe
Stripe released an evaluation framework revealing that while AI coding agents often produce working code, their success rate drops sharply under strict quality criteria.
What: Stripe benchmarked 14 coding agent pairs across 69 integration scenarios. While agents improved with a dedicated 'Stripe agent plugin'—reducing web searches by 96%—the strict pass rate averaged only 29.4%.
Why it matters: The gap between 'working code' and 'production-grade integration' remains the primary bottleneck for autonomous coding agents, necessitating domain-specific plugins and rigorous eval harnesses.
Takeaway: Developers building with agents should use the released Forge eval harness to test integration quality against Stripe's rubrics.
Deep dive
  • Tested 14 coding model/harness pairs including Claude Code, Codex, and OpenCode.
  • Created 69 scenarios covering one-time payments, subscriptions, and Connect platforms.
  • Introduced a 'Stripe agent plugin' providing MCP-based tools and documentation access.
  • Found the plugin reduced API parameter hallucinations from 7% to 1.3%.
  • Discovered that while agents perform well on basic tasks, they struggle with 'last-mile' production requirements like error handling and idempotency.
  • The 'strict pass rate' across all models dropped from a loose 78.1% to a stringent 29.4%.
Decoder
  • MCP (Model Context Protocol): An open standard that allows AI models to connect securely to local data sources and tools without needing custom integrations for every platform.
  • Harness: A testing framework that automates the execution of code within a simulated environment.
  • Idempotency: The property of an API operation where making the same request multiple times produces the same result as a single request, crucial for payment systems.
Original article

Full article content is not available for inline reading.

Read the original article →

Explore all 56 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi