DEVOURED
Meta's ProgramBench benchmark reveals every AI model achieves 0% on recreating executables from scratch, with Claude Opus 4.7 barely reaching 3% partial success across 200 tasks requiring agents to rebuild programs like jq, SQLite, and FFmpeg without source code.
What: John Yang, Kilian Lieret, and researchers from Meta Superintelligence Labs, Stanford, and Harvard released ProgramBench, a benchmark with 200 tasks where agents must recreate programs (jq at 34,541 lines, SQLite at 9,434 lines, PHP compiler at 40,030 lines, FFmpeg at 59,217 lines) given only the compiled binary and documentation. 248,000+ behavioral tests, sandboxed environment with no internet or decompilation allowed. Evaluated with mini-SWE-agent. Claude Opus 4.7 leads: 0% fully solved, 3.0% almost solved (≥95% tests pass). Claude Opus 4.6: 0% / 2.5%. All other models: 0% / 0-1%.
Why it matters: This exposes the gap between code completion and true software architecture. Current models can fill templates and patch existing code but cannot design complete systems from first principles. Unlike SWE-bench or other benchmarks that provide skeletons, requirements docs, or method signatures, ProgramBench forces agents to make every design decision: language choice, module decomposition, interface design, build system.
Takeaway: Public submission portal opening soon. Follow John Yang (@JohnYangSam) and Kilian Lieret (@lieret_k) for updates. Paper: arxiv.org/abs/2605.03546.
Deep dive
- Scope: 200 open-source programs ranging from small utilities (jq, ripgrep, fd) to massive projects (PHP compiler, FFmpeg, SQLite, DuckDB). Lines of code range from hundreds to 79,721 (fzf). Languages: C, C++, Rust, Go, Haskell, Java.
- Task: Agent receives execute-only binary and docs, must recreate the program from scratch. No source code, no decompilation (binary has execute-only permissions blocking objdump/strings/disassemblers), no internet (to prevent cloning repos or downloading code). Must choose language, design architecture, write all code, produce build script.
- Tests: 248,000 behavioral tests generated via agent-driven fuzzing. A solution passes only if all tests for that task pass. Tests compare candidate program behavior against original.
- Why scores are low: Building from scratch is fundamentally hard. No hints/structure provided (no method signatures, no file layout, no requirements docs). No harness tuning (single generic mini-SWE-agent harness for all 200 tasks, unlike concurrent work that tunes harnesses per-task). Agents must truly architect. Models submit deliberately (don't hit time/step limits, runs cost up to $5k for Sonnet 4.5).
- Best partial scores by program: jq 90%, nnn (file manager) 98%, htmlq 94%, pueue 15%, PHP 5%, FFmpeg 5%, QuickJS 4%, indicating smaller/simpler programs are more tractable.
- Cheating prevention: Early trials showed models cloning GitHub repos or downloading via package managers. Now: sandboxed containers, no network, execute-only binary permissions, no decompilation tools.
- Contamination: All tasks from open-source repos seen in training, but scores suggest memorization isn't helping. Different-language ablation (force different implementation language than original) showed similar scores, confirming performance isn't driven by source code memorization.
- Why mini-SWE-agent: Widely adopted baseline (used by SWE-bench Verified, SWE-bench Multilingual, Terminal-bench), deliberately minimal scaffolding (reduces confounds between model capability and harness design), stable (unlike Claude Code with "several 100k lines of code" that changes constantly).
- Metrics: Primary: fully resolved instances. Secondary: almost resolved (≥95% tests pass). Rejected average test pass rate as misleading (even simple tests like --help flag inflate scores). Some tasks have 15k tests, so even 1% failure = 150 broken tests.
- Paper: arxiv.org/abs/2605.03546 by John Yang, Kilian Lieret, et al. from Meta Superintelligence Labs, Stanford, Harvard.
Decoder
- mini-SWE-agent: Minimal agentic scaffolding used as baseline for SWE-bench and other coding benchmarks. Provides basic tools (read/write files, run commands) without complex planning or multi-agent orchestration, isolating model capability from harness design.
- Behavioral tests: Tests that verify program behavior by running it with inputs and checking outputs match expected results, rather than checking implementation details. ProgramBench compares candidate program output against original program output across diverse inputs.
- Agent-driven fuzzing: Using AI agents to automatically generate test inputs that explore different program behaviors and edge cases, rather than manually writing tests.
- Cleanroom implementation: Software development approach where engineers recreate a program from scratch using only behavioral specification/documentation, without access to original source code. Legally used to create compatible implementations without copyright infringement.
Original article
ProgramBench challenges agents to recreate software executables without source code, using only documentation and experimentation. The tasks range from terminal utilities to complex software like compilers and libraries, offering over 248,000 behavioral tests across 200 tasks. Agents must design and implement entirely from scratch in a secure, sandboxed environment, emphasizing software architecture skills without external aids or decompilation.
Explore all 73 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi