DEVOURED
How enabling two settings tripled our scores on the ARC-AGI-3 benchmark
Researchers found that simple API configuration adjustments and prompt-level tweaks tripled benchmark scores for the GPT-5.6 Sol model.
What: Enabling 'retained reasoning' and 'compaction' settings in ChatGPT and Codex increased performance on the ARC-AGI-3 benchmark and reduced total output tokens by 6x.
Why it matters: This highlights that state-of-the-art performance is increasingly dependent on the 'harness'—the orchestration layer and parameter tuning—rather than just the raw capabilities of the underlying model.
Takeaway: When optimizing AI performance, prioritize adjusting reasoning parameters and output compaction settings before concluding that model size or fine-tuning is the primary bottleneck.
Decoder
- ARC-AGI-3: A benchmark designed to measure an AI's ability to solve novel reasoning problems that it has not encountered during training.
- Retained reasoning: A prompting technique where the model stores and references its intermediate reasoning steps throughout the generation process.
- Compaction: A technique to condense model output or context window usage without losing information.
Original article
GPT-5.6 Sol scores just 7.8% on the ARC-AGI-3 benchmark despite having solved longstanding open problems in mathematics and beaten games like Pokémon FireRed. Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting. Researchers discovered that turning on retained reasoning and compaction in ChatGPT and Codex tripled scores and cut output tokens by 6x on the benchmark.
Explore all 59 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi