DEVOURED
Artificial Analysis Capability Indices v1.1
Artificial Analysis updated its Capability Indices to v1.1, featuring deeper domain tuning and confirming Claude Fable 5.1's dominance across key industries.
What: The update incorporates new benchmarks like AutomationBench-AA (Agentic Tool Use) and AA-Briefcase (Agentic Knowledge Work) across six professional domains, with Claude Fable 5.1 leading in all categories.
Why it matters: The inclusion of more private, held-out datasets indicates a concerted effort to curb the 'gaming' of public benchmarks by model labs.
Deep dive
- The updated indices cover Finance, Strategy, Legal, Healthcare, Engineering, and Economics.
- Claude Fable 5.1 is the top-performing model across all six capability indices.
- Open-weight models like Kimi K3, DeepSeek V4.1, and GLM-5.3 are showing strong, specific competitive performance.
- AA-Briefcase is a new, in-house benchmark designed to test multi-week, complex knowledge work projects.
- 40% of the Intelligence Index weighting is now composed of private, held-out test sets to prevent model overfitting.
Decoder
- Elo rating: A system originally designed for chess, here used to measure the relative skill levels of AI models in head-to-head performance benchmarks.
- Held-out test set: Data that the model was not trained on and is kept secret from the developers, used to ensure the evaluation is accurate and not merely reflecting memorization of the questions.
Original article
Full article content is not available for inline reading.
Explore all 47 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi