DEVOURED
Initial Results on Legal Agent Benchmark
Harvey's Legal Agent Benchmark shows frontier models like Claude Opus 4.7 achieving only 7.1% on legal tasks under an "all-pass" standard, highlighting current AI limitations in complex legal work.
What: Harvey's initial results on its Legal Agent Benchmark, using an "all-pass" standard where every rubric criterion must succeed, reveal low performance from top AI models. Claude Opus 4.7 led with 7.1%, followed by Sonnet 4.6 (5.4%), Opus 4.6 (4.2%), GPT-5.5 (2.1%), and Gemini 3.5 Flash (0.8%).
Why it matters: These low scores underscore that despite rapid advancements, even the most capable AI models are far from saturating the complex and nuanced requirements of professional legal work, suggesting significant challenges remain for AI agents in highly specialized domains.
Takeaway: If you are working on AI applications for the legal sector, be aware that current frontier models still have very limited capabilities for complex tasks under strict performance criteria, requiring substantial human oversight and refinement.
Decoder
- All-pass standard: A stringent evaluation criterion where a task is only considered successfully completed if every single requirement or sub-criterion in the rubric is met without error.
Original article
Harvey baselined frontier models on its Legal Agent Benchmark holdout under an "all-pass" standard requiring every rubric criterion to pass, and Claude Opus 4.7 led at just 7.1% with Sonnet 4.6 at 5.4%, Opus 4.6 at 4.2%, GPT-5.5 at 2.1%, and Gemini 3.5 Flash at 0.8%, signaling legal work is far from saturated by frontier intelligence.
Explore all 49 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi