DEVOURED
Harvey open-sourced a 1,200-task legal agent benchmark graded on 75,000 all-or-nothing criteria, reflecting how law firms actually review work: missing one issue doesn't make a deliverable 80% useful, it makes it materially incomplete.
What: Harvey released Legal Agent Benchmark (LAB) on May 6, an open-source benchmark with 1,200+ tasks across 24 practice areas, authored by Niko Grupen, Gabe Pereyra, and Julio Pereyra. Each task mirrors law firm workflow: agents receive partner instructions, a client matter with mixed relevant and peripheral documents, and must produce reviewable work product graded against expert rubrics. Example M&A task: analyze change-of-control provisions for the $458 million acquisition of Crestview Software Solutions using a virtual data room with 8 material contracts, produce a deal-team memo, graded on 57 criteria covering 9 legal issues. Uses all-pass grading: every criterion must pass, no partial credit. No leaderboard yet; Harvey will publish baseline results in coming weeks.
Why it matters: This suggests legal AI is approaching an inflection point similar to coding agents, which shifted from 'basically didn't work before December and basically work since' according to SWE-Bench. The all-or-nothing grading reflects how law firms need to understand which tasks can be fully delegated to agents versus requiring human review, since in high-stakes legal work, partial completion is materially incomplete, not partially useful.
Takeaway: The benchmark is on GitHub now. If you build legal agents, research long-horizon agent systems, or work at a law firm evaluating AI deployment, you can run the benchmark, audit rubrics, or reach out to Harvey's research team.
Deep dive
- LAB contains 1,200+ tasks across 24 legal practice areas (corporate M&A, securities, litigation, regulatory, advisory), evaluated by 75,000+ expert-written rubric criteria
- Task structure mirrors law firm workflow: partner instruction (averaging 50 words) → client matter with document mix → agent produces work product → graded against detailed rubric
- Example M&A task: analyze change-of-control provisions for $458 million acquisition of Crestview Software Solutions; agent receives virtual data room with 8 material contracts plus peripheral docs, must produce deal-team memo with executive summary, risk mapping, contract-by-contract analysis, severity ratings, recommendations
- Rubric for M&A example contains 57 criteria covering 9 legal issues, checking facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts, formatting
- All-pass grading: task marked complete only if every criterion passes - reflects that catching 8 of 10 issues is not 80% useful, it is materially incomplete in legal practice
- Launching open-source without leaderboard first to gather community input; will publish baseline results and submission standards in coming weeks
- Built to help law firms identify where agents can be deployed versus where human-in-the-loop review is needed, enabling ROI measurement of AI investments
- Positioned alongside other agent benchmarks showing capability inflection points: SWE-Bench Pro/Verified for coding, GDPval for real-world knowledge work, OSWorld-Verified for computer use, MCP Atlas for tool use, FinanceAgent for financial analysis
- Spencer Poff led open-source technical work on harness design and agent sandboxing; Julio Pereyra led task design with document and scenario generation pipeline
- Existing legal benchmarks (LegalBench, CUAD, LEXam, BigLaw Bench) test short-horizon reasoning; LAB tests long-horizon agent work on realistic legal tasks
- Future expansion: add more tasks within existing practice areas, expand to all BigLaw practice areas, cover in-house legal work, extend to adjacent knowledge-work domains like asset management and banking
- Already being used internally at Harvey for product evaluation and by research community for open-weight post-training, auto-research, memory, domain-specific legal skills, harness optimizations
Decoder
- All-pass grading: Evaluation method where a task is marked complete only if every rubric criterion passes, with no partial credit. Harvey uses this because in legal work, missing one issue in a deliverable makes it materially incomplete, not partially useful.
- BigLaw: Large, elite law firms (typically AmLaw 100/200) that handle complex corporate transactions, major litigation, and regulatory matters for Fortune 500 companies and institutional clients.
- Change-of-control provision: Contract clause defining what happens when a company is acquired, often triggering consent requirements, pricing changes, exclusivity modifications, or termination rights.
- Virtual data room (VDR): Secure online repository used in M&A transactions where the seller uploads contracts, financials, and other due diligence materials for buyer review.
Original article
Harvey's Legal Agent Benchmark (LAB) is an open-source tool for assessing AI agents' performance in legal tasks.
Explore all 73 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi