A benchmark map of the US economy

AI benchmarks have multiplied in recent years, but coverage of economically valuable work remains limited. To measure this, we compiled relevant benchmarks to measure their coverage of American jobs. This Benchmark Map is our first step toward building a comprehensive, continuously updated resource for understanding AI capabilities across the economy.

of 923 occupations covered

208

of 18,797 tasks covered

924

benchmarks in the index

29

The Benchmark Map

Each tile represents one of the 923 O*NET occupations making up the US labor force, grouped into 22 job families. Occupations are shaded according to the share of their tasks which are represented in our initial list of AI benchmarks. A single benchmark prompt can be mapped to multiple O*NET tasks.

In our data set: only 208 of 923 occupations have a task with at least 5 benchmark prompts, and 6 of 22 job families have no such occupations.

Our findings suggest that 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks. This makes it very hard to know how good AI models are at the work Americans actually do.

The coverage of existing benchmarks is not random, either. Our map suggests that the AI community is focusing on the largest labor markets first.

For example, benchmarks in our initial list heavily cover AI-exposed lawyers’ tasks, but barely cover any of the O*NET tasks of similarly AI-exposed legal secretaries or court clerks. The long tail of smaller, middle-income knowledge work jobs appears undercovered.

As we build out our coverage of model capabilities across the economy, we hope to both surface and help fill more gaps in our understanding of what models can do.

Methodology in brief

To break down the economy into sectors, occupations, and tasks, we used the US Department of Labor’s O*NET taxonomy. O*NET divides the US workforce into 923 occupations; for each occupation, O*NET attempts to provide a comprehensive list of “tasks” which make up the occupation, totaling 18,797 tasks across all occupations.

Example — 6 of the 22 tasks O*NET associates with Lawyers

1. Prepare, draft, and review legal documents, such as wills, deeds, patent applications, mortgages, leases, and contracts.
2. Advise clients concerning business transactions, claim liability, advisability of prosecuting or defending lawsuits, or legal rights and obligations.
3. Interpret laws, rulings and regulations for individuals and businesses.
4. Analyze the probable outcomes of cases, using knowledge of legal precedents.
5. Present evidence to defend clients or prosecute defendants in criminal or civil litigation.
6. Supervise legal assistants.

To measure benchmark coverage, we start with a list of 29 benchmarks selected for their quality and economic relevance. In selecting benchmarks for this list, we prioritized:

• Public Availability. Included benchmarks have at least 25 publicly available tasks.
• Agentic Tasks. Included benchmarks ask models to perform agentic tasks, rather than answering questions or advising the user on how to complete a task.
• Economic Realism. Benchmark tasks resemble tasks which humans (or agents) are frequently asked to do in real-world enterprises.
• Economic Diversity. We capped the number of included benchmarks per job family.
• Balanced Difficulty. Benchmarks are not yet saturated by frontier models.

We ask an LLM judge to tag each benchmark prompt with as many O*NET tasks as it thinks correspond to that prompt. An O*NET task counts as “covered” if our LLM judge has mapped it to at least 5 prompts from our 29 benchmarks.

Limitations

Our estimates are based on the initial list of 29 benchmarks, which is not a complete or perfectly representative sample. Any incomplete list will necessarily bias down estimates of economic coverage. We also capped the number of included benchmarks in each job family to maximize breadth, which biases down estimates for the most-benchmarked job families.

By selecting for unsaturated benchmarks, we risk understating coverage on easier tasks. Conversely, including saturated benchmarks could overstate performance due to lack of realism in those benchmarks. Several criteria, such as "Economic Realism," are inherently subjective.

Patterns in coverage

The benchmarks in our index have relatively deep and narrow coverage of the overall economy. They disproportionately focus on the economic tasks most exposed to AI automation, with O*NET tasks rated as “exposed” about 6x more likely to be represented on benchmarks than tasks rated “unexposed.”

After controlling for occupation and AI-exposure score, O*NET “task importance” is another predictor of benchmark coverage. The tasks scored most important are about 6–10x more likely to be represented on benchmarks than those scored least important.

Can O*NET + benchmarks tell us how good AI is at your job?

Measuring model capabilities by job occupation is essential. A simple average of benchmark performance would give too much weight to frequently benchmarked tasks. O*NET weighting allows us to weight performance on each O*NET task equally. While current results show similar performance between weighted and unweighted averages, we are hopeful that expanding our benchmark set will improve the utility of this metric.

However, even with more benchmarks, several obstacles remain:

1. Low statistical power: The breadth of the US economy means we have relatively few benchmark items per occupation/task.
2. Varying difficulty and fidelity: Benchmarks vary in quality, saturation, and realism, and we lack a formal way to weight them by quality.
3. Automation vs. Augmentation: Most benchmarks measure end-to-end automation, but real-world AI is often used for human augmentation.
4. Measurement gaps: ~63% of jobs involve physical-world components that are currently difficult to capture in digital-only benchmarks.