How AI-ready is your design system? Now you can measure it.

Every design system team is having the same conversation right now, and nobody can back it with numbers. We built the eval, ran it against four real systems, and published the harness.

Get on GitHub

Why this exists

Every design system team is having the same conversation right now: developers are generating UI with coding agents, and nobody can say with a straight face whether the output actually follows the system. Anecdotes go both ways. There is no dashboard for this.

If that output ignores the design system, brand drift, accessibility gaps, and rework show up across every feature team. Docs, AGENTS.md files, MCP servers, and skills have been hard to justify because there was nothing to measure them against.

We built open-design-system-bench because we were stuck in that conversation. It is an open benchmark for one question: when an AI agent builds UI against your component library, does it use your system correctly, and how much does each layer of guidance you write actually help?

It started at Nord Security against Aurora, its production design system, and is now generalized for any React and TypeScript component library. Every number is tool-derived and re-runnable. No telemetry.

What it is

Two tools in one repo

The audit runs in seconds, needs no API key, and costs nothing. Seven static checks score the things agents depend on: is there an AGENTS.md, can the component catalog be extracted and how well documented it is, is every public component actually reachable from the package entry point, do your names match what models expect, are your tokens machine-readable, can an agent read your deprecations, are your docs greppable text or a JS-rendered site. You get an AI-Readiness Score and a tier: Emerging, Invested, or AI-native.

The benchmark runs real coding agents against your real components. The prompts describe a user need and never name the expected component. A headless agent works in an isolated workspace, and the output is graded on six dimensions: imports, API fidelity, token discipline, static accessibility, compilation, and an LLM-judged rubric for the calls no parser can make. The headline is a matrix of system by context level by model, so you can see what adding an AGENTS.md or a skill bundle is actually worth.

What a task looks like

# tasks/confirm-account-deletion.yaml
id: confirm-account-deletion
title: Confirmation before account deletion
prompt: >-
  In the account settings screen, users can delete their account. Deletion is
  permanent. Add a step that makes sure users don't delete their account by
  accident, following our design system.
rubrics:
  - id: uses-confirmation-pattern
    text: >-
      Reaches for the design system's confirmation dialog pattern rather than
      a generic overlay, browser confirm(), or inline warning
    weight: 0.4
    critical: true
  - id: destructive-not-default
    text: >-
      The destructive action is visually distinct (destructive emphasis) and
      is NOT the default-focused or visually primary action
    weight: 0.35
  - id: easy-backout
    text: >-
      The user can easily back out, and the copy states the permanent
      consequence plainly
    weight: 0.25

# Optional, and omitted by all ten starter tasks: the expected component,
# recorded per system and never shown to the generating agent.
hiddenExpectations:
  componentsAnyOf: { reactkit: [Modal], appkit: [Modal] }

How one cell runs

A cell is one task, one system, one context level, one model. Generations run in isolated fixture workspaces against the published packages, so the only thing that comes back is a diff. Single-shot API cells and agentic CLI cells are never ranked against each other: agentic cells cost roughly 80x more and see compiler feedback.

Grade the worst dimension, not the average

One early lesson shaped the scoring model. Identical average scores hide opposite failures. One generation uses your system heavily and gets the API subtly wrong. Another avoids your system entirely, hand-rolls everything, and passes every API check by never touching the API. A third builds something impressive that does not compile. Averages treat these as equal. So the gate is the worst dimension, and "ignored the design system" is a first-class metric. A model that never touches your system passes every API check, and for a design system team that is the worst of the three.

What it found

We pointed open-design-system-bench at four real design systems: Aurora, the production design system at Nord Security where the harness started, plus three widely used open-source design systems, anonymized here as System A, System B, and System C. The five findings below are what the static audit and the generation benchmark turned up in that code.

Finding 1: every system it touched had a real bug

The extraction step never trusts a hand-written component list. It computes the public API from source: the root barrel union the package.json exports map. That single rule has found shipped defects in every production system audited so far:

• Three components in one production kit (a Toggle among them) had proper per-directory entry points but were never re-exported. Teams believed they were public. Consumers could not import them.
• A second production kit had an entire component directory unreachable via either the barrel or the exports map, and a changelog whose version headings no parser could read. Agents migrate what they can read, and no agent could read this one.

Finding 2: nobody is AI-native yet

The AI-Readiness Score assembles seven static checks into a 0 to 100 composite with three tiers: Emerging (below 40), Invested (40 to 70), and AI-native (70 and up). The first four-system distribution:

Finding 3: the failure modes are not what the average shows

Across 898 graded generations (6 model configurations, across Aurora's two component kits), three failure shapes kept appearing, and they can produce identical average scores:

• Engage and err: uses the system heavily, gets APIs subtly wrong.
• Avoid the system: hand-rolls everything; passes every API check by never touching the API. It looks like the safe result, and for a system team it is the worst one.
• Architect and collapse: ambitious structure, does not compile.

Finding 4: models arrive with a vocabulary, and it is not yours

The convention lexicon in the repo is empirical: it is the names AI models invented across those 898 generations when they guessed instead of reading. Models expect Switch, Box, TextField, spacing, as. Systems whose names sit close to that prior get compliance without extra enablement; systems with idiosyncratic vocabulary pay a hallucination tax on every generation.

Finding 5: the tool's own assumptions did not survive contact either

An honest one: the day we pointed the extractor at the three open-source systems, three of its own assumptions broke. It assumed barrels use export * from, but two of them re-export by name. It assumed barrel targets are plain re-export files, but one system's barrel points straight at JSX implementation files. It assumed directory indexes are index.ts, but a third of that system's are index.tsx.

Run it on your system

What a team does with this, in order:

1. Run the audit (seconds, free). Fix the fails: they are usually export hygiene and deprecation legibility, and they are real bugs.
2. Ship the cheap wins: AGENTS.md, machine-readable changelog, committed token file, llms.txt.
3. Run the benchmark before and after. The delta between bare and guided contexts (Lift) is the number that holds up in a budget conversation: tool-derived, reproducible, and not a survey.

The 5-minute audit (free, no API key)

git clone https://github.com/christophhdesign/open-design-system-bench
cd open-design-system-bench
npm install

# point the harness at your design system (interactive wizard)
npx tsx src/cli.ts init

# extract your component catalog from source
npx tsx src/cli.ts extract

# score it
npx tsx src/cli.ts audit --verbose

The benchmark (bring your own key)

# sanity pass: 2 cells, a few dollars, ~10 minutes
npx tsx src/cli.ts run --profile smoke

# the meaningful monthly sweep
npx tsx src/cli.ts run --profile medium

What gets measured

Tier 1: the static audit

1. Enablement surface: AGENTS.md/CLAUDE.md presence, llms.txt, MCP hints, skills, etc.
2. Catalog quality: Share of components with typed props, defaults, and descriptions.
3. Export hygiene: Barrel and exports-map consistency.
4. Vocabulary convention-distance: Names diffed against an empirical lexicon.
5. Token machine-readability: CSS custom properties or DTCG.
6. Deprecation legibility: @deprecated annotations and machine-readable changelogs.
7. Docs greppability: Docs reachable as text.

Tier 2: the generation benchmark

1. Imports: Only the system's package, React, and local files.
2. API fidelity: No hallucinated components or invented props.
3. Token discipline: No raw hex or arbitrary values.
4. Static accessibility: AST checks for aria, labels, and alt text.
5. Compilation: tsc --noEmit against the fixture.
6. LLM-judged rubrics: Scored by a separate model for logic and design quality.

Common questions

"Isn't this just linting with extra steps?" The audit half is static, yes. It earns its keep because it checks export reachability computed from source, vocabulary distance against mined lexicons, and deprecation legibility—none of which are in standard linters.

"LLM-as-judge is unreliable." Partly true, which is why only one of six dimensions is judged. Five are mechanical. The judge covers rubrics no parser can score, and its reasoning is stored verbatim for inspection.