Any modality, Many Applications, One Unified Verification Framework

LLM-as-a-Verifier achieves SOTA performance across agentic benchmarks, including Terminal-Bench, SWE-Bench Verified, MedAgentBench, RoboRewardBench and more. We invite the community to contribute more use cases!

Installation

pip install llm-verifier

To install the latest from a clone:

pip install -e .

What's new in 0.2.0:

• Prefix-cache optimization: ~3.4× fewer uncached input tokens on trajectory-heavy benchmarks
• Terminal-Bench 2.1 self-verification benchmark
• deepseek-v4-flash verifier backend
• Token accounting (llm_verifier.token_usage())

About

LLM-as-a-Verifier is a general-purpose framework that provides fine-grained feedback for any agent. The key idea is simple: 1) use fine-grained scoring granularity, 2) take the expectation over the full logprob distribution of LLM score tokens, and 3) scale repeated evaluation and criteria decomposition. The resulting fine-grained feedback can be used for test-time scaling, progress tracking, and reinforcement learning.

Quickstart

Simple Best-of-N Selection

Run a first end-to-end selection (requires DEEPSEEK_API_KEY or VERTEX_API_KEY in .env, or an OpenAI-compatible server that returns logprobs — e.g. vllm serve Qwen/Qwen3.5-9B with OPENAI_BASE_URL=http://localhost:8000/v1):

import llm_verifier

problem = "Write a function that reverses a string."
candidates = [
    "def rev(s): return s[::-1]", "def rev(s): return s", "def rev(s): return ''.join(sorted(s))",
]

result = llm_verifier.select(
    problem=problem,
    candidates=candidates,
    criteria={"Correctness": "Does the code actually reverse the string?"},
)
print(result.index)   # index of the best candidate: 0
print(result.scores)  # candidate scores: [0.73104, 0.38446, 0.38449]

Score a pair of candidates directly

select is built on a pairwise reward model. For the raw fine-grained rewards of a single comparison, call compare:

reward_a, reward_b = llm_verifier.compare(
    problem, candidates[0], candidates[1],
    criteria={"Overall": "Does the code solve the problem?"},
)
print(reward_a, reward_b)   # fine-grained rewards in [0, 1]: 0.99994 0

Fine-grained Progress Tracking

The same fine-grained reward can also score an agent's progress after each step with track:

steps = [
    'Read the problem statement',
    'Wrote def rev(s): return s ',
    'Tested: rev("abc") returned "abc"',
    'Changed to def rev(s): return s[::-1]',
    'Tested: rev("abc") returned "cba"',
]

result = llm_verifier.track(problem=problem, steps=steps,
                            checkpoint_steps=[1, 2, 3, 4, 5], n_evaluations=4)
print(result.scores)  # progress after each step: [0.00106, 0.02417, 0.03143, 0.62004, 0.99978]

Self-Verification (Terminal Bench 2.1)

Can a model verify its own rollouts? On Terminal-Bench 2.1 we generate 5 mini-swe-agent trajectories per task with deepseek-v4-flash and use the same model as the verifier. Selection lands well above Pass@1 even though the verifier is judging its own model's work:

The trajectories ship in data/terminal_bench_2.1_trajs/; scoring only needs DEEPSEEK_API_KEY in .env. Each configuration has its own reproduction script:

python scripts/run_bo3.py                    # best-of-3
python scripts/run_bo5.py                    # best-of-5

Test-Time Scaling for Agentic Benchmarks

Progress Tracking for Coding Agents

The same fine-grained reward can score a trajectory at every step. Below, we track two Terminus-2 runs of the Terminal-Bench task pytorch-model-cli. The successful trajectory exhibits consistently increasing verifier scores, whereas the failed trajectory is characterized by erroneous behaviors, resulting in lower scores throughout the execution. Reproduce it with:

python scripts/terminal_bench_progress.py    # scores both runs then plots

Online progress tracking

track scores a finished trajectory. To monitor an agent while it runs, use ProgressTracker: feed it each step as it happens and get a live progress score back — e.g. to stop a hopeless rollout early or decide when to resample. Since the verifier only ever sees the steps so far, it cannot peek at the future.

Multi-Modal Support

With a multimodal verifier model, every entry point accepts images — a single image or a list of images, each a local file path, an http(s) URL, or raw bytes.

Claude Code Plugin

TurboAgent brings LLM-as-a-Verifier to Claude Code as a drop-in LLM API proxy. It sits between your client and the model provider, generating multiple candidate responses in parallel and selecting the best one with a Probabilistic Pivot Tournament.

pip install git+https://github.com/llm-as-a-verifier/TurboAgent

How it works

Fine-grained Reward Estimation

Rather than reducing each distribution into a single discrete score (as in LLM-as-a-Judge), LLM-as-a-Verifier approximates the reward of a trajectory $\tau$ on task $x$ as:

$$ R(x, \tau) = \frac{1}{CK} \sum_{c=1}^{C} \sum_{k=1}^{K} \sum_{g=1}^{G} p_{\theta}(v_g \mid x, c, \tau),\phi(v_g) $$

Probabilistic Pivot Tournament

To pick the best of N candidate trajectories, a round-robin tournament scores all $\binom{N}{2}$ pairs. Probabilistic Pivot Tournament (PPT) is a cost efficient ranking algorithm in which every candidate is compared only against a small set of pivots, reducing the budget from $\mathcal{O}(N^2)$ to $\mathcal{O}(Nk)$ .

Citation

@misc{kwok2026llmasaverifiergeneralpurposeverificationframework,
      title={LLM-as-a-Verifier: A General-Purpose Verification Framework}, 
      author={Jacky Kwok and Shulu Li and Pranav Atreya and Yuejiang Liu and Yixing Jiang and Chelsea Finn and Marco Pavone and Ion Stoica and Azalia Mirhoseini},
      year={2026},
      eprint={2607.05391},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.05391}, 
}