In December 2023, Chris Bakke talked a Chevrolet dealership’s website assistant into agreeing in chat to sell him a 2024 Tahoe for $1.00, and got it to agree the deal was “a legally binding offer – no takesies backsies.” The screenshot went around the internet, the dealership pulled the bot, and the story became the canonical example of what happens when you put a language model in front of your business with nothing between it and the customer.

LLM guardrails check inputs, generated replies, or proposed tool calls against application rules, then allow, flag, or block the operation. They can combine model-based checks, such as detecting an unauthorized commitment, with deterministic rules, such as rejecting a quote below an approved price.

When these checks run before an operation proceeds, they add latency to the request. This underscores an engineering problem: how do you enforce the rules within your application’s latency and cost budgets?

For model-based checks, options include a general-purpose LLM judge, a fine-tuned small model, or a purpose-built decision model such as TypeSafe’s Jev. This post compares Jev with OpenAI’s GPT-5.4 nano in a dealership chatbot demo, measuring guardrail latency and cost across two attack sequences. We also discuss the deployment tradeoffs of fine-tuning a small model, which we did not benchmark here.

We created a demo to show guardrails with Jev, compared against LLMs. The code, attack transcripts, and measurement harness are all in the typesafe-guardrails repo.

How we tested Jev and GPT-5.4 nano as the LLM judge

The demo we created rebuilds the dealership: a chatbot that can quote prices and call a tool to record an offer. Except this time, we added guardrails. There are several, each screening a different boundary:

• the inbound customer message, before the agent sees it
• the draft reply, before it is sent back
• the tool arguments the agent wants to record
• a deterministic below-floor price check that needs no model at all

We run that whole set in one of three configurations:

• No guardrail: nothing sits between the model and the customer, so you can reproduce the original $1 Tahoe incident, albeit as a demo, not by buying an SUV for $1.
• Jev: a purpose-built decision model, reached through TypeSafe System One.
• LLM-as-a-Judge: a general model, GPT-5.4-nano, called through structured outputs. This is the baseline.

With the guardrail running, a normal purchase goes straight through, while the $1 Tahoe attack is stopped before the offer is ever recorded.

Under the hood, Jev is TypeSafe AI’s first “System One” model. The name is a nod to Kahneman’s “System 1” thinking: the fast, intuitive judgment you make in a single pass, as opposed to the slow, deliberate, step-by-step reasoning of System 2.

A generative LLM writing out its reasoning token by token is doing System 2 work for a job that only needs System 1. Jev is built for exactly that kind of structured, snap decision rather than open-ended generation. Instead of writing an answer token by token, it returns a typed answer with a probability distribution in a single parallel pass, which is where most of the speed comes from.

It is trained with Reinforcement Learning for Calibrated Decisions, which TypeSafe describes as training for calibrated probabilities.

We ran the same three-turn Tahoe attack through the Jev guardrail and the LLM-as-a-Judge guardrail, and measured two things:

• Latency: how long a single guardrail call takes, and how long the whole conversation takes to screen end to end.
• Cost: what each guardrail costs per conversation, worked out from the measured token counts against published prices.

Both engines use the same question definitions, thresholds, and decision functions. The agent’s generated replies can vary between runs, however, so the engines don’t necessarily evaluate identical text. These results compare the two engines in a live demo.

Same questions, same thresholds, same decisions, and the same answers in 0.7 seconds instead of 9.2 seconds. On the longer slow-burn attack the per-call gap holds, 117ms against 1,811ms, and both engines reach identical verdicts on both attacks.

Jev vs. GPT-5.4 nano: guardrail latency and cost

Across these two demo runs, Jev was 15-to-18x faster and 12-to-14x cheaper per call. Jev clears all five checks in 0.7 seconds where GPT-5.4-nano takes 9.2, and does it at list prices, not negotiated rates, with identical verdicts. That reduction makes Jev worth testing against your application’s latency budget and accuracy requirements.

The cost gap is structural, not a pricing quirk. Jev returns a typed distribution, so it emits far fewer output tokens, and the ones it does emit are free, while the LLM writes about three times as many output tokens at the highest per-token rate on its bill.

Why we chose GPT-5.4 nano as the LLM judge

The easiest way to win a benchmark is to pick a weak opponent. We went the other way and picked the baseline that flatters Jev least.

We used GPT-5.4 nano as a small-model baseline with structured outputs for the guardrail decisions.

The obvious cheaper candidate, GPT-4.1-nano, disqualified itself on accuracy: asked whether a reply containing the verbatim words “and that’s a legally binding offer” implied a binding commitment, it scored the claim at 0.1, near-certain that nothing binding had happened. That is the exact failure the guardrail exists to catch. The model we chose is the one that made Jev’s win the narrowest.

How the guardrail checks agent inputs, replies, and tool calls

Speed from a smaller model would be a hollow result if it came from a dumber guardrail. It does not, because the design difference is structural rather than a matter of scale.

The first difference is that Jev returns typed answers. Ask it a yes-or-no question and you get back a probability, not a sentence you have to interpret. Ask it to pick a level of risk and you get an ordered choice. The answer says what the model concluded, and the shape of the distribution says how sure it is.

{
  "claims_binding": { "probability": 0.97 },
  "commitment": {
    "score": 2.0,
    "confidence": 0.94,
    "probabilities": {
      "no commitment, informational or a question": 0.01,
      "informal encouragement, no price agreed": 0.05,
      "states a firm price or makes a commitment": 0.94
    }
  }
}

The second difference is where the guardrail sits. It checks three boundaries, not one: the inbound customer message, the draft reply before it is sent, and the arguments the agent wants to pass to a tool.

That last boundary is the one an input-only guardrail can never cover. The second attack in the repo never jailbreaks the conversation at all. Every message looks fine. The block lands only at the tool boundary, on the arguments the agent assembled to record a $1 offer. If you are only screening inbound text, you never see it.

And because each check runs in around 100ms, stacking all three boundaries on the hot path still costs the customer a fraction of a second, so you can run every guardrail on every turn without a noticeable lag.

The third difference is that the policy is not in the prompt. The floor price for the Tahoe LT, $54,500 against an MSRP of $58,195, lives in guardrail configuration and is kept out of the customer-facing agent’s context.

None of this is exotic. It is the difference between a model that returns structured decisions and a model that returns prose you hope to parse correctly. That difference is what lets the guardrail be small, fast, and cheap without being naive.

Jev vs. fine-tuned small models: deployment tradeoffs

There is a legitimate middle path between a general LLM-as-judge and an off-the-shelf decision model: fine-tune your own small language model (SLM) on your own guardrail decisions.

On the one axis this post has hammered, latency, it genuinely works. A well fine-tuned small model can run in less than 200ms on the right hardware, right in Jev’s range. So the SLM route is not slow. If speed were the only question, it would be a fine answer.

The catch is that speed is the cheap part, and everything around it is not. The bill for a fine-tuned SLM lands in three places, none of them the inference call:

• The first is data, and it is the largest. The training compute is genuinely cheap. The expensive part is the labelled corpus you feed it: a set of allow, review, and block decisions across your whole policy, labelled well enough to trust.
• The second is serving. A fine-tune is not done when training finishes; it has to run somewhere, in production, at your latency.
• The third, and the sharpest in practice, is time to ship. Jev works today. It is a configuration change. A fine-tuned SLM screens nothing until you have gathered the data, labelled it, run the fine-tune, and stood up serving.

So the argument for a purpose-built decision model over a fine-tuned SLM is not that SLMs are slow. It is that you get the same real-time latency off the shelf, per call, without the upfront data bill, the standing serving cost, or the wait.

What to evaluate using Jev for production guardrails

Two engines, two attack sequences, identical decisions, and roughly an order of magnitude on both latency and cost.

The guardrail that catches the $1 Tahoe is the one that runs on every message, every reply, and every tool call, and it can only do that if it is fast enough and cheap enough that nobody is ever tempted to turn it off. A guardrail you can afford to leave on is the only kind that protects you.

Jev is not the only purpose-built decision model you will hear about. This is a fast-growing space, and more of these small, fast classifiers are shipping all the time, each with its own claims about speed, cost, and calibration.

That is good news, but it also means the model you pick today is a decision you should keep revisiting. This is where evals earn their keep: with a labelled set of your own guardrail decisions, you can measure each new model on your traffic instead of taking the benchmark on faith, and swap in a better one the moment the numbers say so.