A cache hit is not proof that you skipped the work
A simple 'cache hit' report in LLM systems is insufficient evidence that actual work was bypassed, according to a new technical audit framework.
Deep dive
- KV-cache: Memory that stores previously computed key-value tokens for prompts, intended to speed up subsequent requests.
- Oracle: An independent truth-checker that calculates expected reusable prefixes based on exact token identity.
- Fail-closed: A verification design where a system defaults to 'unverified' unless all independent components (oracle, attestation, prompt work, identity) agree.
- Audit boundary: The specific point where input token reuse terminates due to changes in prompt content.
Decoder
- KV-cache: Memory that stores previously computed key-value pairs for prompt tokens to avoid re-generating them during inference.
- Token identity: The strict requirement that input tokens must be identical at the byte/ID level for a cache hit to be valid.
Original article
An exact duplicate reused 8 of 9 tokens and required 1 token of prompt work.
One interior mutation cut safe reuse to 3 of 9 tokens. Namespace isolation forced 0 reuse. A capacity revisit ended as the typed verdict evicted. Across all ten cases, cached and no-cache outputs stayed token-identical and every evaluator passed.
That is the shipped result.
Evidence boundary: this is a deterministic synthetic token-level control. It validates the auditor, oracle, attestation path, evaluator, schemas, privacy rules, hashes, and standalone verifier. It does not establish MLX or vLLM speedup, production cache correctness, provider identity, GPU performance, latency savings, or runtime-memory savings.
A runtime can report a cache hit without proving that the intended prefix matched. It can report eight cached tokens while the prompt path still recomputes them. It can reuse state and return a different answer. It can label an isolated namespace as a miss without telling you whether the result came from isolation, eviction, or a cold cache.
The cache event is one statement from one part of the system.
The engine attestation is the statement under test. LLMTraceFX checks it against five independent checks: the token oracle, observed prompt work, output-token identity, evaluator correctness, and the checksum-bound evidence bundle. The claim survives only when the chain agrees.
What the ten cases prove
The reference control runs a fixed request sequence through a token-granular cache. For each request, an oracle calculates the reusable prefix from exact token identity and the pinned cache policy. The engine reports its reuse. The prompt path records the work it processed. A no-cache control supplies the output identity and correctness checks.
| Case | Input | Expected | Attested | Work | Verdict |
|---|---|---|---|---|---|
| capacity-seed | 4 | 0 | 0 | 4 | verified miss |
| cold | 9 | 0 | 0 | 9 | verified miss |
| exact-duplicate | 9 | 8 | 8 | 1 | verified hit |
| interior-mutation | 9 | 3 | 3 | 6 | partial reuse |
| boundary-mutation | 9 | 4 | 4 | 5 | partial reuse |
| same-length-different-ids | 9 | 0 | 0 | 9 | verified miss |
| suffix-change | 9 | 8 | 8 | 1 | partial reuse |
| namespace-isolation | 9 | 0 | 0 | 9 | verified miss |
| capacity-pressure | 4 | 0 | 0 | 4 | verified miss |
| capacity-revisit | 4 | 0 | 0 | 4 | evicted |
The first two rows establish cold state. capacity-seed inserts a four-token candidate. cold then submits a nine-token request with no reusable prefix. Both are verified_miss.
The exact duplicate is the clean positive control. The oracle expects eight reusable tokens, the engine attests eight, and the prompt path processes one. The verdict is verified_hit.
The interior and boundary mutations change where the exact token prefix stops. The interior mutation reuses three tokens and processes six. The boundary mutation reuses four and processes five. Both are partial_reuse.
The same-length case is the trap. Its request still contains nine tokens, but the token IDs differ. The oracle expects zero reuse. The engine attests zero. The prompt path processes all nine. Equal length creates no identity.
The suffix-change case keeps an eight-token prefix. It processes the changed final token and returns partial_reuse, not a full hit.
Namespace isolation also forces zero reuse. Capacity pressure fills the controlled cache. The later capacity revisit has zero reuse and four tokens of prompt work, but its verdict is not a generic miss. The retained predecessor proof and controlled residency observations show that the earlier candidate left the cache, so the verifier returns evicted.
Every row preserves output identity: yes and evaluator: yes. Those checks do not change the cache verdict. They answer separate questions: did reuse alter the output, and did the result remain correct?
A hit needs an evidence chain
The independent oracle matters because the engine cannot verify itself. It derives the safe prefix from the exact request, cache state, namespace, and policy. That expected count then has to match both the engine's attestation and the amount of prompt work observed.
Output identity adds another boundary. A correct reuse count does not prove that the response stayed the same. The evaluator adds one more. Token-different output can still be correct, and token-identical output can still fail a task if the control itself is wrong.
Three lessons from nine tokens
Equal token count is not equal token identity
cold, same-length-different-ids, and namespace-isolation all contain nine input tokens and all process nine tokens. The reasons differ.
Mutation position defines the reusable prefix
Similarity is not the rule. Prefix identity is.
Eviction and isolation are not the same operational case
Zero reuse describes an amount. It does not describe a cause.
Reproduce the public proof
From the merged LLMTraceFX repository:
make kv-cache-demo
What this result does not prove
- MLX or vLLM speedup;
- production cache correctness;
- provider or hardware identity;
- GPU performance;
- latency reduction;
- runtime-memory savings;
- block-level cache behavior; or
- that a runtime integration has enough evidence merely because it emits cache events.
It proves a narrower and more useful thing: for a deterministic synthetic token-level control, the auditor records full reuse, partial reuse, verified misses, and eviction while the case evidence preserves namespace isolation. Output identity, evaluator correctness, privacy rules, and hash-bound verification remain intact.
The result
A cache hit can be true and still fail to prove that work was skipped.
The useful claim is stricter: the independent oracle expected the prefix, the engine attested it, the prompt path skipped it, the output stayed identical, the evaluator passed, and the verifier bound those facts to the exact public bundle.
That is when a cache event becomes evidence.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi