Back to Devoured - May 07, 2026
DEVOURED
vLLM V0 to V1: Correctness Before Corrections in RL

vLLM V0 to V1: Correctness Before Corrections in RL

AI infrastructureml
ServiceNow AI's migration from vLLM 0.8.5 to 0.18.1 broke RL training until they debugged four inference issues, the last being an fp32 precision mismatch in the final projection layer that silently corrupted token probabilities.
What: Rafael Pardinas and Ehsan Kamalloo from ServiceNow AI migrated PipelineRL from vLLM 0.8.5 to 0.18.1 and found training metrics diverged from their reference run. They restored parity through four fixes: switching to processed_logprobs mode (includes temperature scaling and filtering), disabling V1's new prefix caching and async scheduling defaults, matching weight updates to V0's keep-inflight-requests behavior, and computing the lm_head final projection in fp32 rather than lower precision.
Why it matters: Inference engine updates can silently break RL training through logprob discrepancies that only surface in policy-ratio and KL computations. ServiceNow's methodology—fix backend correctness before adding algorithmic corrections—prevents masking broken infrastructure with compensatory objective-side changes. The fp32 lm_head issue echoes similar problems in MiniMax-M1's technical report and appears in ScaleRL's published recipe.
Deep dive
  • ServiceNow AI uses vLLM as the inference engine for PipelineRL rollout generation; the engine samples tokens and returns logprobs that the trainer uses to compute policy ratios, KL divergence, clip rate, entropy, and reward
  • The initial vLLM V1 (0.18.1) migration from V0 (0.8.5) broke training: clip rate, KL, entropy, and reward all diverged from the reference run early in training
  • First fix: logprob semantics—V1 returns raw logprobs by default (before temperature, penalties, filtering), but PipelineRL expected processed logprobs from the sampling distribution; fixed with logprobs-mode=processed_logprobs
  • Second fix: runtime defaults—V1 enabled prefix caching and async scheduling by default; these were V1-only runtime differences that changed execution paths; disabled both to match V0: enable-prefix-caching=false, async-scheduling=false
  • Prefix caching is normally correctness-preserving, but in online RL with inflight weight updates, a cache hit can reuse state computed before a weight update when the cache policy ignores the update boundary
  • Third fix: weight update path—made V1 match V0's inflight update model using await engine.pause_generation(mode="keep", clear_cache=False) rather than draining requests or invalidating cache
  • Runtime lag (steps the rollout server lags behind the trainer) was higher in the broken V1 path than in the corrected run, serving as a diagnostic signal
  • Fourth fix: fp32 lm_head—the trainer used fp32 for the final projection; the rollout backend had to match that precision for logits computation to maintain parity
  • The fp32 head issue appears in MiniMax-M1's technical report (they traced a train/inference token-probability mismatch to the output head and fixed it with fp32) and in ScaleRL's published RL recipe
  • After all four fixes, the V1 run tracked the V0 reference across clip rate, KL, entropy, and reward; policy ratio stayed centered at 1.0 within 0.0001 deviation
  • Negative results ruled out common explanations: processed_logprobs alone didn't fix training mismatch; batch-size invariance tests still showed mismatch; treating the first V1 run as a fair baseline was wrong because it had multiple V1-only defaults enabled
  • Core principle: fix backend correctness first, then add corrections for remaining mismatch; adding objective-side corrections (truncated importance sampling, ratio reweighting) before fixing inference correctness mixes two questions and can mask broken infrastructure with algorithmic compensations
  • The next improvement after backend parity is the standard async/off-policy cleanup: keep explicit behavior-policy logprobs from rollout time, recompute trainer-side old-policy logprobs at optimization time, separate backend mismatch correction from policy-update ratio
Decoder
  • vLLM: Fast inference engine for large language models, used to generate rollouts in RL training
  • lm_head: Final projection layer in a language model that converts hidden states to token logits (scores for each vocabulary token)
  • processed logprobs: Log probabilities computed after applying sampling parameters (temperature scaling, top-k/top-p filtering, repetition penalties) rather than from raw model outputs
  • policy ratio: In RL training, the probability of a sampled token under the current policy divided by its probability under the policy that originally generated the rollout; used to compute importance weights
  • prefix caching: Optimization that reuses key-value cache computation for repeated prompt prefixes across inference requests
  • inflight weight updates: Updating model weights while inference requests are still being processed, without draining the request queue
Original article

The vLLM V1 update improved inference correctness by addressing discrepancies in logprob computation, runtime defaults, inflight weight updates, and final projection precision. Key fixes included adjusting processed logprobs, disabling prefix caching, matching weight update models, and ensuring fp32 lm_head computation to align with vLLM V0's behavior. These changes resolved initial training mismatches, ensuring the new engine maintains expected RL performance without unnecessary objective-side corrections.

Explore all 73 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi