DEVOURED
vllm v0.28.0
vLLM v0.28.0 launches with massive performance improvements for Kimi-K3 and DeepSeek V4, including new tiered KV cache offloading.
What: The update features 584 commits and adds support for ROCm (AMD), Intel XPU, and disk-based KV cache offloading. It also introduces 'Decode Context Parallel' (DCP) support and adaptive speculative token budgeting to significantly reduce Time-to-First-Token (TTFT).
Why it matters: By offloading KV cache to disk and improving kernel-level efficiency, vLLM continues to push the limits of serving massive models on consumer or constrained hardware.
Takeaway: If you are running production inference, upgrade to v0.28.0 to benefit from the new memory-saving shared-expert sharding (saving ~17 GiB per GPU) and prefix caching enabled by default.
Deep dive
- Kimi-K3/DeepSeek V4: Dedicated optimizations including fused kernels and sparse MLA support.
- Speculative Decoding: New candidate selectors and confidence-scheduled verification to speed up token generation.
- Memory Management: Disk offloading for KV cache allows larger context handling with limited VRAM.
- New Defaults:
max_num_batched_tokensdoubled to 16,384 and prefix caching enabled for Mamba models. - Infrastructure: Improved ROCm and XPU support, plus a new standalone Rust frontend.
Decoder
- KV Cache: The key-value cache used by LLMs during inference to store pre-computed attention states, reducing redundant calculations.
- TTFT (Time-to-First-Token): The latency between sending a request and receiving the first part of the model's response.
- MLA (Multi-Head Latent Attention): An attention architecture used in models like DeepSeek to reduce memory usage during inference.
- Speculative Decoding: A technique where a smaller, faster model guesses tokens, and a larger model verifies them to increase throughput.
Original article
Full article content is not available for inline reading.
Explore all 65 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi