DEVOURED
Pushing the Limits of Serving DeepSeek-V4-Pro
LMSYS researchers detail how they optimized DeepSeek-V4-Pro serving on H20 GPUs by using workload-specific profiles and fused kernel operations.
What: The team successfully served the 1.6-trillion-parameter DeepSeek-V4-Pro model on NVIDIA H20 GPUs by customizing deployment topologies—using PP2/PP4 for prefill and DP/EP for decode—and leveraging Humming MXFP4AFP8 weight reduction.
Why it matters: This proves that hardware constraints can be mitigated through software-level co-design and profile-specific optimization, avoiding the need for the latest top-tier Blackwell GPUs for every enterprise workload.
Deep dive
- Methodology over Hardware: The team emphasizes selecting profiles based on SLOs, context length, and concurrency rather than raw hardware specs.
- Humming Integration: Used
MXFP4AFP8(MXFP4 weights with FP8 activations) to reduce weight footprint on hardware lacking native FP4 support. - KV Capacity: Expanded cache capacity using 'Online C128' to allow for larger concurrent batches.
- Prefill Optimization: Switched from MoE-EP (Expert Parallelism) to MoE-TP (Tensor Parallelism) to avoid expert skew and stragglers.
- Decode Gains: Used DSpark speculative decoding and removed redundant quantization steps to improve throughput.
- Results: Achieved competitive performance ratios (1.42x) compared to benchmarks run on high-end hardware (B300), demonstrating the power of tailored serving stacks.
Decoder
- HBM (High Bandwidth Memory): A specialized memory architecture designed for high-performance processors, providing significantly faster data rates than standard DDR memory.
- MoE (Mixture-of-Experts): An architecture where only a fraction of the model's parameters are activated for each token, improving efficiency during inference.
- TTFT (Time To First Token): The delay between a user sending a request and the model generating its first response token.
- TPOT (Time Per Output Token): The average time taken to generate each subsequent token, a primary metric for interactive latency.
Original article
Full article content is not available for inline reading.
Explore all 57 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi