DEVOURED
China's Xiaomi MiMo Is Now 15X Faster Than ChatGPT and Claude
Xiaomi’s new MiMo-V2.5-Pro-UltraSpeed model hits 1,000 tokens per second, claiming 15x faster inference than top-tier models like Claude and ChatGPT.
What: Xiaomi and inference partner TileRT reached 1,000 tokens/second using a 1-trillion-parameter model by applying FP4 quantization to expert layers and employing DFlash speculative decoding on an 8-GPU node.
Why it matters: The shift toward speculative decoding and sub-8-bit quantization is becoming the primary path for scaling inference throughput without exponentially increasing hardware costs.
Deep dive
- Architecture: 1-trillion-parameter model with expert layer optimization.
- Technique: Uses DFlash speculative decoding to propose entire blocks of tokens at once instead of sequential generation.
- Quantization: Employs FP4 to reduce memory bandwidth bottlenecks during inference.
- Hardware: Tested on standard 8-GPU commodity nodes, highlighting a move away from specialized, proprietary hardware.
Decoder
- FP4 Quantization: A technique that reduces numerical precision to 4 bits to compress model weights and speed up processing.
- Speculative Decoding: A method where a smaller, faster model drafts a sequence of tokens, which a larger model then validates in parallel.
- Token per second (TPS): A metric measuring the speed of LLM text generation.
Original article
Full article content is not available for inline reading.
Explore all 42 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi