DEVOURED
Mixture-of-Kittens: our open-source MoE megakernel for NVL72s
Cursor's open-source 'Mixture-of-Kittens' megakernel boosts MoE training speed by 1.41x on NVL72 GPU racks by fusing computation and communication.
What: Engineers Stuart Sul, Nash Brown, Henry Wildermuth, William Lin, and Federico Cassano developed Mixture-of-Kittens (MoK), a deterministic megakernel for Mixture-of-Experts training. By leveraging pull-based communication and ring token buffers, it eliminates bottlenecks on NVIDIA's GB300 NVL72 hardware.
Why it matters: As training clusters reach massive scale, communication overhead becomes the primary limiting factor; fusing these operations at the kernel level is becoming a standard requirement for high-performance AI research.
Takeaway: Visit the Cursor GitHub repository to implement MoK in your training stack if you are running large-scale MoE models on NVL72 racks.
Deep dive
- Fuses communication and computation into a single kernel to maximize NVLink saturation.
- Uses pull-based communication to eliminate multi-node signalling overhead.
- Implements ring token buffers (macrobatching) to avoid CPU-GPU synchronization bottlenecks.
- Fully deterministic design supports consistent training across different hardware schedules.
- Achieves up to 2.37x faster MXFP8 forward throughput compared to existing public baselines.
- Built to run on Blackwell-generation hardware with NVL72 interconnects.
Decoder
- Megakernel: An optimized kernel design that fuses multiple operations into a single GPU instruction set, reducing the overhead caused by frequent kernel launch boundaries.
- NVL72: A high-density server rack configuration featuring 72 interconnected Blackwell GPUs, designed for massive-scale training workloads.
- MXFP8: A micro-scaling floating point format that provides the efficiency of 8-bit precision with closer-to-FP16 accuracy, ideal for training large models.
Original article
Full article content is not available for inline reading.
Explore all 55 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi