DEVOURED
Inside the megakernel serving engine for North Mini Code
Cohere introduced a megakernel-based serving engine that outperforms vLLM by up to 1.58x in decode throughput.
What: The engine uses a 'megakernel'—a persistent GPU kernel running an entire forward pass—to eliminate overheads like kernel launch latency and wave quantization stalls. It supports continuous batching and tool calling on H100 GPUs.
Why it matters: Megakernels are becoming essential for optimizing memory-bound autoregressive decoding, where traditional kernel-per-operation frameworks fail to saturate GPU hardware.
Deep dive
- Megakernel Architecture: Runs the entire forward pass as a single persistent kernel with exactly one threadblock per Streaming Multiprocessor (SM).
- Task Scheduling: Replaces explicit kernel boundaries with task descriptors and fine-grained counter-based barriers in global memory.
- Wave Quantization: Allows 'backfilling' idle SMs with ready work, unlike traditional kernel-based execution which suffers at small batch sizes.
- Weight Prefetching: Weights are streamed from HBM into shared memory before activation dependencies are satisfied, harvesting idle bandwidth.
- Integration: Compatible with continuous batching and paged attention while maintaining an OpenAI-compatible API.
Decoder
- Megakernel: A GPU execution strategy that fuses an entire model's forward pass into one persistent kernel, rather than launching dozens of separate kernels.
- Wave Quantization: An inefficiency where some SMs sit idle because there is not enough work to fill the final wave of GPU blocks.
- HBM (High Bandwidth Memory): Specialized GPU memory designed for high-speed data transfer to the processor.
- TMA (Tensor Memory Accelerator): Hardware-level engines in Nvidia Hopper GPUs for asynchronous data movement.
Original article
Full article content is not available for inline reading.
Explore all 55 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi