Accelerating Gemma 4: faster inference with multi-token prediction drafters

Back to Devoured - May 06, 2026

DEVOURED

Accelerating Gemma 4: faster inference with multi-token prediction drafters

AI llmperformanceinference

Google's Multi-Token Prediction drafters make Gemma 4 inference 3x faster via speculative decoding, where a lightweight model drafts multiple tokens that the main model verifies in parallel, no quality loss.

What: Google released Multi-Token Prediction drafters for Gemma 4 models (E2B, E4B, 26B MoE, 31B Dense) using speculative decoding to achieve up to 3x speedup. A lightweight drafter predicts several tokens while the target model processes one, then the target verifies all predictions in parallel. Released under Apache 2.0, available on Hugging Face and Kaggle, supports transformers, MLX, vLLM, SGLang, Ollama. Gemma 4 has 60 million downloads since launch weeks ago.

Why it matters: By solving the memory-bandwidth bottleneck, speculative decoding makes frontier-class models practical on consumer GPUs and edge devices, enabling a shift from cloud-dependent to local-first LLM deployment.

Takeaway: Download Gemma 4 MTP drafters from Hugging Face or Kaggle and integrate them with your Gemma 4 models to achieve up to 3x faster inference.

Decoder

Speculative decoding: Technique where a lightweight drafter model predicts multiple future tokens that a larger target model verifies in parallel in a single pass, generating multiple tokens in the time normally needed for one
KV cache: Cached attention computation results from previous tokens, allowing models to avoid redundant recalculations during sequential text generation
Memory-bandwidth bound: Bottleneck where data transfer between VRAM and processor takes longer than actual computation, forcing compute units to sit idle while waiting for parameters
MoE (Mixture of Experts): Architecture splitting the model into multiple specialized sub-networks with a routing mechanism that selects which experts process each input

Original article

Gemma 4 models reduce latency bottlenecks and achieve improved responsiveness for developers by using Multi-Token Prediction drafters. These drafters deliver up to a 3x speedup without any degradation in output quality or reasoning logic due to a specialized speculative decoding architecture. Speculative decoding decouples token generation from verification. It utilizes idle compute to 'predict' several future tokens at once with the drafter in less time than it takes for the target model to process just one token. The target model then verifies all of these suggested tokens in parallel.

Explore all 66 articles in today's digest