DEVOURED
Gemma 4 Technical Report
Google DeepMind's Gemma 4 model family introduces a 'thinking mode' and encoder-free architecture to improve efficiency across 2.3B to 31B parameter sizes.
What: Gemma 4 models (2.3B-31B) feature a new thinking mode that generates reasoning traces, speculative decoding via Multi-Token Prediction (MTP) drafters, and a unified encoder-free architecture for audio and vision data, all under an Apache 2.0 license.
Why it matters: The shift toward encoder-free architectures and thinking modes indicates a move toward more flexible, natively multimodal models that require less overhead than traditional separate-encoder designs.
Takeaway: If you are running on-device or memory-constrained applications, test the quantized 2.3B or 4.5B models; they show significant efficiency gains in audio processing over Gemma 3n.
Deep dive
- New 'thinking mode' generates reasoning traces to improve performance on math and coding.
- 12B model features an encoder-free architecture, mapping raw audio/image patches directly to the embedding space.
- Speculative decoding supported by MTP drafter heads reduces latency.
- Long-context efficiency achieved via KV cache sharing and optimized RoPE positional encodings.
- Models trained on TPUv5p/TPUv6e clusters.
Decoder
- Mixture-of-Experts (MoE): An architecture where only a subset of parameters ('experts') is activated for any given input, enabling larger model capacity with lower compute costs.
- Multi-Token Prediction (MTP): A training technique where a model predicts several future tokens simultaneously, which can be leveraged to accelerate inference through speculative decoding.
- Speculative Decoding: Using a small, fast 'drafter' model to predict multiple tokens at once, which a larger model then verifies in parallel to reduce latency.
- Quantization-Aware Training (QAT): Training a model while simulating quantization effects to ensure that low-precision weights (e.g., int8 or int4) perform well at inference.
Original article
Full article content is not available for inline reading.
Explore all 56 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi