DEVOURED
Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention
Transformers aren't being replaced but made 10x more complex, as DeepSeek V4 and peers add KV compression to slash long-context memory costs.
What: Sebastian Raschka's technical deep-dive analyzes Gemma 4 (cross-layer KV sharing saves 2.7GB at 128K tokens), Laguna XS.2 (per-layer query-head budgeting), ZAYA1-8B (compressed convolutional attention in latent space), and DeepSeek V4 (manifold-constrained hyper-connections and compressed attention achieving 10% of V3.2's KV cache size at 1M tokens).
Why it matters: This signals state-of-the-art LLM architecture is evolving through incremental, specialized efficiency optimizations rather than paradigm shifts, with transformer implementations growing from ~50 lines (GPT-2) to 10x complexity (modern models) as long-context inference becomes the primary bottleneck.
Takeaway: Visit sebastianraschka.com/llm-architecture-gallery for visual explainers on GQA, MLA, sliding-window attention, and other attention mechanisms referenced in recent model releases.
Deep dive
- Gemma 4 E2B/E4B use cross-layer KV sharing where later layers reuse key-value states from earlier layers, saving roughly half the KV cache (2.7GB for E2B at 128K context)
- Per-layer embeddings (PLE) in Gemma 4 increase capacity through embedding parameters rather than scaling the transformer stack, making E2B "5.1B parameters" but "2.3B effective"
- Laguna XS.2 implements layer-wise attention budgeting with per-layer query-head counts: sliding-window layers get 8 query heads per KV head, full attention layers get 6
- ZAYA1-8B's Compressed Convolutional Attention (CCA) performs attention directly in compressed latent space (unlike MLA which decompresses before attention), reducing both KV cache and attention FLOPs
- CCA adds convolutional mixing on compressed Q and K tensors to give them local context before computing attention scores
- DeepSeek V4 introduces manifold-constrained hyper-connections (mHC), replacing single residual stream with parallel streams mixed via doubly stochastic matrices
- mHC adds only 6.7% training time overhead for 4 residual streams while enabling faster convergence (reaches baseline performance in half the tokens)
- DeepSeek V4's Compressed Sparse Attention (CSA) compresses every 4 tokens with sparse top-k selection; Heavily Compressed Attention (HCA) compresses every 128 tokens with dense attention
- CSA/HCA compress along sequence dimension (fewer KV entries) unlike MLA which compresses per-token representation
- At 1M tokens, DeepSeek V4-Pro uses 27% of V3.2's FLOPs and 10% of KV cache; V4-Flash uses 10% FLOPs and 7% cache
- Raschka estimates transformer block code complexity increased 10x from GPT-2 (~50-100 lines) to modern architectures with specialized attention variants
- Most 2026 open-weight models (Gemma 4, Qwen3.6, ZAYA1, Laguna, DeepSeek V4) focus on long-context efficiency over raw parameter scaling
- Article includes from-scratch implementations and architecture diagrams in Raschka's LLM Architecture Gallery
- Raschka's new book "Build A Reasoning Model (From Scratch)" (528 pages, going to print) covers inference-time scaling, self-refinement, reinforcement learning, and distillation
Decoder
- KV cache: Storage of previously computed key and value tensors in attention mechanisms, avoiding recomputation but growing linearly with context length and becoming the main memory bottleneck at long contexts
- Cross-layer attention: Architecture where later transformer layers reuse KV projections from earlier layers rather than computing their own, roughly halving KV cache size
- Per-layer embeddings (PLE): Technique giving each transformer layer its own token-specific embedding slice, adding capacity through cheap lookup parameters rather than expensive attention/FFN weights
- mHC (manifold-constrained hyper-connections): DeepSeek V4 design replacing single residual stream with parallel streams mixed via constrained linear transformations where matrices are doubly stochastic (non-negative, rows and columns sum to 1)
- CSA/HCA (Compressed Sparse/Heavily Compressed Attention): DeepSeek V4 attention variants that reduce cache by compressing along the sequence dimension (fewer KV entries) unlike MLA which compresses per-token representation
- CCA (Compressed Convolutional Attention): Performs attention computation directly in compressed latent space with convolutional mixing, reducing both KV cache and attention FLOPs during prefill and training
Original article
Full article content is not available for inline reading.
Explore all 67 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi