DEVOURED
Lighthouse Attention
Nous Research's Lighthouse Attention delivers 17x faster training at 512K context using symmetric pooling and vanilla FlashAttention, no sparse kernels.
What: Nous Research published Lighthouse Attention (arXiv 2605.06554), a hierarchical attention method achieving 17x faster forward+backward passes at 512K context on a single B200 GPU. It pools queries, keys, and values symmetrically across a pyramid, scores via L2 norms, and runs FlashAttention on the gathered sub-sequence. Validated at 530M parameters with 16k optimizer steps over 50B tokens, delivering 1.4-1.7x pretraining speedup at 98K context.
Why it matters: This signals a shift toward sparse attention methods that preserve dense model competence through two-stage training, avoiding architectural lock-in of custom sparse kernels while enabling practical long-context pretraining at previously prohibitive scales.
Deep dive
- Lighthouse Attention addresses the quadratic compute cost of long-context training through selection-based hierarchical attention that runs ~17× faster than standard attention at 512K context on a single B200 GPU
- Pools queries, keys, and values symmetrically across a multi-resolution pyramid (unlike prior asymmetric methods that keep queries at full resolution), enabling O(S² · d) complexity where S ≪ N
- Uses parameter-free L2 norm scoring to select entries across pyramid levels, avoiding learned scorer heads, auxiliary losses, or straight-through estimators that complicate training
- Gathers selected entries into a contiguous dense sub-sequence and runs vanilla FlashAttention — no custom sparse attention kernel required, inheriting all upstream FlashAttention optimizations
- Two-stage training recipe: majority of training with Lighthouse selection enabled, then brief standard-attention resume to convert back to a competent dense model
- Validated at 530M Llama-3 parameters with three split points (10k+6k, 11k+5k, 12k+4k steps); every recovered run matches or beats dense-from-scratch baseline at the same 50B-token budget
- Delivers 1.4× to 1.7× end-to-end pretraining speedup at 98K context while reducing computational costs by 75-106 B200-hours per run
- Pyramid hyperparameters (L ∈ {3,4,5}, p ∈ {2,4,8}) are forgiving — all combinations land within ~0.02 nats of each other
- Supports context parallelism for 1M-token training across 32 B200 GPUs through ring attention on the dense sub-sequence, avoiding sparse-aware collectives
- Central correctness claim: sparse training does not compromise the model's ability to use full attention at inference, validated through successful SDPA-resume with 1-1.5k step recovery
- Limitations include requirement for dense-SDPA resumption (autoregressive decoding violates symmetric pooling assumption) and uncharacterized behavior in regimes where K must scale with N
- Reference implementation available as patch on torchtitan plus two new source files, supporting norm/dilated/GLA scorers and context parallelism
Decoder
- B200: Nvidia's Blackwell architecture datacenter GPU released in 2024, successor to H100
- Ring attention: Distributed attention technique that rotates key-value pairs across GPUs in a ring topology, enabling training on sequences longer than single-GPU memory
- Context parallelism (CP): Parallelization strategy that partitions a long sequence across multiple GPUs, as opposed to data parallelism (different sequences) or tensor parallelism (model weights)
- torchtitan: Meta's PyTorch-native training framework for large-scale LLM pretraining
Original article
Full article content is not available for inline reading.
Explore all 67 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi