Back to Devoured - August 11, 2026
DEVOURED
A Controlled Study of Attention-Only Transformers

A Controlled Study of Attention-Only Transformers

AI researchtransformer ArXiv
Removing feed-forward layers from transformer architectures is viable, as attention-only models perform nearly identically when parameter budgets are reallocated to depth.
What: Researchers including Henry Ndubuaku demonstrated that 'Simple Attention Networks' (SANs) perform within 0.27% of standard transformers when parameter counts and training FLOPs are matched, with remaining gaps primarily in parametric memory/knowledge recall.
Why it matters: This challenges the assumption that feed-forward networks are strictly necessary for transformer performance, suggesting that attention mechanisms can carry significantly more of the model's computational burden than previously thought.
Deep dive
  • Researchers conducted a controlled comparison between standard transformers and attention-only decoders (SANs).
  • SANs were matched for parameter count, training FLOPs, and depth (2-48 layers).
  • Standard transformers outperformed SANs by 0.47 nats at matched depth.
  • Reallocating the 'freed' budget from FFN removal to attention depth closed the performance gap significantly.
  • Remaining performance deficits were isolated to knowledge-heavy tasks, where FFNs act as parametric memory.
  • QK-normalization was identified as a critical factor for maintaining training stability in 48-layer attention-only architectures.
Decoder
  • FFN (Feed-Forward Network): The part of a transformer block that typically accounts for the majority of a model's parameters and is often theorized to store factual knowledge.
  • Nat: A unit of information entropy, often used to measure the loss or performance of probabilistic models.
  • Parametric Memory: The information stored within the weights of a neural network rather than within its contextual input.
Original article

A Controlled Study of Attention-Only Transformers

Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once. We pretrain attention-only decoder transformers (Simple Attention Networks, SANs) against standard transformers matched separately for parameter count, training FLOPs, and depth (2 to 48 layers), for up to 105B tokens at 6M to 87M parameters. Deleting feed-forward layers in place is costly: the standard transformer leads by 0.47 nats at matched depth and 0.26 nats at matched FLOPs. Reallocating the freed budget into attention depth closes the gap: at matched parameters the difference is 0.006 nats (0.27 percent of loss), reproducible to one part in ten thousand across seed pairs, shrinking across 5B, 30B, and 105B budgets, and holding near 0.02 nats across a 29x size range. Three measurements localize the remaining gap to parametric recall: attention-only models are better on context-grounded answers and worse where knowledge must come from weights. Weight spectra show why: routing matrices (Q/K) crystallize early, content matrices accumulate rank slowly, and removing feed-forward layers relocates this accumulation to the attention output projection. QK-normalization, not feed-forward layers or residual gating, keeps 48-layer attention-only stacks trainable. The deficit concentrates on low-context query prediction and localizes there entirely by the largest budget. A pre-registered test confirms the account: it predicts a 0.02 to 0.05 nat gap on knowledge-dense web text; a matched pair trained on fineweb-edu measures 0.040. Within the tested regime, attention does the rest.

Explore all 43 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi