DEVOURED
Model Size Scaling in 2023-2031
Model scaling is shifting from compute-constrained to data-constrained, requiring massive 1.4 quadrillion parameter models by 2031 to maintain performance.
What: Vladimir Nesov analyzes hardware constraints and pretraining compute to project that models will reach 1.4 quadrillion parameters by 2031, with 30x sparsity, to counteract a 15x training data shortfall.
Why it matters: This indicates that future AI scaling will rely heavily on sparsity and algorithmic efficiency rather than just raw dense parameter growth as unique high-quality training data exhausts.
Deep dive
- Projected model sizes scale from 10T parameters in 2026 to 1.4 quadrillion in 2031.
- Token generation speed is increasingly bottlenecked by HBM capacity for weights and KV-cache.
- Pipelining constraints significantly limit the feasible parameter size of models on current hardware.
- Training data unique availability is estimated at 200T tokens, forcing models to repeat data multiple times as they grow.
- Sparsity increases (from 8x to 30x) are required to keep inference costs manageable as model sizes explode.
- Pretraining compute availability is expected to reach 40 GW of first-party infrastructure by 2030.
Decoder
- HBM (High Bandwidth Memory): Specialized memory integrated directly with processors to provide the high-speed data throughput required for AI model weights and context caches.
- KV-Cache: The memory buffer storing keys and values for previously processed tokens, allowing models to generate subsequent tokens without recomputing the entire history.
- Pipeline Stages: Dividing a model's layers across multiple hardware units to balance memory load and increase throughput for large models.
- Sparsity: A technique where only a fraction of a model's parameters (active params) are computed per inference, allowing for larger total model capacity without proportional compute costs.
- Speculative Decoding: An approach where a smaller model drafts future tokens, which a larger model then verifies in parallel to improve total generation speed.
Original article
Full article content is not available for inline reading.
Explore all 42 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi