DEVOURED
BF16 LLM weights waste 33% of their bits—all in the exponent field—because every model from DeepSeek to Qwen clusters magnitudes in the same 2^-7 to 2^-6 range.
What: Analysis of 20+ models from DeepSeek, Qwen, Google, OpenAI, and NVIDIA (spanning 0.6B to 1.4T parameters) shows BF16 weights waste a third of their bits. The 8-bit exponent field carries only 2.6 bits of entropy because every model clusters weight magnitudes in the same 2^-7 to 2^-6 range—a pattern that holds across labs, architectures, and training recipes. FP8 improves to 80% efficiency, while FP4 formats (MXFP4, NVFP4, INT4) hit 93% by using block-level scales that force weight distributions to compress.
Why it matters: This reveals a fundamental tension in LLM serving: fixed-length number formats are hardware-friendly but information-theoretically wasteful. Quantization doesn't actually trade memory for compute as advertised—it just transfers less data to do proportionally more computation. The remaining 7-30% slack across formats suggests variable-length encoding could unlock further compression, but hardware constraints make fixed formats dominant.
Deep dive
- Methodology: Computed Shannon entropy of weight distributions across open-weight models from multiple labs (Qwen, DeepSeek, Google, OpenAI, Moonshot, MiniMax, NVIDIA, StepFun, Zhipu), parameter scales (0.6B to 1.4T), and storage formats (BF16, FP8, MXFP8, MXFP4, NVFP4, INT4). Shannon's source coding theorem sets the theoretical minimum bits needed per symbol.
- BF16 baseline: 16-bit weights carry 10.6 bits of entropy on average—66% efficiency. Waste is entirely in the exponent field: 2.6 bits used of 8 allocated. Mantissa fully utilized at 7/7 bits. Sign bit behaves like fair coin (1/1 bit).
- Universal magnitude distribution: All measured models cluster weight magnitudes between 2^-7 and 2^-6 in a narrow, unimodal distribution with long left tail. When normalized by mean and standard deviation, nearly every model collapses onto the same curve—independent of architecture, lab, or training recipe.
- Why the exponent wastes bits: Floating-point exponent encodes magnitude (roughly log₂|w|). BF16's 8-bit exponent can represent 256 magnitude ranges, but trained models only occupy a narrow band. This regularity is robust across all models measured.
- FP8 improvements: 1/4/3 bit allocation (sign/exponent/mantissa) carries 6.5/8 bits of entropy (~80% efficiency vs 66% for BF16). Improvement comes from reduced mantissa precision, not exponent efficiency—4-bit exponent still over-allocates for the ~2.6 bits needed.
- Sub-byte formats shift the distribution: FP4 formats give per-element exponent only 2 bits, less than the ~2.6 bits natural distribution wants. Solution: block-level scales absorb absolute magnitude, per-element bits cover within-block variation. This forces weight distribution to adapt rather than format absorbing slack.
- FP4 efficiency: MXFP4 (32-element blocks, E8M0 scale), NVFP4 (16-element blocks, E4M3 scale + FP32 tensor scale), and INT4 (4-bit integers + group scales) all reach ~93% efficiency. Per-element bits pushed to saturation; remaining 7% slack lives in block scales.
- Format comparison: BF16 and FP8 let natural weight distribution sit inside available budget. FP4 constraints tighten past natural floor, forcing distribution to compress. Different models at FP4 have different magnitude distributions than at BF16—format shapes the distribution.
- Memory vs compute bottleneck: LLM inference is bottlenecked by data transfer (weights, KV cache) when compute units idle waiting for memory bus. True solution would transform memory into compute: transfer less data, reconstruct via computation on receiver side.
- Quantization as compression: Quantization is lossy compression with unique property that you don't decompress—computation in compressed format is more efficient. But this means you never actually trade memory for compute; you transfer half the data to do twice the computation.
- Remaining opportunity: 7-30% slack remains depending on format. Variable-length encoding could extract remaining slack by abandoning fixed-length formats, but hardware optimization requires fixed formats. Fundamental tension between information theory and hardware constraints.
Decoder
- Shannon entropy: Information-theoretic measure of average information content per symbol in a distribution. Maximum entropy means all values equally likely (no wasted bits). Lower entropy means clustering, indicating format allocates more bits than needed.
- BF16/FP8/FP4: Floating-point number formats allocating different numbers of bits to sign, exponent (magnitude), and mantissa (precision). BF16 = 1+8+7 bits, FP8 = 1+4+3, FP4 = 1+2+1 plus block-level scales.
- MXFP4/NVFP4: Block-quantized 4-bit formats. MXFP4 groups 32 weights with an 8-bit shared scale. NVFP4 groups 16 weights with FP8 scales plus a tensor-level FP32 scale (double-quantization).
- QAT (Quantization-Aware Training): Training models with simulated quantization so weight distributions adapt to format constraints during training rather than post-hoc compression.
Original article
A lot of LLM inference is transferring data from one place to another and then computing on it when it's there. The most frustrating bottleneck in the system is when compute units sit idle because the data bus feeding them isn't fast enough. The solution is to transform memory into compute. Quantization is a nice trick, but it doesn't actually trade memory for compute - it transfers half as much data to a place to do twice as much computation.
Explore all 66 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi