End-to-End 4-Bit AI Training
Thinking Machines demonstrates how to train language models using 4-bit precision end-to-end, achieving significant speedups on NVIDIA GB300 hardware.
Deep dive
- NVFP4 offers 3.6x fewer bits than BF16 but provides up to 6x speedup on GB300 hardware.
- NVFP4 quantization error is managed through block-scaling (1x16 groups).
- Forward/backward mismatch is mitigated using stochastic re-quantization.
- RHT (Random Hadamard Transform) is used during tensor transmission to smooth values and reduce quantization errors.
- BBBK (Big Beautiful Blackwell Kernel) was developed to maximize NVLink, ALU, and Tensor Core utilization simultaneously.
Decoder
- NVFP4: A 4-bit floating point format optimized for NVIDIA's Blackwell GPU architecture.
- MXFP8: A 8-bit Microscaling format that provides a flexible range for representing values in deep learning.
- Hadamard Transform: A linear transformation used to redistribute values in a matrix, reducing outliers and aiding quantization efficiency.
- GEMM (General Matrix Multiply): The core mathematical operation (matrix-matrix multiplication) in deep learning inference and training.
Original article
Training large language models involves calculating and storing a lot of numbers. Using fewer bits for each value reduces memory usage and allows GPUs to perform calculations faster. However, there is a tradeoff: fewer bits also mean fewer representable numbers, leading to more aggressive rounding. Lower-precision representations ultimately hurt accuracy by adding error to the calculations.
For example, suppose we want to store the number 3.2, but the nearest values our numerical format can represent are 3 and 4. Rounding to 3 introduces a quantization error of − 0.2, the difference between the actual value and the one that can be represented in the quantized format. A single error like this may seem small, but these errors accumulate over training as rounding happens repeatedly across layers and optimization steps, which can impair the learning process.
This is the central challenge of low-precision training: how can we benefit from faster arithmetic without introducing errors that interfere with learning?
We manage this tradeoff through our use of three key numerical formats during training: BF16, MXFP8, and NVFP4. Each format’s precision is fixed, but its speed depends on the hardware being used. We train on NVIDIA’s Grace Blackwell Ultra platform, the GB300, which is designed to optimize matrix multiplications with NVFP4. Notably, NVFP4 is stored with 3.6x fewer bits than BF16, but it runs up to 6x faster on GB300s.
| Numerical format | Bits per value | GB300 PFLOPS |
|---|---|---|
| NVFP4 | (4 × 16 + 8) ÷ 16 = 4.5 | 15 |
| MXFP8 | (8 × 32 + 8) ÷ 32 = 8.25 | 5 |
| BF16 | 16 | 2.5 |
The NVFP4 acceleration on GB300s is very useful in high-volume inference, where the tradeoff leans in favor of speed. But compared to BF16 and MXFP8, the format greatly increases quantization error. This error matters more at different stages of training, and the tradeoff must be considered for each one separately: forward and backward passes in pretraining and supervised fine-tuning (SFT), sampling and training in reinforcement learning (RL).
Speed matters most in high-volume steps: pretraining, which runs on trillions of tokens, and RL sampling, which generates tens of thousands of tokens in each rollout. Each sampled token also requires loading the model’s weights from global memory, the cost of which scales with the number of bits needed to store the weights.
The backward passes that update the weights average over many tokens in a batch, which dilutes the impact of randomly distributed rounding errors. However, backward passes are very sensitive to quantization bias, introduced when values consistently round up or down during quantization. Instead of averaging out, this bias can compound across layers, samples, and training steps.
In training Inkling 2 Small, we made the following decisions:
- NVFP4 precision in the expert layers for forward passes, bolstered by a dedicated kernel.
- MXFP8 precision for backward passes with re-quantization and stochastic rounding, techniques for reducing gradient bias.
- Quantization techniques for accelerating GPU communications, which alleviated a bottleneck that surfaced after we sped up computation.
Inkling 2 Small is the largest model we are aware of that was trained end-to-end with a native 4-bit forward pass. We configure numerics in the same way from the start of pretraining through the final RL rollout. This makes training simpler and eliminates the need for calibration when a quantized checkpoint is delivered at the end of training. In our largest jobs, this recipe provides us with an 8% wall-clock speedup in pretraining and a 20% speedup in RL over full MXFP8 training.
Choosing a target for quantization
Ideally, we would like to reduce precision where it has the least effect on model accuracy, while offering a substantial overall speedup. The layers that best fit this profile are Mixture of Experts (MoE) layers, which are computationally expensive and empirically tend to harm accuracy less than other layers when quantized.
Earlier this year, we implemented full MXFP8 training of our MoE layers and found it barely affected our model’s compute efficiency (CE), a measure of how training and evaluation losses change when a new technique is introduced. However, our initial experiments with full NVFP4 training resulted in very large hits to our model’s CE, degrading accuracy beyond what we were willing to accept.
When isolating the effects of NVFP4 quantization in the forward and backward passes, we found that focusing on the forward pass generally offered better tradeoffs between speed and accuracy. But quantizing just the forward pass carries a number of implications for interactions with the backward pass, GPU communications, and more. To understand these, we’ll first examine how NVFP4 works in detail.
Low-precision formats on Blackwell
Four bits can represent up to 16 different values: in the case of FP4, these are ±{0, 0.5, 1, 1.5, 2, 3, 4, 6}. The obvious limitation is in representing numbers larger than 6, which is solved by block scaling: storing a scale factor that multiplies all values in a consecutive block. NVFP4 scales each block of 16 consecutive values based on the largest one in the block (in absolute terms). If that value is −54, for example, it would be represented as −6 with a scale factor of 9, and every value in the block would be rounded based on that scale.
Blackwell GPUs also offer microscaling (MX) formats such as MXFP4 and MXFP8, which scale blocks of 32 values. The formats used for these scale factors are different as well: NVFP4 scale factors are E4M3, making them more precise but limited in their range, while MXFP4/MXFP8 scale factors are E8M0, far less precise but able to represent a very wide range of values. The increased precision offered by NVFP4 scale factors tends to represent input data more accurately in LLM workloads, resulting in less quantization error than MXFP4.
This combination of FP4 and FP8 allows NVFP4 to represent magnitudes as low as 2⁻¹⁰ and as high as 2688. In practice, we will often want to represent values outside of this range, so NVFP4 is generally used with a second outer-level scale factor which is saved in FP32 and is applied across a very large group of values, such as an entire tensor. This allows each quantized value to be reconstructed as xᵢᶠᵖ⁴ × xᵢ//₁₆ᶠᵖ⁸ × xᶠᵖ³², which effectively allows us to shift the representable range up or down. Setting xᶠᵖ³² to 2, for example, shifts the range of representable magnitudes from [2⁻¹⁰, 2688] to [2⁻⁹, 5376], and setting it to 0.5 shifts the range in the opposite direction.
Block-scaled backward passes
A forward pass through the network multiplies the quantized activations by the quantized weights: y = x_q w_qᵀ. Computing a dot product with NVFP4 takes blocks of 16 values in a row of x_q and a column of w_qᵀ, multiplies the pairs, and then multiplies the sum by the scale of each block.
To compute the backward pass, we need to transpose the tensors from the forward pass: dx = g w_q and dw = gᵀ x_q. Now, the blocks take 16 values from columns of x_q and rows of w_qᵀ: each value is grouped with 15 different neighbors, resulting in different scale factors and different quantized values represented. This creates a mismatch between the tensors used in the forward and backward passes, which can lead to inaccurate gradient computation. For weight tensors, a straightforward solution is suggested by NVIDIA: rather than computing scale factors for each 1×16 block, they compute a single scale factor based on the largest value in each 16×16 square tile, keeping the scale factors identical when the tensor is transposed.
The main drawback of this approach is that it can cause a lot of values to underflow. Increasing the block size from 16 values to 256 makes it highly sensitive to outliers: a single large value among 256 can push the scale factor up, causing the remaining values to be rounded to zero.
We tried this approach but ultimately found computing scale factors over 16×16 tiles to be too coarse, hurting CE beyond what we were willing to accept. For NVFP4 training to work, we needed to quantize weights with 1×16 scale factors while avoiding the mismatch introduced during a block-scaled transpose.
Re-quantization with stochastic rounding
The use of 16×16 tiles eliminates the mismatch by enabling bitwise-identical representations between the forward and backward passes — w_q = (w_qᵀ)ᵀ — but perfect identity isn’t strictly necessary. The important requirement is that the tensors used in the forward and backward passes are equal in expectation: E[w_q] = E[(w_qᵀ)ᵀ]. Preserving the expectation addresses the problem of bias in the backward pass, and loosening the constraint on bitwise equality allows for a few different approaches to the transpose problem posed earlier.
We opted for a straightforward solution: de-quantizing tensors from the forward pass, transposing them in higher precision, and then re-quantizing them with stochastic rounding (SR). Unlike standard quantization, also known as round-to-nearest quantization, SR rounds each value up or down probabilistically, in inverse proportion to its distance from each neighbor.
For example, let’s assume that the forward pass tensor had a value of 5.4, quantized in a block with scale 9. 5.4 divided by 9 equals 0.6, which rounds to the nearest FP4 value of 0.5. The quantized value used in the forward pass is 4.5, which is now the value we want to recover in the backward pass (in expectation).
We can de-quantize this value to have it represented in high precision, but in the backward pass that value is now grouped in a different block. Let’s take its new scale to be 2. If we were re-quantizing with NVFP4, 4.5 ÷ 2 = 2.25 would now need to be quantized to FP4. Deterministic rounding would represent it as a 2, for a quantized value of 2 ⋅ 2 = 4. Stochastic rounding instead rounds it to 2 three quarters of the time and to 3, its nearest neighbor from above, a quarter of the time. The value of 4.5 from the forward pass is preserved in expectation after re-quantizing: 2_scale ⋅ (0.75 ⋅ 2 + 0.25 ⋅ 3) = 4.5. Crucially, stochastic rounding is also hardware-accelerated on Blackwell GPUs.
In designing this approach we drew inspiration from TetraJet-v2 and Quartet II, which also re-quantize the tensors from the forward pass. A benefit of applying SR directly to de-quantized values is that it lets us choose any precision during re-quantization. We found that stochastically re-quantizing weights and activations to MXFP8, and then computing gradients using MXFP8 matrix multiplications, offers the best tradeoff between accuracy and speed.
Quantized communications
With our new NVFP4 matrix multiplications (GEMMs), our MoE layer’s forward pass should theoretically have gotten about three times faster compared to our MXFP8 implementation. But after reducing precision, we noticed we were running up against a bottleneck: communications during expert parallelism.
Since MoE layers are often very large, it is common to split up experts between different GPUs. When computing the MoE layer, a small router decides where each token should go, and tokens are then dispatched to the GPUs containing the routed experts. After completing the GEMMs, the computations of these routed experts are returned to the source GPU and combined to create the MoE layer’s final output. On an NVIDIA GB300 NVL72 system, the tokens will typically be sent over NVLink, which has a bandwidth of 900 GB/s in each direction. At the same time, the NVFP4 GEMMs will run at a speed of up to 15 PFLOPS. Dividing one by the other, each GPU could perform as many as 16,000 floating point operations in the time it takes to receive a single byte. Unless every byte that arrives brings at least that much computation with it, the Tensor Cores sit idle waiting on NVLink.
We found this imbalance created bottlenecks where either the MoE GEMMs were unable to execute because they were waiting to receive the dispatched tokens, or the following operation (in Inkling 2 Small, short convolutions) was unable to start because it was waiting to receive the combined tokens. During dispatch, we avoid this issue by dispatching tokens in NVFP4. Like TetraJet-v2 and Quartet II, our backward pass is computed using quantized — not high-precision — activations from the forward pass, which reduces the amount of data sent over NVLink enough to alleviate the dispatch bottleneck. During combine, however, the output activations need to remain in higher precision (by default, BF16) to avoid impairing model performance.
We alleviate the communications bottleneck by quantizing the data transmitted during combine as well. We initially experimented with quantizing the resulting activations to MXFP8, but found that this impaired performance too much due to MXFP8’s poor representation of outlier values. Instead, we found that performing a 32-dimensional random Hadamard transform (RHT) and then quantizing to block-scaled S2F6, another 8-bit format, resulted in no measurable accuracy degradation during training.
RHTs apply a rotation which tends to smooth out the input data. This can reduce quantization error when the input data contains outliers. S2F6 is a fixed-point format with two signed integer bits and six fractional bits, and it can represent linearly interpolated values from -128/64 to 127/64, closely resembling an integer format.
Kernels for low-precision training
Taking full advantage of NVFP4 requires specialized kernels which we’ve developed in-house for our low-precision recipes. Our MoE fusion kernel stack is codenamed BBBK, the Big Beautiful Blackwell Kernel. It powers training and inference with peak hardware utilization while maintaining fully deterministic computation.
The process for computing MoE layers is generally dominated by matrix multiplications. We saturate the Tensor Cores using the largest 2SM UMMA instruction unique to GB300 (K=96), fed by a 6-stage multiplicand pipeline and a 7-stage scale factor pipeline. Our precision recipes, however, also place heavy demands on the GPU’s arithmetic logic units (ALUs). Even when the Tensor Cores are saturated, kernels can stall on ALU-heavy steps such as computing scale factors, re-quantizing weights, applying stochastic rounding, and coalescing data transfers.
Recipe details
We took inspiration from several other existing works on NVFP4 training to round out our recipe. We keep the last 16.6% of layers in higher precision, which we compute in MXFP8 rather than BF16. When quantizing activations, we use Four Over Six (4/6), a technique which selects between two scale factors during NVFP4 quantization in order to reduce quantization error. We forgo 4/6 when quantizing weights.
Drawing inspiration from Composer 2, we use per-expert outer scales when quantizing weights and per-token outer scales when quantizing activations, allowing for higher efficiency, batch-invariance, and no worry about causality leakage. In our largest training jobs, this recipe has worked without changes during both SFT and RL. In pretraining Inkling 2 Small it costs about 2% in compute efficiency, meaning the model reaches the loss it would have reached with about 2% fewer FLOPs if trained in MXFP8, in exchange for the speedup provided by NVFP4. In RL the gain was much larger: a substantial end-to-end speedup with no discernible difference in reward.
Closing thoughts
A decade ago, it was unclear whether neural networks could be trained using fewer than 32 bits per parameter: the default format remained FP32 as recently as 2020, when BF16 first came to NVIDIA GPUs with Ampere. Today, we often find ourselves referring to BF16 as “high precision,” and we have found robust use cases for training with 8-bit and even 4-bit formats! We’re inspired by NVIDIA’s forward-thinking hardware design that has made these advancements possible. We will continue to push forward: we are actively exploring recipes that enable FP4 computation of the backward pass, as well as how our recipes will evolve on NVIDIA’s latest Vera Rubin systems.
The acceleration of both our pre- and post-training is the result of a constellation of improvements in science and engineering, touching every part of the stack and compounding on each other. Months ago, a training run of a model similar in size to Inkling 2 Small took several weeks; Inkling 2 Small was pretrained in just 68 hours on GB300 NVL72 racks. Our numerics recipe is just one piece of these many improvements, and one we’re continuing to iterate on.
Citation
Please cite this work as:
Cook, Jack and Thinking Machines Lab, "End‑to‑End 4‑Bit Training",
Thinking Machines Lab: Connectionism, Nov 2026.
@article{cook2026lowprecision,
author = {Jack Cook and Thinking Machines Lab},
title = {End‑to‑End 4‑Bit Training},
journal = {Thinking Machines Lab: Connectionism},
year = {2026},
note = {https://thinkingmachines.ai/blog/end-to-end-4-bit-training/},
} Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi