DEVOURED
Parallelizing Transformer Training
An interactive guide to transformer training parallelism, visualizing the bottlenecks where communication costs outweigh compute capacity.
What: Ezyang's guide details five strategies: pure data parallelism, FSDP, tensor parallelism, pipeline parallelism, and expert parallelism. It illustrates how communication overhead in AllGather/ReduceScatter operations interacts with compute-heavy matmuls to determine whether a configuration is compute-bound or bandwidth-bound.
Why it matters: As model sizes grow, selecting the right parallelism strategy depends on the operational intensity (FLOPs per byte) of the specific hardware interconnect, shifting the challenge from raw compute to managing network congestion.
Takeaway: Use the guide's model-specific interactive toggles to check if your cluster configuration (e.g., TPU v5p vs H100) is communication-bound for your target batch size.
Deep dive
- Data parallelism is simple but limited by weight replication memory usage.
- FSDP reduces memory usage by sharding optimizer states and parameters, but adds communication overhead.
- Tensor parallelism shards activations across devices, which helps reduce batch-per-device memory constraints.
- MoE models change the math by increasing weight-transfer traffic while keeping active FLOPs low, necessitating expert parallelism.
- Overlapping communication with computation is the primary goal to keep silicon utilization high.
- Bandwidth-bound regimes occur when the per-device batch size falls below the cluster's operational intensity.
Decoder
- AllReduce: A communication collective operation where multiple devices aggregate data (like gradients) and distribute the result to all participants.
- AllGather: An operation where each device collects data from all other devices, ensuring everyone has the full set.
- ReduceScatter: An operation that combines data from all devices and distributes specific shards of the result to each device.
- Roofline Model: A framework for evaluating hardware performance, plotting achieved FLOPs against memory or interconnect bandwidth to identify bottlenecks.
- Rematerialization: Also known as activation checkpointing; a technique that saves memory by recomputing intermediate activations during the backward pass instead of storing them.
Original article
Full article content is not available for inline reading.
Explore all 58 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi