DEVOURED
GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model
Meta achieved a 2x increase in foundation model training efficiency for ads by co-designing kernels, precision, and parallelism for jagged recommendation workloads.
What: Meta's GEM model training now hits 20-25% MFU using techniques like Jagged Flash Attention (JFA), MXFP8 precision, and SM-free collectives. The team scaled the model 4x in 12 months by addressing data skew and communication bottlenecks specific to recommendation systems.
Why it matters: It shows that standard LLM training optimizations often fail on recommendation data due to variable sequence lengths; co-designing the entire hardware/software stack is becoming mandatory for scaling.
Deep dive
- Jagged Flash Attention (JFA): A custom kernel handling variable-length user sequences without compute-wasting padding.
- GDPA (Generalized Dot-Product Attention): Unified attention module for diverse RecSys interaction patterns.
- SM-free collectives: Offloads data transfer from compute units to Copy Engines, reclaiming ~23 SMs.
- BBS (Base Batch Shuffling): Interleaving long/short sequences to balance load without cross-rank communication.
- MXFP8: End-to-end block-scaled low-precision training for faster throughput.
- 5D Parallelism: Topology-aware strategy mixing FSDP, Expert Parallelism, and model parallelism.
- NCCLX: Custom library for copy-free, SM-free communication.
Decoder
- MFU (Model FLOPs Utilization): The ratio of achieved hardware performance to the theoretical peak performance of the GPU.
- SM (Streaming Multiprocessor): The fundamental compute unit inside an NVIDIA GPU.
- FSDP (Fully Sharded Data Parallel): A technique to distribute large model parameters across multiple GPUs to fit them in memory.
- RoCE (RDMA over Converged Ethernet): A protocol allowing direct memory access between servers over Ethernet without involving the host CPUs.
Original article
Full article content is not available for inline reading.
Explore all 58 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi