DEVOURED
Inside Meta's Gigawatt-Scale AI Supercluster
Meta’s gigawatt-scale Prometheus supercluster aims to unite 1 million GPUs for training frontier models using closed-loop liquid cooling and advanced Ethernet networking.
What: Prometheus is a gigawatt-scale supercluster in New Albany, Ohio, built in phases to support Meta's Muse model family. It utilizes NVIDIA GB200/GB300 NVL72 hardware, custom liquid cooling, and a 800G Ethernet-based backend fabric for massive GPU-to-GPU interconnects.
Why it matters: This represents the emerging standard for 'hyperscale AI infrastructure,' where data center design, network topology, and power management become as critical to model performance as the model weights themselves.
Deep dive
- Goodput Focus: Prioritizing high 'goodput' (effective training progress) over raw peak uptime, achieving ~90% efficiency on RL jobs.
- Fault Domain Expansion: The shift to NVL72 architectures increases the fault blast radius by 9x, requiring a fundamental redesign of redundancy and sparing strategies.
- Networking Hierarchy: Employs a dual-network design (frontend web + backend GPU-to-GPU), using non-scheduled fabric (NSF) switches with adaptive routing to prevent congestion.
- Power Transients: Uses a cluster-wide power management system, Dynamo Dimmer, to cap GPU load and prevent infrastructure failures during synchronized compute bursts.
- Liquid Cooling: Employs closed-loop facility liquid cooling (FLC) to maintain thermal stability for high-density GPU racks that would be impractical to cool with air alone.
Decoder
- Goodput: The measure of productive work completed by a system, excluding resources wasted on retries, restarts, or hardware failures.
- NVL72: An NVIDIA rack architecture that interconnects 72 GPUs into a single compute domain with unified memory and high-speed NVLink bandwidth.
- Blast radius: The scope of a hardware or software failure, representing how many systems are impacted by a single component fault.
Original article
Full article content is not available for inline reading.
Explore all 43 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi