DEVOURED
Inside Prometheus: Meta's Journey to a Gigawatt-Scale Supercluster
Meta is scaling its 'Prometheus' supercluster to 1 million GPUs, achieving over 85% goodput through liquid cooling and custom networking.
What: Located in New Albany, Ohio, the Prometheus cluster uses NVIDIA GB200 and GB300 platforms to train frontier models, leveraging custom Ethernet fabrics and in-rack spare hardware to minimize interruptions.
Why it matters: Managing reliability at the million-GPU scale is shifting from software-only fault tolerance to co-engineered hardware systems where physical infrastructure design directly dictates training stability.
Deep dive
- Deploying a gigawatt-scale cluster across multiple buildings interconnected by thousands of miles of fiber.
- Uses 72-GPU domains (NVL72) as the primary failure unit, with in-rack spares to prevent full-job restarts.
- Achieved a 99% reduction in interruption events by focusing on NVLink signal integrity and automated triage workflows.
- Implements 'Dynamo Dimmer' for cluster-wide power management to handle 30-40% load transients during synchronous compute phases.
- Uses Non-Scheduled Fabrics (NSF) with 51.2T switches to manage high-bandwidth backend traffic.
- Standardizing on facility liquid cooling (FLC) to maintain thermal limits at high power densities.
Decoder
- Goodput: The amount of productive time a cluster spends performing useful computation as a percentage of total training time.
- NVL72: An NVIDIA rack architecture where 72 GPUs are connected via NVLink, acting as a single, massive accelerator domain.
- Collectives: Operations in distributed training (like All-Reduce) that coordinate data across all GPUs in a cluster, requiring high-bandwidth, synchronized communication.
Original article
Full article content is not available for inline reading.
Explore all 55 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi