DEVOURED
Inside Prometheus: Meta's Journey to a Gigawatt-Scale Supercluster
Meta's Prometheus supercluster is scaling toward 1 million GPUs, utilizing massive liquid cooling and custom fabric designs to maximize training uptime.
What: Meta has built a gigawatt-scale infrastructure, Prometheus, spanning multiple buildings in New Albany, Ohio. It features NVIDIA GB200 and GB300 NVL72 platforms with 130+ kW racks, closed-loop liquid cooling, and a dedicated backend network based on Ethernet with 800G connectivity to maintain high training goodput.
Why it matters: Reliability at this scale is an engineering bottleneck; Meta is prioritizing 'goodput' (productive training time) over simple uptime, treating the entire cluster as a single distributed computer.
Deep dive
- Goodput is maintained above 85% for pre-training and 90% for RL runs.
- Employs spare GPUs in-rack for immediate recovery upon failure.
- Uses closed-loop liquid cooling for efficient power density.
- Networking stack includes 'Back-End' Ethernet fabric with sub-packet-level multipath routing.
- Future integration of NVIDIA Vera Rubin NVL72 platforms is already underway.
Decoder
- Goodput: The amount of productive work completed by a system, excluding overhead like data re-transmission or idle time during recovery.
- Blast Radius: The potential negative impact area of a system fault.
- NVL72: An NVIDIA platform architecture where 72 GPUs act as a single tightly coupled accelerator domain.
Original article
Full article content is not available for inline reading.
Explore all 58 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi