Key points
- The Prometheus supercluster in Ohio will reach approximately one million GPUs at full scale.
- Engineering improvements reduced daily training interruptions by 99% during the first phase.
- Meta uses reserved hardware and power management software to maintain stability at this scale.
Meta has significantly improved the reliability of its Ohio-based supercluster, reducing daily training interruptions by 99% through extensive engineering work. This stability was essential for training the Muse family of foundational models, which includes the Muse Wonder variants recently released to subscribers and API users. The improvements address the inherent fragility of running hundreds of thousands of interconnected processors in lockstep, a challenge that previously caused frequent disruptions to long-running computational jobs.
How it unfolded
- Two years ago, the New Albany, Ohio site operated as a conventional data centre for web traffic.
- The first phase of the gigawatt-scale Prometheus cluster was completed, running hundreds of thousands of Nvidia GPUs.
- Mid-2025 data showed interruptions peaked at nearly 1,000 events per 8,000 GPUs daily during early training runs.
- A year of engineering work focused on signal quality and memory faults, cutting those interruptions by 99%.
Who is affected
System administrators and AI researchers relying on large-scale training infrastructure benefit from these reliability gains. Meta’s engineering team reported that during the training of Muse Wonder, the cluster maintained more than 85% productive pre-training time and over 90% for reinforcement learning. The average time between interruptions increased from about 47,000 GPU hours to above 54,000 hours following firmware fixes. These metrics indicate that large-scale model development is becoming more predictable, reducing the computational waste associated with restarting failed training jobs.
What happens next
Meta plans to expand the Prometheus supercluster to roughly one million distributed GPUs at full scale. The current infrastructure uses two Nvidia platforms: Catalina with GB200 NVL72 racks and Clemente with the newer GB300 NVL72 racks. A third platform, San Miguel, will introduce Nvidia’s Vera Rubin NVL72 racks. To manage the power demands of such a massive compute array, Meta employs software called Dynamo Dimmer to cap GPU power gradually, supported by battery storage at the rack, facility, and grid levels.
The background in plain language
Large AI models require thousands of processors to work simultaneously, creating a system where a single failure can halt the entire operation. To understand this challenge, consider that these systems are like a relay race where every runner must move in perfect sync. If one runner trips, the whole team stops. Engineers must therefore build in redundancy, such as spare components, and implement strict power controls to prevent sudden surges from destabilizing the grid. This approach ensures that the massive computational workload continues smoothly despite minor hardware issues.
What to do and how to stay safe: Meta
- Monitor your own cluster health metrics closely to identify trends in interruption frequency before they impact major projects.
- Verify that your firmware versions are up to date, as vendor patches often address stability issues in high-density GPU environments.
- Review your power management policies to ensure they can handle sudden load changes without triggering protective shutdowns.
- Establish clear protocols for resuming training jobs after failures to minimize data loss and computational waste.
Step-by-step guide: Mobile Device Security for Small Business: Control Without Cost
General security guidance from the Circuit Observer newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
How many GPUs will the Prometheus cluster have at full scale?
Meta states that Prometheus will comprise roughly one million distributed GPUs at full scale.
What was the primary cause of early training interruptions?
The biggest early problem was signal quality on more than 600,000 NVLink connections, followed by memory faults in GB300 racks.
Does Meta use renewable energy for this facility?
Yes, Meta pays the full cost of energy used in New Albany and matches it with new clean and renewable energy.
