Back to Devoured - May 18, 2026
DEVOURED
AWS Outage May 2026: Lessons for Database Disaster Recovery

AWS Outage May 2026: Lessons for Database Disaster Recovery

Data SingleStore
A 20-hour data center overheating event in AWS US-EAST-1 on May 7-8 took down Coinbase, FanDuel, and CME, proving Multi-AZ can't protect single-zone latency-optimized workloads from physical failures.
What: On May 7-8, 2026, cooling failures in AWS availability zone use1-az4 (Northern Virginia) caused a 20-hour outage affecting Coinbase (7 hours offline), FanDuel, and CME Group. The incident began at 23:50 UTC when data center overheating physically damaged EC2 instances and EBS volumes. Coinbase CEO Brian Armstrong admitted their centralized exchange ran in a single zone for latency optimization, and backup systems failed to activate, forcing manual recovery. AWS SLA credits cover roughly 10% of monthly compute but not revenue losses research pegs at $300K-$5M per hour for enterprises.
Why it matters: Reveals the fundamental tension between latency optimization and resilience in cloud architecture: trading platforms sacrifice Multi-AZ redundancy for single-zone deployment to minimize microsecond-level latency, leaving them vulnerable to physical failures that software-based high availability cannot protect against. Cloud SLAs are financial credits, not business continuity insurance.
Takeaway: If you use AWS US-EAST-1, map which services run in single zones versus Multi-AZ, then test your disaster recovery runbook by walking through what happens if that region is offline for six hours starting now.
Deep dive
  • AWS US-EAST-1 availability zone use1-az4 suffered cooling system failures starting 23:50 UTC May 7, 2026, physically damaging EC2 instances and EBS volumes on affected racks over a 20-hour period until cooling was restored May 8 at 13:50 PT
  • Coinbase experienced a 7-hour total outage of its centralized exchange, trading platform, Prime, international venue, and derivatives exchange—occurring during an already difficult week following 14% workforce reduction (700 employees) and Q1 earnings showing $394M net loss with 31% year-over-year revenue decline
  • CEO Brian Armstrong acknowledged most Coinbase systems tolerate single-zone failures, but the exchange deliberately runs in a single zone to minimize latency for traders and support customer co-location; backup systems failed to activate as designed, forcing manual disaster recovery execution
  • FanDuel went offline at 21:00 ET during Game 2 of Lakers versus Thunder playoff semifinal, preventing users from placing or cashing out live bets during peak betting window—worst possible timing for a sportsbook
  • CME Group reported login and latency issues on CME Direct institutional trading platform, creating regulatory and risk management concerns beyond just technical downtime
  • Root cause was physical infrastructure failure (overheating), not software, meaning automated failover orchestration cannot reroute around the damage—illustrating limits of software-based high availability
  • AWS SLA provides roughly 10% credit on monthly compute spend for affected instances, but research shows 90% of mid-to-large enterprises lose $300K or more per hour during outages, with 41% losing $1M-$5M per hour; financial services losses run higher
  • Multi-AZ high availability and cross-region disaster recovery solve different problems: Multi-AZ protects against zone-level failures but offers no protection for region-wide events or for workloads that must run single-zone for latency or performance reasons
  • Cross-region disaster recovery requires asynchronous replication (accepting eventual consistency), choice between idle versus hot standby compute (cost versus RTO trade-off), and full topology replication including not just data but users, permissions, firewall rules, and configuration
  • Article recommends three immediate actions: (1) map every application tier to its actual region and zone, identifying assumed-multi-region services that are not; (2) stress-test RTO by walking through actual failover procedures and identifying gaps in runbooks; (3) calculate cost of one hour of downtime versus cost of disaster recovery replication to make informed insurance decision
  • Disaster recovery plan must extend beyond database to include application reconnection logic, networking, private endpoints, DNS or connection management, operational ownership, validation procedures, and regular testing—database replication alone is insufficient
  • Lesson is not that AWS is unreliable but that being on a reliable provider is not the same as being resilient; resilience is owned through conscious architectural choices about data location, replication frequency, and cross-region application deployment capability
Decoder
  • RPO (Recovery Point Objective): Maximum acceptable data loss measured in time (e.g., 10-minute RPO means up to 10 minutes of recent data could be lost during a disaster)
  • RTO (Recovery Time Objective): Maximum acceptable downtime duration—how long it takes to restore service after a failure
  • Hot Standby: Disaster recovery mode where backup compute resources are already running and ready for immediate failover, versus cold standby where resources must be provisioned first (faster RTO but higher cost)
  • Co-location: In trading context, physically placing servers in the same data center as the exchange to minimize network latency to microseconds for high-frequency trading
Original article
Let me extract and clean the article directly:

At 23:50 UTC on Thursday, 7 May 2026, a room in an Amazon data centre in Northern Virginia overheated. Multiple cooling units in availability zone use1-az4 failed. Within minutes, EC2 instances and EBS volumes on the affected racks were losing power. Within the hour, traders trying to close positions on Coinbase, bettors trying to cash out during Game 2 of the Lakers and Thunder Western Conference semifinal on FanDuel, and institutional users on CME Direct were staring at error screens.

For most of those users, the next several hours were the operational definition of helplessness. There was no failover button. There was no secondary region to switch to. They could only refresh.

If you run a mission-critical workload on a single AWS region, this post is for you. SingleStore Smart DR provides cross-region replication with a target RPO of up to 10 minutes and no idle compute cost in the secondary region until you fail over.

Key takeaways

  • A single-zone thermal event in AWS US-EAST-1 caused multi-hour outages at Coinbase, FanDuel and CME Group on 7–8 May 2026.

  • Coinbase was offline for approximately seven hours. AWS recovery extended into the following afternoon.

  • The standard AWS service credit covers about 10% of monthly compute spend on impacted instances. It does not cover lost revenue, regulatory exposure or customer trust.

  • Multi-AZ high availability did not save Coinbase, because their latency-sensitive matching engine ran in a single zone by design. Multi-region disaster recovery is a different problem.

What happened on 7–8 May 2026

The thermal event began at approximately 17:25 PDT on Thursday, 7 May. Cooling capacity in a single data centre hall dropped, triggering a power loss that physically damaged EC2 instances and EBS volumes on affected racks. AWS shifted traffic away from the affected zone, but recovery depended on physically restoring cooling capacity before damaged hardware could safely return to service. Cooling was stabilised at pre-event levels at 13:50 PT on Friday, 8 May, more than 20 hours after the incident began. Most affected instances and volumes were restored at that point.

The root cause matters. This was not a software bug or a misconfiguration. It was a building that got too hot. Software orchestration cannot automatically reroute around physical hardware damage. That is why the rest of this post is about cross-region disaster recovery rather than high availability.

Who was impacted

Coinbase was offline for approximately seven hours. Trading, exchange access, balance updates, Prime, the international venue and the derivatives exchange all went dark. The disruption arrived at the end of an already difficult week for the company: on Monday, Coinbase had announced a 14% workforce reduction of around 700 employees. On Thursday afternoon, hours before the outage began, the company reported a Q1 net loss of $394 million and a 31% year-on-year revenue decline.

CEO Brian Armstrong was unusually direct about what went wrong. In a public statement on X, he wrote that the outage was "never acceptable", and acknowledged that while most Coinbase systems are designed to tolerate the failure of a single AWS availability zone, the centralised exchange did not. Coinbase's Head of Platform Rob Witoff later confirmed that the primary exchange systems run in a single zone to minimise latency, and that backup systems "did not work as expected during the incident, extending the outage and forcing engineers to manually execute disaster recovery procedures".

FanDuel went offline at approximately 21:00 ET, just as Game 2 of the Lakers and Thunder semifinal tipped off. This is, by some margin, the worst possible window for a US sportsbook to fail. Live bets could not be cashed out. Users posted screenshots demanding refunds and bonuses, some threatening legal action. FanDuel acknowledged "technical difficulties prohibiting users from accessing our platform" before confirming the AWS link about two hours later.

CME Group reported login and latency issues on CME Direct, its institutional trading platform. For a regulated derivatives exchange, even short outages create a regulatory and operational risk management question, not just a technical one.

These are the three companies whose outages made the news. The actual downstream was much wider. Any business with production workloads in US-EAST-1 that depended on EC2 instances or EBS volumes in availability zone use1-az4 may have experienced impairments. That includes some SingleStore Helios customers running on AWS. (Helios is our fully managed cloud database service.) Many had architectures that absorbed the disruption cleanly. Others felt it directly. The companies in the press are visible because public markets oblige them to file statements. The teams who quietly spent the night on a bridge call do not show up in headlines, but the impact on their business is no less real.

The hidden cost: SLA credits versus reality

AWS will compensate affected customers under the standard EC2 service level agreement. The credit is typically 10% of monthly compute spend on impacted instances. There is no compensation for lost revenue, lost customer trust or regulatory exposure. Independent research consistently puts the real cost much higher: the 2024 ITIC Hourly Cost of Downtime survey found that 90% of mid-size and large enterprises lose more than $300,000 per hour during an outage, and 41% lose between $1 million and $5 million per hour. In finance and trading the losses run higher still.

The cloud SLA is not your business continuity plan. It is a small refund.

High availability and disaster recovery solve different problems

The temptation, after an event like this, is to conclude that multi-AZ is the answer. Two things complicate that conclusion.

First, Coinbase already had multi-AZ for most workloads. Their statement made this explicit: "Coinbase systems are designed to be resilient to a single zone outage. In this case, we observed failures impacting multiple AWS zones." The exchange itself ran in a single zone by design, optimised for latency and customer co-location. If you run a real-time operational workload that is genuinely latency-sensitive, you have probably made similar trade-offs.

Second, even where multi-AZ is in place, it does nothing for a region-level event. AWS treats availability zones as the failure domain for high availability. Disaster recovery is about regions, and regions are independent on purpose. A thermal event in US-EAST-1 will not move your data to US-WEST-2 unless you have explicitly arranged for it to do so.

The distinction matters. High availability protects you from a bad day in one rack or one zone. Disaster recovery protects you from a bad day in one region. Most production workloads need both.

Where Smart DR stops and your DR plan begins

We are being honest about the constraints. Smart DR protects the database recovery path. It does not remove the need for an end-to-end application disaster recovery plan.

You still need to think through application behaviour during failover, networking, private endpoints, DNS or connection management, operational ownership and runbooks. Who makes the failover decision. How the application reconnects. How you validate correctness in the target region. How you test the process regularly. These are questions Smart DR does not answer for you, and we would rather say so than pretend otherwise.

Replication is asynchronous, which is the right trade-off for an operational database but means it is not a synchronous multi-region active-active topology. The 10-minute RPO is a ceiling, not a guarantee; actual RPO depends on workload characteristics and replication lag at the moment of failure.

Three things to do this week

  1. Map your region affinities. For each tier of your application (presentation, API, application logic, primary database, cache, message bus, object storage, analytics), write down the AWS region and, where applicable, the availability zone. You will probably find at least one tier you assumed was multi-region and is not.

  2. Stress-test your assumed RTO. Sit with your engineering lead and walk through what it would actually take to restore service if US-EAST-1 went offline for six hours starting now. Be specific. Who runs the runbook. Where is the runbook. When was it last exercised. What is the connection string for the secondary region. What does DNS look like.

  3. Decide what you are buying. Cross-region disaster recovery is an insurance product. The premium is database replication cost. The payout is everything you do not lose when the chillers fail. Set a number for what an hour of downtime costs your business. Compare it to the replication cost. The decision usually becomes obvious in one direction or the other.

Closing

The internet still runs in buildings, and buildings can overheat. That is the part of the story no architecture diagram makes visible. The teams at Coinbase, FanDuel, CME and the other affected platforms responded well to a hard situation, and we have a great deal of sympathy for the engineers who spent Thursday night and Friday morning on a recovery call.

The lesson we take from May 2026 is not that AWS is unreliable. AWS is, on the whole, extremely reliable, and the engineering response during this event was professional. The lesson is that being on a reliable provider is not the same as being resilient. Resilience is something you own. It lives in the choices you make about where your data is, how often it replicates, and how quickly you can run your application in a different region. Smart DR is one option for that. There are others. The only choice that is not available is to not make a choice.

Explore all 67 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi