Patching at Fleet Scale, Twice: How DigitalOcean Closed Januscape and the AMD Safe RET Issue Without Customer Impact

In early July, security researcher Hyunwoo Kim discovered Januscape (CVE-2026-53359), a flaw in KVM’s handling of nested virtualization that could allow a malicious guest to escape into the host hypervisor. It was disclosed publicly on July 6 via the Linux oss-security mailing list. For a cloud provider, a guest-to-host escape is the most serious class of vulnerability there is: the hypervisor is the boundary that keeps each customer’s workloads isolated from each other, and from our infrastructure itself.

We responded, patched the entire fleet in eight days with zero confirmed customer-facing impact, and drafted a post about how we did it. Then, before we could hit publish, it happened again. In late July we learned of a second and unrelated vulnerability affecting our entire AMD hypervisor fleet, that could not be livepatched. Roughly 1,600 hypervisors needed a kernel update and a reboot.

So now this story is about two responses, three weeks apart. The first built the muscle. The second proved it was repeatable, at a larger scale, and on a harder constraint. Here’s how both played out, and why two of the most serious vulnerability classes in cloud computing ended up feeling like just another couple of weeks for us.

Act one: Januscape

The fast path: fleet-wide livepatching

Our response kicked off the same night the vulnerability was disclosed. When public exploit code surfaced late in the evening of July 6, the Kernel Engineering team was paged and dug in immediately. Engineers reproduced the exploit in an isolated environment, confirmed which kernel lines were affected, and built the first working livepatch before 1 AM, roughly 45 minutes after answering the page. Livepatching lets us fix a running kernel in place, with no reboot, no migration, and no observed disruption to the customer. A few hours later, patches for the kernel versions (6.1 and 6.12) that run the majority of our hypervisor fleet were ready to ship. For the remainder, we had to follow a different approach; more on that below.

The rollout followed the discipline we apply to any change that touches every host we operate: patches were built, validated in our staging environment, and promoted through a gradual production ramp with health checks and commitment to pause at any sign of regression. Within roughly 48 hours of the patches being built, thousands of hypervisors were protected and the 6.x fleet reached full coverage shortly after.

For most of the fleet, the story was that the vulnerability was quietly closed before most customers had finished reading the CVE announcement. The exception is where the harder work began.

The long tail: hosts with no upstream fix

Every large fleet has a long tail, and how you handle the long tail under pressure says more about your engineering culture than how you handle the easy 95%.

A few hundred dedicated hypervisors were running a 5.10-series kernel that the upstream fix didn’t cover. As there was no upstream patch to cherry-pick for that kernel line, the Kernel Engineering team wrote an entirely new livepatch implementation from scratch. It passed initial testing, but as it didn’t correspond to an upstream-reviewed fix, there was a level of risk above what we normally accept for a change to a running production kernel.

Faced with a working-but-unproven patch on one side and a slower-but-durable path on the other, our team made the call to evacuate and upgrade the affected hosts entirely. This meant live-migrating customer workloads off each machine, repaving with a new OS and modern kernel, and returning it to service fully patched. This is the heaviest maintenance operation we can perform on a host: a full repave with each machine out of service for the duration of the rebuild. The custom livepatch was held in reserve and hardened in parallel rather than rushed to production.

Our customers choose our dedicated fleet because they have the highest expectations for isolation and availability, and the least tolerance for unanticipated downtime. Shipping an unproven kernel patch to the customer segment least able to absorb a destabilizing event would have traded our security risk for their availability risk. The solution was clear: take the path that is slower for us, addresses the security risk, and is designed to be invisible to customers.

Turning a kernel problem into a capacity problem

Evacuating hosts means live migrating running workloads, and in a hot, heavily utilized dedicated fleet, spare capacity is scarce by design. At the maintenance pace the fleet’s capacity could support, full remediation was projected to take an unacceptable amount of time at three and a half weeks. To move faster, without sacrificing any quality of work, the Security, Capacity, Product, and Fleet Management teams got creative:

• Temporary refleeting: Servers were moved from an adjacent premium pool into the constrained fleet to serve as evacuation targets, with a plan to return them after the burn-down.
• Reclaiming stranded capacity: Teams audited the region for hosts sitting idle (machines held for completed projects, stale reservations, hosts stranded in standby after prior maintenance), validated their health, and released them back into production as landing space.
• Raising maintenance concurrency: With real evacuation targets in place, the number of hosts that could be safely drained in parallel nearly tripled.

As a result, the projected remediation time dropped to less than 1.5 weeks, and the actual burn-down ran faster still. All of it moved through our standard live-migration machinery, with no customer-visible interruption observed and capacity buffers preserved throughout.

In order to keep the accelerated remediation running smoothly, an incident was formally declared. This creates a single coordination point, clear ownership, structured status reporting, and an auditable record for every capacity and fleet decision made under pressure. Additionally, our CloudOps team watched the evacuation pipeline 24/7. Automated fleet maintenance at elevated concurrency inevitably hits snags, and every stalled host is a host still exposed. By clearing workflow failures around the clock, the team kept the pipeline saturated and the burn-down rate steady.

While remediation ran, our Security and Virtualization teams tested the public proof-of-concept exploit in a controlled environment, characterized the signals an exploitation attempt would leave behind, and built fleet-wide detection and alerting around those indicators. Detection did not replace patching. It meant that even on the not-yet-remediated tail, an exploitation attempt would not have gone unnoticed.

Finishing the job: proving out the 5.10 livepatch

Choosing evacuation as the primary path did not mean shelving the livepatch. From the moment the incident was declared, the Kernel Engineering team kept hardening it in parallel: extending test coverage, exercising it against the proof-of-concept exploit, and letting it soak in staging while the evacuation burn-down shrank the exposed fleet hour by hour. By July 13, the 5.10 patch earned the confidence it initially lacked and was promoted through the same staged discipline as the 6.x patches. On July 14, the rollout reached 100%. Eight days after disclosure, the entire fleet was protected.

Act two: the AMD Safe RET issue

A different starting gun

Vulnerabilities do not always announce themselves on a public mailing list. The security industry runs on coordinated vulnerability disclosure (CVD): when a researcher reports a flaw to a vendor, the vendor privately notifies affected parties under strict confidentiality before the public announcement, so that fixes can be built, tested, and deployed before attackers learn the details. These pre-disclosure programs are one of the quiet mechanisms that keep the internet safe, and participating in them is part of our job as a cloud provider.

In late July, we were notified of a new vulnerability affecting AMD “Zen” architecture processors ahead of its public announcement. Researchers had demonstrated that a precisely timed interrupt could disrupt “Safe RET,” the default Linux mitigation for Speculative Return Stack Overflow, potentially weakening that protection and allowing information disclosure across privilege boundaries. The details are now public in AMD’s security bulletin and the researchers’ TONTOU paper presented at this year’s Black Hat conference. This was not an AMD-specific weakness, but rather an instance of a problem the whole industry shares. The same research detailed similar interrupt-timing attacks against Intel processors, and mitigations for speculative-execution issues across all vendors remain an active area of hardening. AMD’s coordinated disclosure gave us the time to fix this before it was public - exactly how the process is supposed to work. Our response centered on our AMD fleet simply because Safe RET is the mitigation deployed on those hosts. For us, the risk read the same way Januscape did: a guest-to-host information disclosure concern on shared infrastructure, in this case covering our entire AMD hypervisor fleet of roughly 1,600 hypervisors across twelve regions.

This time, there was no fast path. The fix could not be livepatched; the affected code had been inlined by the compiler, leaving no function boundary for a livepatch to replace. Every one of those ~1,600 hosts needed a new kernel and a reboot.

The encore is easier when you just rehearsed

Three weeks earlier, the Januscape long tail forced us to evacuate and fully rebuild hosts, and undergo the heaviest per-host operation in our maintenance toolbox. In hindsight, that effort was a full-dress rehearsal. The AMD response needed the same choreography (evacuate live workloads, take the host down, bring it back patched), but with a much lighter per-host step: no rebuild, just a reboot into a kernel that was already staged on the machine. Pre-staging the kernel package on every affected host ahead of its maintenance window meant each host’s downtime was minutes, not the hours a repave would require, and the same automation could push far more hosts through the pipeline per day.

Everything we had exercised came straight back into relevance here:

• The capacity playbook: The same unlock formula from Januscape was applied across every affected region, with servers refleeted on short notice to regions that needed extra landing space.
• The automation: A single workflow handled each host end to end: evacuate workloads via live migration, reboot into the new kernel, run health verification, and return the host to service. When a couple of constrained fleets hit failed migrations, the capacity and fleet teams refleeted servers within hours to unblock them.
• The operational rigor: A formal incident was declared on day one, and CloudOps monitored the pipeline 24/7 across shifts, clearing stalled migrations and hardware snags so the burn-down never sat idle.
• The prioritization: Fleets serving customers with the highest availability expectations were patched first, the same customer-centric ordering logic we applied to the Januscape long tail.

The kernel team also banked a dividend from the first act: the new kernel build folded in the upstream Januscape fix alongside the AMD patch, closing multiple items with a single reboot and eliminating the need to re-livepatch the new kernel afterward.

What two responses in one month taught us

Individually, each response is a story about speed and safety. Together, they are a story about repeatability.

The Januscape response was improvised in the best sense: teams inventing capacity where none existed, weighing an unproven patch against a slow rebuild, and choosing the path that protected customers. The AMD response, three weeks later, was that same system running as designed. The capacity formula was applied, not invented. The automation was tuned, not built. The incident structure, the 24/7 monitoring, the customer-first prioritization: all of it was muscle memory. A harder constraint (mandatory reboots, four times as many hosts) produced a smoother operation, because the per-host work was lighter and the machinery was already warm.

No single fix or team got us here. Kernel engineers built and qualified patches at remarkable speed and had the judgment to know when not to ship one. Capacity planners found headroom in fleets that had none to spare, twice. Operations engineers watched pipelines around the clock for the better part of a month. Security engineers built detection for both vulnerabilities so that exposure windows were monitored, not merely tolerated. Support teams were ready with answers to customers before they were asked, both times.

Speed and safety are usually framed as a tradeoff. Two fleet-wide remediations in a single month, with zero confirmed customer-facing impact between them, is our evidence that with enough coordination they don’t have to be.

OK. What’s next?

A fleet-wide remediation in eight days, followed by a 1,600-host reboot campaign completed ahead of public disclosure, is a result we’re proud of. It is also a pace we expect to beat. The window between vulnerability disclosure and working exploit code keeps shrinking, and AI-assisted vulnerability discovery and exploit development are compressing it further. The Zero Day Clock tracks time-to-exploit across tens of thousands of CVEs, and the trend is blunt: the gap between disclosure and exploitation is collapsing toward zero, if not already past. Our planning assumption follows directly. Next time, we expect less warning, and we are investing in the ability to roll the entire fleet at a moment’s notice.

Every incident at DigitalOcean closes with a post-incident review, and both of these are no exception. We are examining every stage of both responses for time to claw back: how we detect and alert faster, decide faster, and start moving the fleet sooner.

Eight days was fast. Six days, at four times the scale, was faster. The next one needs to be faster still, and we intend to be ready.