Beyond Kubernetes at Modal: How to Scale 1 Million Concurrent Sandboxes in Seconds

In a recent article, Colin Weld and Connor Adams, staff engineers at Modal, describe how they rebuilt their sandbox infrastructure from the ground up to support millions of concurrent sandboxes and tens of thousands of sandbox creations per second.

According to Weld and Adams, traditional container orchestration systems such as Kubernetes struggle to operate at this scale because they rely heavily on centralized coordination and strongly consistent state.

Running 1 million sandboxes pushes the limits of any container platform, both because of the sheer number of containers, but also because running this many sandboxes requires many tens of thousands of compute nodes. There will be many operations which are either O(containers), O(nodes), or both, which will cause traditional container platforms to hit scaling limits.

In Kubernetes' case, they explain, the load on both the scheduling algorithm and the central durable store (etcd) grows with the number of nodes and pods. Additionally, both pods and nodes write to etcd multiple times, "which can create serious issues under high pod creation rates or high pod churn, and etcd is not natively shardable within a keyspace". They also note that overcoming these limitations is feasible, but requires "serious work", including rewriting or replacing etcd and parallelizing the scheduling algorithm.

To optimize for scale, we decided that everything taking O(sandboxes) or O(nodes) load must be horizontally scalable by default, the sandbox creation path should be as simple as possible, and everything else should be secondary.

The fundamental change Modal's engineers made to their platform was to stop coordinating globally and make scheduling look more like load balancing. Instead of relying on a central datastore as the source of truth, each worker became its own source of truth. Likewise, rather than using a single, serialized scheduler, they deployed a fleet of scheduling servers operating in parallel, allowing the scheduling layer to scale horizontally.

Once a scheduling server decides which worker to create a sandbox on, it contacts the worker directly via RPC to request that a sandbox is created. Workers accept the scheduling request if they have free resources, or otherwise reject it.

The resulting architecture has only one bottleneck, they say: all workers publish their state as a single Redis stream. However, "load testing has suggested that this remains viable until well over 100,000 workers". In their benchmark, they created 1 million sandboxes in under a minute, with median startup-to-code time under 0.5 seconds.

Commenting on the announcement on LinkedIn, Hopsworks CEO Jim Dowling noted that "new technical problems arise at every order of magnitude increase in scale", suggesting the team had to iterate on their design multiple times to achieve a reliable 50,000 sandbox creations per second. More substantively, AWS principal AI engineer Alex Jones observed that a key part of Modal's achievement was not trying to extend Kubernetes but rather "walking around the whole thing" after understanding its limitations. Jones considers this "the first credible signal that Kubernetes isn't adapting fast enough to what GenAI infrastructure actually needs" and argues that:

We're heading for a decoupling of coordination from execution. The execution plane wants what Modal built; isolation boundaries that appear in milliseconds. The coordination plane (where multi-agent workflows need shared memory and overlapping security boundaries) still wants what Kubernetes-shaped systems are good at.

Modal is a serverless compute platform built specifically for AI workloads, providing programmable access to CPUs, GPUs, containers, inference, training, batch jobs, and isolated sandboxes. It is not alone in seeking to "rebuild the cloud" around highly scalable infrastructure and sub-10-millisecond cold starts. Other projects pursuing similar goals include Unikraft, Google Substrate, and Overdrive.