DEVOURED
How to achieve truly serverless GPUs
Modal cut GPU inference server boot time 40x by snapshotting CUDA contexts directly into memory, taking replica spin-up from 2000 seconds to 50 seconds across 15 million production deployments.
What: Modal engineers Charles Frye, Jonathan Belotti, Erik Bernhardsson, and Akshat Bubna built a system combining GPU buffers, lazy filesystem loading via ImageFS, CPU checkpoint/restore with gVisor runsc, and CUDA memory snapshotting to reduce inference server cold starts from tens of minutes to tens of seconds. The system restored 15 million GPU snapshots and 35 million CPU snapshots across February-April 2026, with customers like Reducto seeing 6x improvements (70s to 12s cold starts).
Why it matters: This reveals how serverless economics for AI inference depends on solving GPU Allocation Utilization, not just Model FLOP/s — spiky user-driven demand means fixed allocations commonly achieve only 10-20% utilization, making fast replica scaling the critical bottleneck for profitable inference services at scale.
Takeaway: If you're running inference servers at scale, Modal offers $30/month free credits at modal.com. For DIY implementation, their CUDA checkpoint/restore approach uses recent Nvidia driver features for device memory snapshotting combined with gVisor runsc for host-side state.
Deep dive
- Modal reduced GPU inference server replica spin-up from 2000+ seconds to ~50 seconds through four architectural optimizations spanning the entire stack
- Cloud buffers: Maintain idle, health-checked GPUs ready for immediate scheduling, removing tens of minutes of instance allocation latency from the hot path; GPU failure rates are significant enough to require active boot checks and weekly dcgmi diagnostics
- Custom filesystem (ImageFS): Built with libfuse to lazily load container images from a content-addressed, multi-tier cache (page cache → SSD → AZ cache → CDN → blob storage); most container files are never read, so lazy loading with metadata-first start cuts boot from minutes to ~100ms
- Disabled gzip compression (single-threaded DEFLATE bottlenecks at ~100 MB/s), tuned read_ahead_kb to 32 MB, achieving multi-GB/s throughput from cache
- CPU checkpoint/restore: Using gVisor's runsc runtime (emulates Linux kernel in userspace), they serialize container state to disk and restore 10x faster than cold start; snapshots are host-sensitive (e.g., AWS g6.12xlarge lacks pclmulqdq instruction), requiring multiple snapshots per deployment
- GPU memory snapshotting: Recent Nvidia drivers checkpoint device memory into host memory, which then gets checkpointed to disk, enabling full device state restoration including CUDA graphs and Torch compiler artifacts
- Typical speedups are 4-10x for GPU workloads — vLLM boot latency dropped from 95.7s to 13.8s mean, SGLang from 83.7s to 17.5s for 1 GiB models (Qwen 3 0.6B) across tens of thousands of cold starts
- GPU snapshots require adjustments: weight offloading before checkpoint, KV cache recreation, and currently limited to single-GPU (multi-GPU nccl programs deadlock on pause)
- Weight loading bottleneck: billions to trillions of bytes at few GB/s = seconds to hundreds of seconds; could be >3x faster with AZ weight server, >10x with RDMA over RoCE/InfiniBand
- Production usage Feb-April 2026: ~35M CPU snapshot replicas (>5M GPU-hours), ~15M CPU+GPU snapshot replicas (>2M GPU-hours) across hundreds of organizations
- Reducto case study: Document processing platform (known for indexing Jeffrey Epstein's JMail correspondence) scales to thousands of GPUs on-demand for enterprise jobs with tens-of-minutes deadlines
- System aggregates capacity from multiple cloud providers for low cost and high availability, requiring snapshot compatibility across heterogeneous instance types
- GPU Allocation Utilization (GPU-seconds running code ÷ GPU-seconds paid for) is distinct from nvidia-smi metrics and is the critical bottleneck for inference economics — industry average is <70% at peak, often 10-20% in practice
- Spiky demand from external user behavior creates high peak-to-average ratios that wreck economics with fixed allocations but are profitable with fast auto-scaling
Decoder
- GPU Allocation Utilization: GPU-seconds spent running application code divided by GPU-seconds paid for — distinct from nvidia-smi "utilization" (fraction of time kernel code runs) and Model FLOP/s Utilization (algorithmic operations divided by arithmetic bandwidth)
- CRIU (Checkpoint/Restore In Userspace): Linux transparent checkpoint/restore system that serializes running process state (heap, threads, file descriptors) to recreate processes from storage faster than re-executing
- gVisor/runsc: Google container runtime that emulates Linux kernel in userspace for security isolation, enabling checkpoint/restore without host kernel cooperation via nvproxy for GPU driver communication
- libfuse: Library for writing Linux filesystems in userspace with kernel intermediation, simpler than kernel modules but with double context switches
- CUDA graphs: GPU execution graphs made up of pointers to tensors and kernels with no native serialization, expensive to recreate (tens of seconds to minutes) during inference engine startup
- Content-addressed cache: Cache indexed by file content hash rather than path, enables reuse when same files appear in different container layers or images
- vLLM/SGLang: Popular LLM inference servers that perform model loading, CUDA graph capture, and Torch compilation during startup
Original article
Full article content is not available for inline reading.
Explore all 50 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi