DEVOURED
Rapidly scaling online storage to serve over 1 billion ChatGPT users
OpenAI manages over 500 petabytes of data for a billion users by evolving its Azure Cosmos DB configuration from shared containers to feature-specific partitioning.
What: OpenAI's storage platform, Habitat, successfully mitigated noisy-neighbor issues by implementing request priorities, hot-key throttling, and a serverless mode. They transitioned from a single shared container model back to a per-feature container approach to ensure better scaling for high-traffic launches.
Why it matters: At extreme scales, default cloud database configurations often hit multi-tenancy limits, requiring custom application-level traffic management and precise partitioning to maintain uptime.
Decoder
- Hot key: A specific database partition or document that receives a disproportionate amount of read or write traffic, leading to performance bottlenecks.
Original article
OpenAI's storage platform, Habitat, serves over 500 petabytes to over 1 billion users. Its Azure Cosmos DB setup went from one shared container to one per feature, then back to shared after teams forgot to pre-scale for launches. With hot key throttling, request priorities, and a new serverless mode, it has had no noisy-neighbour incidents since.
Explore all 69 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi