DEVOURED
Scaling Git at OpenAI
OpenAI built a custom Git backend called Pando to handle its massive monorepo, achieving a 5x improvement in push latency.
What: As OpenAI's codebase expanded rapidly over the last year, the company developed Pando to optimize Git operations. Pando handles nearly a petabyte of data daily and integrates seamlessly into existing workflows without requiring developers to change their standard Git tooling.
Why it matters: Standard Git server implementations often struggle to maintain performance at the scale of massive, fast-growing AI engineering organizations, necessitating custom-built storage backends.
Deep dive
- Scale requirements: OpenAI's monorepo necessitated a system capable of managing massive, frequent concurrent branch pushes.
- Seamless integration: The solution was built as a backend replacement for standard Git protocol, ensuring no disruption to developer experience.
- Performance metrics: The team observed a 5x improvement in median push latency during production deployment.
- Throughput: The system now manages nearly a petabyte of Git data per day, demonstrating the heavy reliance of modern AI labs on high-velocity source control.
Decoder
- Monorepo: A software development strategy where code for many projects is stored in a single repository rather than separated into multiple individual repositories.
Original article
OpenAI's monorepo's size and demands increased at a staggering pace over the last year, necessitating a revamp of the AI lab's source control system. Its technical staff built Pando, a new backend for Git repositories optimized for concurrent pushes to many branches and to maximize read throughput. The team implemented the backend without interrupting any engineering work or changing how the team uses Git. Pando cut median push latency in production by nearly 5x, and it now delivers nearly a petabyte of Git data per day.
Explore all 41 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi