DEVOURED
Git at Any Scale
Cursor's research highlights the shift from Git's packfile-based bottleneck to a write-ahead log architecture for massive scalability.
What: Vicent Martí details why traditional Git hosting (Spokes) struggles with 3PC consensus at scale, proposing 'Continuity'—a WAL-based system using S3 as a source of truth.
Why it matters: As AI agents generate more code and PRs, version control systems are hitting horizontal scaling limits that necessitate rethinking standard Git repository hosting.
Deep dive
- Git’s packfile-based design creates random read patterns that struggle on distributed filesystems.
- Spokes (GitHub-style) uses 3PC to manage consensus, but this limits throughput as replicas increase.
- Continuity treats repositories as stateless caches, using a Write-Ahead Log (WAL) in S3 as the source of truth.
- Reads are consistent because replicas verify against the S3 ETag (metadata-only call).
- Compaction is moved to the primary, which then pushes compacted packs to S3, reducing CPU load on replicas.
- The system supports linear scaling of read performance by adding more replicas.
Decoder
- Packfile: Git's binary serialization format for storing objects (blobs, trees, commits).
- 3PC: Three-phase commit, a consensus protocol ensuring all replicas acknowledge a transaction.
- WAL: Write-Ahead Log, a technique that records transactions in a log before applying them to the main data store to ensure durability.
- Linearizable: A consistency model where operations appear instantaneous and atomic to all observers.
- Rendezvous Hashing: A algorithm to map items to nodes in a distributed system, minimizing remapping when nodes join or leave.
Original article
Full article content is not available for inline reading.
Explore all 56 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi