DEVOURED
AI load breaks GitHub – why not other vendors?
GitHub's recent spate of data integrity incidents and outages, including a 85% uptime, is attributed by CTO Vlad Fedorov to an unexpected surge in AI agent-fueled load, which GitHub, unlike competitors, seemingly failed to adequately anticipate.
What: GitHub has suffered significant reliability issues, including a data integrity incident on April 23rd impacting 2,092 pull requests, and multiple outages reducing uptime to 85% in the last 90 days, leading Mitchell Hashimoto to declare it "unfit for professional work." GitHub CTO Vlad Fedorov blamed an unanticipated ~3.5x load increase from AI agents over two years, initially planning for 10x capacity by October 2025 but now targeting 30x.
Why it matters: This exposes a significant disparity in how major tech companies anticipated and prepared for the infrastructure demands of AI, suggesting that GitHub's accumulated tech and organizational debt, combined with its ongoing migration to Azure, left it uniquely vulnerable compared to more agile competitors like Vercel or better-prepared giants like Google.
Takeaway: Developers heavily reliant on GitHub for mission-critical work might consider evaluating alternatives like GitLab, Bitbucket, or self-hosted solutions like Forgejo if GitHub's reliability issues persist.
Deep dive
- GitHub experienced a critical data integrity incident on April 23rd where squash-merged PRs lost commits, affecting 2,092 pull requests.
- The platform has seen widespread outages, including missing PRs and issues due to an Elasticsearch overload, leading to an estimated 85.51% uptime over the last 90 days.
- Mitchell Hashimoto, HashiCorp founder, publicly stated he is moving off GitHub due to its unreliability, calling it "unfit for professional work."
- GitHub CTO Vlad Fedorov attributed the outages to a ~3.5x increase in load over two years, largely driven by AI agents, which compounded issues with GitHub's 18 years of tech debt and an ongoing migration to Azure.
- GitHub initially planned for a 10x capacity increase by October 2025 but has since adjusted this to 30x due to the unexpected load.
- Competitors like GitLab and Bitbucket, and other infrastructure providers like Vercel and Linear, do not report similar widespread reliability issues despite experiencing AI-driven growth.
- The article suggests GitHub's engineering organization did not anticipate the scale of AI load as effectively as some other major tech companies, such as Google, which prepared for a 10x increase in code generation from AI tools.
Decoder
- Squash merge: A Git operation that combines all commits from a feature branch into a single new commit on the main branch, simplifying commit history.
- Elasticsearch: A distributed, RESTful search and analytics engine often used for full-text search, structured search, analytics, and complex aggregations on large datasets.
- Forgejo: An open-source, self-hostable Git service and code forge, often seen as an alternative to platforms like GitHub or GitLab.
Original article
The fact that Microsoft's competitors seem to be keeping up with increased load due to AI suggests that the company has not been responding to its growth like a world-class engineering organization.
Explore all 68 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi