Details of incident.io's AI spend
Incident.io’s deep dive into their AI spend reveals that agent-assisted development significantly boosts output but creates complex new reliability and cost management challenges.
Deep dive
- Incident.io introduced the 'effort point' metric to standardize the complexity of PRs regardless of whether they were human-written or agent-generated.
- AI agents now contribute to nearly 80% of production-bound PRs at the company.
- Headcount for product engineering grew 2.7x, but release frequency grew 4x.
- Claude 5.5 adoption reduced AI spend per engineer from $4,500/month to $1,600/month.
- Tenured engineers see a 70% increase in PR velocity with AI, while new engineers may suffer performance degradation if they start using AI too early.
- Most post-AI incidents are now caused by lack of context about the production environment rather than flawed code logic.
- Infrastructure teams exhibit significantly higher token usage per 'effort point' than other teams due to exploratory tasks like benchmarking and production environment analysis.
Decoder
- Effort point: A custom metric used to quantify the complexity of a pull request by categorizing it from XS to XL, allowing for comparison between human and AI-generated code.
- Worktree: A Git feature allowing a single repository to have multiple branches checked out simultaneously, enabling developers or agents to work on different parts of the code without context switching.
- Forward-deployed engineer (FDE): Engineers who work closely with specific customers or internal teams to build custom integrations or tools rather than just working on the core product.
Original article
incident.io is a software reliability platform founded in 2021 that is obsessed with measuring everything. In that spirit, the company has shared details about AI spending, changes to PR frequency, and the ROI at the company from using AI.
They have come up with the concept of the “effort point” as a way to analyse AI spend. One effort point is a medium-sized pull request written by a human or generated by an agent. The measurement is far from perfect, but it is one step more granular than looking at lines of code, and lets them compare the complexity of AI-generated PRs with those written by human devs in the past. Here’s what has changed in the last two years:
Pull request stats
AI agents started to generate production code in June 2025, which was around the same time as the first agent-generated got pushed at incident.io. The company has used AI-assisted IDEs since the beginning, such as GitHub Copilot and Cursor.
In June this year, a few things happened in quick succession, the incident.io team told me:
- Claude Agents launched, making it easier to coordinate several agents running at once
- Worktree tooling improved a lot, making it easier for agents to do parallel work on the same codebase with worktrees
- Also around this time, more engineers realized that they should be running multiple agent sessions using different worktrees, rather than driving just one session at once.
Today, nearly 80% of PRs that ship to production include code generated by an agent.
More:
- 2.7x headcount growth in product engineering (engineers who ship to production numbered 18 in 2024, and 50 today)
- 3.4x PR requests shipped to production in September compared to a year ago
- 3.6x increase in PRs with no human reviewer shipped to prod (12% to 43%)
- 4.8x increase in the size of PRs (from 22 to 111 lines)
- 6x “effort points” shipped to prod: “effort points” awarded by an AI model estimating the complexity of a PR, and ranking it from XS to XL.
- 80% decrease in human comments per “effort point”
- 60% decrease in incidents per 1,000 pull requests
Stats on release frequency:
- 47% rise in releases/eng/day when taking headcount into account: an average of 2.8 releases/day/eng, up from 1.9 releases/day/eng in 2024
- Oct 2026: 140 releases/day to production with 50 engineers. In Oct 2024, the figures were 35 releases/day to prod with 18 engineers
AI spend
This year, spending on AI tools is on track to be around $1.6M at the company. This is around $2,700 per engineer each month. AI spending peaked this summer and has been decreasing since. In July, the company spent more than $300K on AI.
Opus 5.5 brought a massive cost reduction. Before it, the company was spending around $4,500/engineer/month, mostly on Opus 5. But now, with the 5.5 version, the cost is down to circa $1,600.
Cost per “effort point” also peaked over the summer. The nice thing about the “effort point” metric is that it makes it possible to compare the cost of generating a PR to how much it cost a year ago, when the company started using AI agents.
Infrastructure teams spend more tokens on the same “effort point.” A fascinating detail from incident.io comes from separating the cost of an “effort point.” The infrastructure team stands out, spending ~5x more on a medium-sized PR than almost every other team.
"This is a great example of why it’s hard to use a single target for cost per PR that applies universally. Our AI team, for example, does work that can be very token-intensive such as iterating on evals. Infrastructure teams might use Claude to explore production environments or do lots of analysis or run benchmarks etc which don’t lead to PRs, so their cost per-PR looks higher. Different types of work will simply have different cost profiles."
Measuring ROI
Today, average spend is down at ~$1,600/engineer/month, of which around 80% is on Claude, with 20% going to Cursor. The number of PRs shipped per engineer weekly tracks with tenure.
- The median engineer shipped ~10% more PRs (10.1 PRs in Oct 2024 with no AI; 10.7 PRs per week in Oct 2026 with AI).
- The mean engineer shipped ~30% more PRs (11.1 PRs in Oct 2024 vs 14.8 PRs in Oct 2026)
- Tenured engineers shipped ~70% more PRs (11.1 PRs in Oct 2024 vs 19 in Oct 2026)
New engineers are told to hold off on using AI for a month after they join. The AI SRE startup collected feedback from new engineers and found that new joiners using a lot of AI right after onboarding ended up performing worse in the medium/long term. It’s worth ignoring expectations of shipping velocity during the first month. Instead, new devs learn how to navigate the codebase.
Harder to quantify: shipping faster & incidents
Compared to pre-AI, a benefit of AI agents and AI reviews is faster feedback than waiting on someone to review a PR.
There are fewer incidents per change, but more overall. This is both good and bad news: there’s fewer incidents per PR, but more overall due to the existence of more PRs. The team looked into why incidents occurred, and found that most outages were because of incorrect code that could have been caught in review. The team took steps to address this:
- Improved the codebase for AI use
- Started leaning more on Cursor’s Bugbot, added stricter rules, and saw the tool start to give better feedback and catch more code-related bugs in review
- Newer models produced better code.
Today, most incidents are no longer due to faulty code, but because there’s information absent about the production environment. They are working on building feedback loops where agents can observe and learn from production.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi