The current impact of AI on engineering velocity

Recorded live at DX Annual, Abi Noda, co-founder and CEO of DX, joins Brian Houck of Microsoft to share an early look at DX’s new research on AI’s impact on engineering velocity.

Drawing on data from a sample of DX customers, they discuss what companies are actually seeing as AI adoption matures. Most organizations in the study saw pull request throughput increase by 10 to 15 percent—far more modest than the 10x gains often promised in industry headlines.

They explore why coding remains only a small part of developer work, where time saved by AI may be going, and the unintended consequences of moving faster, from shifting bottlenecks to “false velocity.” Abi also shares how engineering leaders are applying AI beyond coding and how DX is evolving its measurement framework to account for both human and agent productivity.

Some takeaways:

Most organizations are seeing modest gains from AI

• PR throughput is increasing by about 10 to 15 percent. Across DX’s sample, most organizations saw measurable improvements, but the gains were far smaller than the 10x productivity increases often cited in industry headlines.
• The median improvement was closer to 8 percent. While some organizations saw larger gains, the typical impact was more incremental than transformational.
• Even modest gains can be meaningful at scale. A 10 percent increase in throughput can represent a significant improvement when applied across hundreds or thousands of engineers.

Coding is only one part of the productivity equation

• Developers spend only about 14 percent of their time writing code. If AI primarily accelerates coding, its impact on overall engineering velocity will naturally be constrained.
• The biggest bottlenecks often lie elsewhere. Planning, reviews, testing, documentation, and coordination still consume the majority of engineering time.
• Time savings do not map neatly to output gains. Organizations can see meaningful reductions in coding effort without a proportional increase in pull request volume.

Why productivity gains are lower than many leaders expected

• Coding is not the primary bottleneck. Improving a small slice of the development process only moves the overall system so far.
• Automation creates new bottlenecks. Faster code generation can increase pressure on reviews, QA, and technical oversight.
• Social friction slows adoption. Skepticism, inconsistent usage, and unrealistic expectations can limit the benefits of AI tools.
• Tool and skill gaps compound over time. Engineers need both the right tools and the knowledge to use them effectively.
• AI tools still lack context. Limited understanding of business logic and codebase nuances can reduce output quality.

Beware of false velocity

• More code does not necessarily mean more business value. Teams can increase pull request counts without meaningfully accelerating roadmap delivery.
• Quality and cost remain critical concerns. Organizations are closely monitoring technical debt, token spend, and long-term maintainability.
• Faster output can create delayed consequences. The full impact of AI-generated code may not become apparent until months later.

The biggest opportunities lie beyond coding

• The remaining 86 percent of engineering work is the next frontier. Leaders are applying AI to planning, documentation, incident response, and other parts of the SDLC.
• Autonomous agents can augment human capacity. Instead of simply speeding up developers, organizations are exploring how agents can work in parallel.
• Developer experience still matters. Improving focus time, documentation, and workflow friction can amplify the benefits of AI.

Measurement frameworks are evolving

• Some metrics remain constant. Velocity, quality, and developer experience are still essential signals.
• Acceleration and augmentation should be measured separately. Leaders need to distinguish between human productivity gains and work performed autonomously by agents.
• Agent experience is an emerging concept. DX is beginning to survey AI agents directly to understand their constraints, bottlenecks, and effectiveness.

Cognitive debt is a new concern

• AI can reduce understanding while increasing output. Developers may ship code more quickly while building a weaker mental model of the systems they maintain.
• Short-term efficiency can create long-term costs. Reduced comprehension may make future debugging and maintenance more difficult.
• The long-term effects are still uncertain. Engineering leaders are only beginning to understand the human consequences of AI-assisted development.

In this episode, we cover:

Intro

What motivated DX’s research into AI’s impact on engineering velocity

How DX designed the study and selected companies

What DX’s data reveals about AI’s impact on engineering throughput

Why PR throughput was the most practical metric to publish

Why AI productivity gains are lower than many leaders expected

How an all-in culture can amplify AI productivity gains

Why it’s hard to track where AI-generated time savings are going

Unintended consequences of AI-driven productivity gains

Why leaders should look beyond coding to the rest of the SDLC

Cognitive debt and the human costs of AI-assisted development

How DX’s AI measurement framework is evolving

How to make agents more effective

Referenced:

• DX Core 4 Productivity Framework
• DORA, SPACE, and DevEx: Which framework should you use?
• Time Warp: The Gap Between Developers’ Ideal vs Actual Workweeks in an AI-Driven Era - Microsoft Research
• How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt
• Measuring AI code assistants and agents