DEVOURED
TBM 437: Tokens, Hours, Points, and Other Curious Proxies
Measuring engineering productivity through proxy metrics like tokens or hours remains fundamentally flawed because it ignores the actual theory of value.
What: In his newsletter 'The Beautiful Mess', Gergely Orosz analyzes how organizations attempt to quantify AI-driven output. He argues that metrics like LLM token consumption or hours saved are misleading because they prioritize activity over business outcomes, echoing historical failures of 'story points' and 'lines of code' as productivity gauges.
Why it matters: Engineering organizations are struggling to justify AI investments by using metrics designed for manual labor, failing to recognize that value creation in software is increasingly decoupled from raw output volume.
Deep dive
- The industry is repeating the mistake of using 'vanity metrics' (tokens, inference costs) to track engineering value.
- Tokens represent cost, not value; equating more tokens with higher productivity incentivizes inefficiency.
- Hours saved is a subjective measure that often ignores the time spent debugging AI-generated code.
- Value is found in outcomes (customer retention, cycle time reduction), not in the mechanics of how the code was generated.
- Managers should focus on 'lead time to value' rather than counting AI-assisted pull requests or daily token throughput.
- The underlying 'theory of value' defines what a company considers success, which technology cannot define on its own.
Decoder
- Theory of Value: A conceptual framework determining what a company considers valuable—whether it is raw speed, code quality, customer satisfaction, or revenue growth.
Original article
AI introduces some genuinely new twists on the problem of measuring ROI, but in the end, it still hinges on a theory of value.
Explore all 43 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi