Why AI productivity measures are an old, wrong answer

At Equal Experts, we work with our customers to embed AI native delivery into teams at scale. As Head of Modernisation & Platforms, I can see productivity measures on a comeback, as an answer for AI ROI questions. It’s a real concern.

Introduction

The pace of change in AI native delivery is relentless. The shift to token-based pricing by Anthrophic, Microsoft, OpenAI, et al means organisations need to understand the value derived from running models at scale. Stories about excessive AI costs don’t help, like Microsoft cancelling their internal Claude Code licenses, and Uber burning through its 2026 AI budget in four months.

We sympathise with senior leaders who are under pressure to demonstrate ROI for AI native delivery, and engineering productivity measures are an understandable response. Counting active users, AI generated pull requests, AI tokens etc. is easy to implement, easy to report upwards. But productivity measures are an old, wrong answer to ROI questions. I explained why recently on Dave Farley’s Modern Software Engineering channel.

Productivity measures are low value

AI usage measures have a low information value, because:

• They’re outputs, not outcomes. If my AI usage increases to 100%, or decreases to 0%, it doesn’t tell you anything about time to value, or total cost of ownership. You can’t know if something useful has happened
• They’re easily abused. People behave based on how they’re measured, and if they’re measured on AI usage they’ll use AI in strange ways. Amazon shutting down an internal AI leaderboard that incentivised AI usage is a good example of this. It’s Goodhart’s Law in action.

Usage metrics can tell you something about behaviours linked to AI models, but not if any value is derived from them.

Implement the Accelerate metrics

AI won’t change our advice on metrics. We’ve advised our customers to use Accelerate metrics by Dr. Nicole Forsgren for years. They’re the gold standard for engineering metrics. They’re outcome-focussed, and statistically significant predictors of business performance. Once you’re able to see your trends in throughput, technical quality, and reliability, you can search for the presence or absence of AI usage correlations. That’ll make for some interesting conversations.

And… you still might have to track AI usage metrics. My colleague Phil Parker has good advice there – treat them as inputs into your outcomes. Aggregate them to understand adoption, but don’t use them to judge delivery performance.

Conclusion

The ROI of AI native delivery has to be well understood, if it’s to be squeezed into budgets. Productivity measures like counting users, pull requests, or tokens are becoming popular (again), but they’re ineffective. They’re not connected to outcomes, and they’re distorted. Implementing the Accelerate metrics gives you the best chance of understanding technology outcomes, and the impact of AI native delivery in your organisation.