When organizations embark on agile transformations, they often fall into what we call the "metrics trap". They measure success by implementation milestones: number of teams trained in Scrum, percentage of projects using agile methodologies, or velocity points delivered per sprint. These metrics might look impressive in status reports, but they tell us nothing about whether the transformation is actually improving business outcomes.
As one frustrated CEO told us, "My dashboard is green, but our customers are still unhappy and our time-to-market hasn't improved. What are we missing?".
The Problem with Implementation Metrics
Implementation metrics focus on activities rather than outcomes. This is often because organizations want to hear about progress, they don’t want to hear about challenges. So showing that we are busy is safer than the more challenging problem of actually making the expected change. They tell you that you’re doing something, but not whether you're getting better results. Consider these common scenarios:
A team reports high velocity but delivers features nobody uses
An organization celebrates 100% Scrum adoption while customer satisfaction decreases
A department tracks sprint completion rates but ignores growing technical debt
These metrics create an illusion of progress - we might even consider them to be a lagging indicator. However, they potentially mask the very problems the transformation was meant to solve. However, realize that a shift to real impactful metrics can be hard - if your organization demands progress over problem solving, you might be better off tracking activity. Instead, our goal is to encourage you to shift to outcome-based measurement. It’s harder to show continual progress week to week, but it is much easier to show meaningful impact over time.
Shifting to Outcome-Based Measurement
Effective measurement starts with a fundamental question: "What outcomes are we trying to achieve with this transformation?". The answers typically fall into one of four categories:
Delivery Performance The DORA metrics (from the DevOps Research and Assessment team) provide a research-backed framework for measuring software delivery performance:
Deployment Frequency: How often you can successfully release to production
Lead Time for Changes: How long it takes from code commit to code running in production
Mean Time to Recover: How quickly you can recover from failures
Change Failure Rate: What percentage of changes result in degraded service
A healthcare technology client we worked with initially resisted measuring these metrics, arguing that their regulatory environment made frequent deployments impossible. When they finally started tracking the DORA metrics, they discovered their Change Failure Rate was over 60% explaining a big part of their reluctance to deploy frequently: each deployment created a cascade of defects to address. By focusing on quality and automating compliance checks, they reduced this to under 15%, enabling more frequent, smaller, and safer releases.
Flow Efficiency Flow metrics help identify bottlenecks and delays in the software delivery process itself. For example, we might :
Cycle Time: How long work takes from start to finish
Work Item Age: How long current items have been in progress
Flow Efficiency: Percentage of time items are actively worked on versus waiting
Throughput: Number of items completed over time
A financial services client discovered through flow measurement that their approval process was causing 80% of their total cycle time. Furthermore, a large part of that time was spent waiting for approval. Work items spent an average of 12 days waiting for approvals and only 3 days being actively developed. This insight led to a redesigned approval process that reduced Cycle Time by 70% and increased Flow Efficiency.
Customer and Business Impact Ultimately, agile adoption (or any transformation) should improve business outcomes. While business outcomes are often seen through a financial lens, we want to consider a range of different metrics such as:
Customer Satisfaction/Net Promoter Score: Loyalty and satisfaction from customers
Feature Usage and Adoption Rates: Actual adoption or usage of delivered features
Revenue/Cost Impact of New Features: Value created or costs saved from new features
Time-to-Market for New Initiatives: Time from concept to cash
A retail client measured feature adoption and discovered that nearly 40% of their new features went unused. By creating more frequent customer feedback opportunities and proactively encouraging adoption of new features, they doubled feature adoption to over 80% while delivering fewer features overall—a classic case of "less is more".
Aligning Metrics with Organizational Goals
The most powerful metrics connect directly to your organization's strategic objectives. We spend time working with leadership teams to articulate clear objectives, starting with:
What are your top three strategic priorities?
How would you know if your transformation is helping achieve these priorities?
What leading indicators might predict these outcomes?
This exercise reveals metrics that matter in your specific context. A media company aiming to increase subscription revenue might focus on reducing the time to implement personalization features. A manufacturing firm focused on quality might measure defect escape rate and automated test coverage.
Understanding Metric Limitations
Every metric has limitations. Single metrics never tell the complete story, and all metrics can (and probably will) be gamed or misinterpreted:
Deployment frequency without quality controls can increase failures
Velocity increases might come at the expense of technical debt
Customer satisfaction might temporarily drop during major platform changes
To mitigate these limitations:
Use balanced sets of metrics that complement each other
Look for trends rather than absolute numbers
Combine quantitative metrics with qualitative insights
Regularly reassess your metrics as your context evolves
Starting Your Metrics Journey
If you're unsure where to begin with outcome-based metrics, we recommend this approach:
Start with one or two DORA metrics to understand your delivery performance baseline
Add basic flow metrics (cycle time and throughput) to identify process bottlenecks
Connect these to at least one business outcome metric meaningful to leadership
Review metrics regularly and adjust your transformation approach based on insights
Remember that the purpose of measurement isn't to produce perfect numbers but to drive better decisions. The right metrics should prompt meaningful conversations about your transformation journey and highlight opportunities for improvement.
In our next post, we'll explore how to create an organizational structure that supports agile delivery. For now, we encourage you to examine your current metrics: are you measuring implementation activities or actual outcomes? Do your metrics connect clearly to your strategic objectives? Are you using the insights to adapt your approach?
The true measure of agile success isn't in the practices you implement, but in the outcomes you achieve for your business and customers.