An Agile transformation is intended to build new business capabilities- to get things done faster, to improve product quality, to reduce risk, to improve predictability, and to improve productivity. Yet, many Agile transformations lack any plan for measuring success. Executives supporting Agile transformation programs often become frustrated with the lack of meaningful metrics. Meanwhile, practitioners debate the relevancy of metrics, often from a silo rather than an end-to-end perspective. Months into the transformation, executives wonder whether the program even merits continued funding. They struggle to pull together data after-the-fact. Because they didn’t have baseline measurements from the start of the transformation, it is hard to demonstrate progress. Often, any data that is freely available becomes the metric of focus.
Measuring and Proving the Success of your Agile Transformation
The primary outcomes of Agile transformations are apparent across three dimensions:
- Flow: Building value incrementally and quickly. This enables prioritized delivery and feedback, and highlights impediments.
- Value: Building the right thing. This prioritizes "outcomes over outputs,” and delivers more with less.
- Quality: Building the thing right. "Built-in-quality" reduces rework and drives productivity.
Alignment across these three dimensions -- and even four dimensions of Agile metrics -- is critical, as is the alignment across all levels of the organization, from team and program, all the way to portfolio management and strategy. While Agile derives speed and innovation through the decentralization of decisions, some decisions must be centralized for the sake of alignment: strategy, vision, road map, and success measures across technology, product, and flow. The measurement of the abstract concept of the successful alignment across these platforms can then be measured with tangible data.
FLOW: Building it Fast
"Velocity" is the most common metric, and it is the one most often abused beyond the Agile team. In other cases, “feature percent complete” based on linked stories can also provide similarly misleading information rarely worth the investment it takes to assemble.
A good measure is the "cumulative flow diagram" (CFD) of stories moving through states, which provides visibility on Agile cycle time, work-in-progress, velocity, and an estimated time required to complete the overall plan. A cumulative flow diagram provides good feedback on flow and lack of flow, which is caused by impediments. The cycle time of a user story from “in-progress” to “accepted” is another measure of flow. Six sigma practices can be used to identify and reduce variation in cycle times and increase flow accordingly. Cumulative flow for features is another important metric that most often highlights cross-team dependencies owning different components needed for delivery of a feature. Cross-team dependency and solution integration impediments, highlighted by feature (or larger work item) cumulative flow, can often be resolved by executives that own the end-to-end value delivery.
The best flow metrics will also include an element of employee engagement or employee happiness. When this metric is trending the wrong way, soft skills and team emotional intelligence are needed to understand and resolve any contributing issues. In this case, an examination of outliers often reveals the root cause of the real problem, rather than a focus on averages.
Who is responsible for flow in an Agile transformation? The scrum master at the team level and those in the facilitation role as we go up the organization, all the way to facilitation of strategy definition, Agile program management, and governance. In SAFe, these are the Release Train Engineers, the Solution Train Engineers, and the Portfolio Management Team. In the Scrum at Scale Framework, this is the Executive Action Team (EAT).
VALUE: Building the Right Thing
Planning and reviewing, including a demonstration of working software, helps to manage the risk involved in execution and closes the “credibility gap” between IT and business. The frequency of this business and IT collaboration is the primary indicator of whether the right thing is being built. Is there clear communication regarding vision, a road map, a prioritized project backlog, and the purpose of the organization? Is there collaboration and alignment on business value and the highest value release plan? Is there opportunity to review a draft of the release plan and an opportunity for the teams delivering the work to estimate and propose a final release plan? Does the culture validate the hypothesis and assumptions with data from larger programs before going all-in on large investments?
A good practice for reinforcing collaboration and the measurement of distinct, relative business value is the Program (value) Predictability Measure from SAFe.1 In terms of SAFe metrics, Program-Value Predictability involves sponsoring executives assigning a "business value score" during planning meetings for each longer-term business objective. This practice provides the best proxy for business value - it is collaborated by business owners and it is done at the latest-possible responsible moment. It is relative to the ratings of other business objectives, and it is used again upon delivery to provide a plan vs actual score, which even accounts for changing business value between the plan date and deliver date. This SAFe metric uses the same basic construct as the Objectives and Key Results (OKR) measure from Silicon Valley and Intel.2
Who is responsible for value in an Agile transformation? On the team level, the Product Owner (PO), higher-level Product Manager, and higher-level business people. These people need to be aligned with the sponsor(s) of the delivery organization. These can be owners of an operational value delivery stream such as customer service or a product and marketing organization, or VPs and executives of the company. They must make decisions that maximize short and long-term value delivery.
QUALITY: Building the Thing Right
Agile implemented properly has a pre-defined quality gate for each team, with the PO serving as a check and balance to accept stories and ensure that definition is followed. Demonstrated capacity is understood through many data points of actual delivery. Additional quality gates downstream are in place to ensure the product is of customer quality before releasing to production. All the downstream, post-story-acceptance work is "technical debt," which includes bug fixes, running tests, fixing tests, release procedures, change control, including customer feedback, and customer service costs. The ultimate vision is built-in -quality and complete elimination of technical debt. This is only possible through a culture of collective ownership across Development and Operations.
A good quality Agile metric or measurement is a count of defects post story-acceptance. A leading indicator of improved quality includes the calculation of automated unit test coverage, feature test coverage, and non-functional test coverage. A good practice is that downstream defects are not only fixed, but also automated tests are promoted to earlier layers in the test harness so that cycle time to detect such defects is shorter. The best predictive measures of quality improvement are investment in telemetry of the integration and deployment pipeline. Tools in this space highlight the location of major bottlenecks and problem areas, so that the development organization can target the most important area to improve next.
Who is responsible for quality in an Agile organization? On the team level, the development team, the higher-level system or solution architect, and the enterprise architect. Alignment on the technology strategy of the overall organization provides guidance on emerging design formed by the development team that is doing the work.
SUMMARY OF AGILE METRICS AND MEASUREMENTS
Ironically, each of the best-practice Agile metrics are a measure of "flow of business value." Short feedback loops are used to improve flow, value, and productivity. Many documented case studies indicate these are the best metrics to guide successful management of a complex business system in a fast-changing environment.
- “Metrics.” Scaled Agile Framework, 2010-2018, http://scaledagileframework.com/metrics/#T4. Accessed 11 Oct. 2018.
- “The Beginner’s Guide to OKR.” Felipe Castro, https://felipecastro.com/en/okr/what-is-okr/. Accessed 11 Oct. 2018.
