Five Considerations for Measuring Product Performance – Setting the Stage for Great Outcomes

When making decisions about your product or service, relying on anecdotal evidence and opinions can be risky. With the right data to guide you, it can be easier to determine whether you should continue investing, pivot your strategy, or abandon a product idea altogether. Making confident and informed decisions requires access to reliable data. Yet obtaining good, actionable data requires planning. Here are common questions product teams must ask themselves:

• What type of planning is required?
• How do you decide what to measure?
• When to measure it?
• How will you measure it?
• Who looks at the data, and how often?
• What would tell you your product is heading in a good direction? A bad direction?

These are the kinds of questions that regularly come up when we’re working with our clients.

Here are five key elements to consider when planning your product performance measurement approach: 

• Use visual aids to support collaboration and alignment on measurement. Aligning what you are measuring and why is essential to effective product investment.
• Capture hypotheses, benefits, and measurements in your backlog.
• Design your measurements and dashboards before building your product features.
• Get detailed about your metrics and how they will be used. Having data makes it much easier to adapt to changing circumstances.
• Use the data. Being a data-focused product organization takes practice. The more you do it, the more effective you become in designing and using product dashboards to improve your overall business performance.

This article will explore the key challenges and strategies for improving subscription renewal rates using a fictional scenario centered around the software company ContinuLearn.

ContinuLearn is a company seeking to boost the renewal rates of its customized content subscriptions and has selected subscriber engagement and renewal rates as a core product performance metric. This scenario illustrates what it looks like when a company uses metrics well and knows how to integrate them into product planning. Through this narrative, we aim to shed light on the crucial strategies and insights that are pivotal to success in product performance planning and measurement.

Setting the Scene

ContinuLearn sells a Learning Management System that enables customers to create their own internal curriculum. In addition, the tool offers canned training modules they provide with their system such as “Having Difficult Conversations,” “Building Trust,” and “Objective and Key Results (OKR) Development.” The training modules they sell on top of the LMS are currently provided via a yearly subscription fee, with a fee per learner who can access these modules.

ContinuLearn faces a significant challenge. When reports are run at the end of the year, not enough employees use the included modules for companies to pay the cost. This is measured using “consumption rate,” which is the percentage of people who sign up for and complete a module. Often, this leads to companies not renewing their subscription for the canned modules the next year. They are competing against YouTube, which people are comfortable using to learn, both privately and at work. People naturally go to YouTube instead of the ContinuLearn platform for educational content. If ContinuLearn can’t make enough money on subscriptions, they can’t afford to create new modules and keep their current modules fresh, which may cause other subscription users to decide not to renew.

Joanne Ashford, the Product Manager for ContinuLearn, aims to increase the renewal rate by 7%, and a key result is to increase the consumption of modules by 15% in the next year. The underlying hypothesis is this will lead to renewals by existing subscribers and attract new subscribers. There are three major areas she has to focus on:

• Achieve earlier time-to-market with modules on timely hot topics in areas such as Cyber Security and AI.
• Add the ability for learners to get “certified” in a skill by taking an exam, possibly for an additional charge.
• Gamify the platform with leaderboards and achievement badges for learners to show off what they’ve learned to their peers. If people see their peers learning and getting badges, they’ll be drawn to do the same.

Before you start building anything to implement these ideas, there are several things you need to research. For example, would companies pay extra for certifications for their employees? Do their clients have any internal gamification today? If yes, how would ContinuLearn platform fit in? Does ContinuLearn’s vision of popular topics align with the priorities of its user base?

But this piece is about measurement, not other aspects of Product Discovery. How can Joanne Ashford communicate and measure the benefits of hot topic modules?

Use Visual Aids to Facilitate Collaboration and Alignment

Let’s explore creating modules and curriculums for popular topics. One tool that can be used to facilitate discussion is the hypothesis lattice:

The hypothesis lattice is a visual representation that allows people to discuss assumptions and causality. For instance, if we take action (X), we expect outcome (Y). You can easily create one with sticky notes on a whiteboard or with a virtual collaboration tool. Participants can freely discuss the model, critique it, move sticky notes around, and add new links. For example, someone may question whether the model assumes that all participants find the modules useful and suggest an improvement. The diagram can then be updated accordingly.

You could do more with this diagram, but our intent is to give you a general idea. In practice, these get more detailed, take a while to create, and lead to valuable discussions amongst product stakeholders.

Capture High-Level Goals, Benefits, and Measurements in Your Backlog

If you are investing in something substantial, like a new curriculum, it makes sense to have a brief but effective reason for that investment in your backlog. This keeps the goal behind the product development work at the forefront. Without the goal in mind, it is easy to become sidetracked and work on things that don’t support the goal.

Let’s frame that as a simplified variation of an “Epic Hypothesis.”[1] We use the Scaled Agile Framework approach, but any similar format would work.

This is a good start for creating concrete measurements, as it is explicitly tied to business outcomes and leading indicators.

• It will take a while to see if renewals are affected, but this is the reason the company is creating the new modules and, therefore, critical to measure. However, you also need a measurement that provides an early indication if your strategy is working, like course registrations and completions.
• Sponsor surveys are another way to find out if the new modules are valued by people paying for the subscription services. It will likely delay the introduction of new modules a bit. The sponsors will want to see both the rate of usage and module NPS before it is likely to affect what they think.

Develop the Details Behind Your Measurements Before Beginning Work

Plan for measurement before you build out features. This is particularly true if you need to create new measurements, new dashboards, etc. It is common for Product Managers and Product Owners to cobble together data in a rush in response to someone asking, “Is this initiative producing what we were hoping for.” Sometimes, they can’t get the data because the software behind the product wasn’t designed to collect it.

We’d recommend understanding the data you need to determine if an investment has proven worthwhile before you agree to fund it. Include features and user stories addressing data collection and visualization into the backlog.

Get Clear on Measurement Details

There are several things to consider as you build out your measurement dashboard.

• What is the name of the measurement?
• What is being measured?
• Who (person or group) will review the measurements?
• Why are they reviewing this? What decisions will they use the data to make?
• When will the data be reviewed?
• How will the data be collected? Is it already collected today?
• Who owns the creation of the metric? The dashboard?
• How are the data elements defined?
• How will the data be visualized? On what dashboard?
• Is there a time component to the data?
• How will the progress be assessed with the data?

This measurement – Module NPS – is straightforward. What about something more complex, like module consumption? You wouldn’t expect module consumption to peak immediately after it becomes available. For example, module consumption could be affected by:

• Internal campaigns to increase the rate of consumption.
• Word-of-mouth promotion across the company.
• Managers choosing to assign modules to staff.

You might look at typical module consumption for other new offerings and compare that to the consumption rate for the hot topic course. Consumption is typically initially low, builds, and then drops off. For a hot topic module, you might expect:

• Higher rate of consumption – more people take the new AI course vs. other new offerings.
• Faster rate of consumption – consumption of the new module ramps faster.
• Longer tail: There are new people taking the course for a longer period of time than is typical for new courses.

Use the Data

Great dashboards, not used, are a waste. Creating, displaying, and using data to drive product outcomes is a skill that takes a while to develop. Leaders and stakeholders establish the culture for measuring success. When a senior leader talks about key measurements in a public forum–say a town hall–more people start paying attention and thinking about how they can help the company achieve its product goals and, ultimately, its product strategy.

Summary

Companies that use data to steer their products and make better investment decisions are more effective than those that don’t. If you aren’t currently using data, there are specific steps you can take to get there that we’ve outlined in this article. In addition, helping companies move to using metrics and using them effectively is a service we offer. Building a metrics culture is a mix of knowing what to do, how to do it, and doing it. Please reach out if you would like to learn more. One of our Eliassen Group experts will be happy to help you achieve your product discovery goals

[1] SAFe Studio. “Epic.” Scaled Agile Framework, January 24, 2024. https://scaledagileframework.com/epic/.