The Benefits of Failing Faster

If you’ve taken an Agile training course, you’ve probably heard the term “fail fast.” The goal is to be intentional and efficient about your decisions, some of which may lead to failure. When do you learn? How do you learn? How are learning and failure connected? Can you shift learning and failing earlier in the product lifecycle, decreasing the cost and impact of learning? This article explores the process of learning and failing – both integral to product creation.

Why do we talk about failing and not just learning? Isn’t failing a bad thing? In product development, you can sometimes simply learn what you need to know. If you want to learn how many Customer Relationship Management systems exist and what they do, that is directly learnable. However, you can’t directly learn if people will embrace your new CRM product without having your potential customers respond to something. It can be a prototype. It can be a minimal product that addresses a portion of a customer challenge. But after the analysis is done, you do something, and the result you get may be different from what you expected.

In science, this happens all the time. There is a hypothesis, it is tested, and the results may be what was expected or not. If you are in the uncertain business of product development, sometimes things will work as expected, and sometimes things will not. We think a better expression is “Fail Fast, Fail Safely, Fail Efficiently.” This phrase communicates that a thoughtful approach to product discovery can save you time, cost, and reputation. We will use “fail fast” as a shorthand in this article.

When creating products for internal or external customers, the goal is to maximize the return on investment and create something people love and use. Every product idea stems from a set of assumptions about where people currently encounter challenges, how your product could offer a solution, why customers might choose it, and how it will generate revenue or optimize costs. These assumptions may be accurate, or they may not. "Failing fast" essentially means testing our most crucial assumptions as early and cost-effectively as possible to determine whether further investment in our idea is warranted. It also suggests that, depending on what we discover, we may need to pivot or even abandon our product idea completely. This article is dedicated to promoting rapid learning, especially in:

• Creating a product learning culture
• Uncovering your key assumptions
• Prioritizing assumptions for investigation
• Testing assumptions
• Reflecting on what you learned
• Adjusting your plans

Each topic mentioned could be elaborated on extensively, perhaps enough to fill a book. The purpose of this article is to help you begin this journey.

Creating a Product Learning Culture

Anchoring better product outcomes in feedback and rapid learning begins with cultivating a product learning culture. Below are fundamental perspectives encompassed in that culture:

• We know that our understanding of the customer, their needs, and our capability to meet those needs is incomplete.
• We know that our grasp of our potential to create a sustainable, scalable business model from our product is equally imperfect.
• We must find a balance between analysis and action, with a preference for action, because we cannot predict the market's reaction to our releases.
• To wisely utilize our product development budget, we should progressively and iteratively test our assumptions, integrating feedback loops with data into our process. 
• Learning happens throughout the product lifecycle, and it is not limited to what happens on Agile teams.
• Smaller releases with a subset of customer value are preferable to larger releases that carry greater product risk.
• Effective learning necessitates collaboration and transparency among stakeholders and the teams developing the product.

Learning in product development is an ongoing, continuous process. It can occur anywhere in the product lifecycle, from the time you have an initial idea to when you’ve completed the project. Engaging with potential or existing customers—whether through interviews, UX testing, or analyzing product usage data—yields fresh insights to guide your next steps.

Product Managers and Product Owners foster a product learning culture by openly discussing key hypotheses, assumptions, and plans for testing. They incorporate these hypotheses into high-level backlog items, making them visible. They engage with stakeholders and the team, discussing hypotheses and using data to substantiate their learnings, making this data accessible for feedback. They understand that learning is a collaborative effort and is most effective when individuals with diverse perspectives interact and learn together.

The culture within an organization is vital for effective product development. When a company is open to constructively questioning its strategy and product direction, many missteps can be prevented, and capital can be directed toward optimizing business outcomes.

Uncovering Your Key Assumptions

Product Discovery is a key technique for spending a little time to learn quickly instead of spending a lot of time on something that doesn’t produce the desired results. Eric Reiss said this in his book The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses.

“I call the riskiest elements of a startup’s plan, the parts on which everything depends, leap-of-faith assumptions. The two most important assumptions are the value hypothesis and the growth hypothesis. These give rise to tuning variables that control a startup’s engine of growth. Each iteration of a startup is an attempt to rev this engine to see if it will turn.”[1]

While the quote emphasizes “startup,” any time you are investing in a significant new product or update to an existing product, there are likely assumptions that should be tested before going all-in on development. And once you ship the first Minimal Viable Product (MVP), you want to continue learning with rapid feedback loops. When we work with our clients, we help them identify their key assumptions and discuss strategies for economically exploring those assumptions.

Let’s consider an example of a product company trying to reduce customer calls, improve customer satisfaction, and reduce product service costs. Today, they have a chatbot that provides customers with simple options to solve their problems without having to interact with a staff member. Only about 7% of the customer issues get solved via automated prompts. In addition, customers report feeling frustrated having to go through multiple generic prompts before they can get to a person. The Product Management team would like to improve how this works by using a Large Language Model AI to interact with the customer instead. What might be some assumptions to test?

• Customers will interact in natural language with a chatbot instead of waiting for a person to connect to the chat.
• The AI chatbot will do a better job resolving customer issues, enabling an increase in automated problem resolution from 7% to 20%.
• Customers will find the experience satisfying and useful.
• Building and maintaining an AI chatbot will be better overall than enhancing the current prompt system.
• The company will be able to hire people with the skills to build the chatbot or be able to train internal staff to work with Large Language Models.

Developing the list of assumptions will often be more effective with multiple people working collaboratively. For example, a product manager might not think about the challenge of hiring or training people to do the work, whereas a technologist might.

Assumptions usually fit into a few categories:

• Customer value: Our customers and buyers will find the product addresses significant needs. Note that buyers and customers might have different needs. A buyer might focus on cost-effectiveness, scalability, and integrations with other systems. A customer would be more likely to focus on how the product would help them overcome a significant pain point.
• Internal value: The product will contribute to the company’s strategic goals for an acceptable level of investment, time, and capacity.
• Feasibility: The product can be built and integrated effectively into the existing landscape of products.

If you don’t identify and investigate key product assumptions, you may pursue a nonproductive product path, wasting valuable capital. This capability is integral to product success. Are you good at identifying assumptions? Where could you improve? Where will you improve?

Prioritizing Assumptions for Investigation

Prioritizing which assumptions to test is important. Testing assumptions means spending a smaller amount of money now to perform an experiment and learn quickly. This saves money in the long run as well as time, capacity, and potential reputation hits by reducing the likelihood of building the wrong thing. The investments you make testing product assumptions should be balanced by the potential risk of not learning. The assumptions you typically focus on are ones where:

• There is a high risk your product will not produce the expected value.
• There is a way to test the assumption that will provide useful information.
• The cost of conducting the investigation is commensurate with the risk. You wouldn’t spend $200K to mitigate a potential $50K risk.

The assumptions you are testing go onto your Product Backlog. As you learn, you refine and reprioritize that backlog based on what you now know. Prioritizing assumptions is important because, inevitably, there will be more assumptions than you have time or capital to test. Addressing the most important ones is key to reducing product risk.

Testing Assumptions

Testing assumptions are usually done via an experiment that provides an indication of how things will evolve after a product is released to the market. Testing assumptions is a vast topic. Here are some examples:

• We think our customers would love advanced analytics to manage supply chain costs. You can interview a focus group of supply chain managers to learn about how they might react to your product idea This might also be an opportunity to prototype analytics to provide prospective users with a way to test it out with fabricated data. Both ideas are proxies for launching the product. The learning is, hopefully, indicative of what will happen when you launch.
• We think our travel customers would love to book a cruise for a group, say their extended family, if it were easy to do. You could release this feature to the market in a way that is not “built well” – not technically sustainable or scalable – just good enough to get it in front of a group of customers. If the customers respond well, you rewrite the product feature, so it is sustainable and scalable. Note this approach was used regularly by a large, well-known travel company.
• We think our customers would buy a book on planning travel to see the sights of Ancient Greece. You can announce the book and see who signs up for more information. This technique was regularly used by large mass-market book publishers. They’d write the first chapter of a book, design the cover, and send a chapter out to prospective buyers. If enough people signed up to get a copy, they’d write the whole book.

There are many books, articles, videos, and courses on this topic, for example, Bland, David J, and Alexander Osterwalder. Testing Business Ideas. Hoboken, New Jersey, John Wiley & Sons, Inc, 2019.

Interviews are a popular tactic, but they are challenging to execute. Asking a prospective customer, “Would you use this feature?” is unlikely to get a useful response. A “yes” might just mean the prospect was trying to be polite. For someone to do something new, they must have significant frustration with what they do today and be willing to overcome the inertia of moving from their current approach to a new approach. There must be a significant struggle. If you don’t dig into the struggle and really understand it at the root, you might not understand what product they need and would adopt. See Moesta, Bob, and Greg Engle. Demand-Side Sales 101. Lioncrest Publishing, 22 Sept. 2020., for several examples of in-depth interviews.

Testing assumptions is critical to reducing product risk. Designing and executing tests for assumptions is a core competency in effective product culture. If this isn’t an area of focus for your organization, start testing your most critical initiatives. Practice is core to improving.

Reflecting on What You Learned

Gathering and evaluating data on your product assumptions is not just critical; it's a skill as crucial as learning from the outcomes. Yet extracting meaningful insights from experiments often proves more complex than anticipated. Human biases about customer preferences can cloud judgment, making it surprisingly simple to overlook evidence that challenges our expectations.

Consider the case of an airline exploring new service enhancements, such as luggage pickup and delivery or food pre-ordering options. The team conducted interviews with passengers in the waiting areas but didn't encounter much enthusiasm for the proposed ideas. Despite this lukewarm reception during their review session, the team remained confident in the value of their original ideas.

Faced with such a dilemma, should the team have set aside their ideas or refined them for further customer evaluation? Eric Ries's concept of “pivot or persevere” applies here: the path forward isn't always obvious post-experiment. It demands that teams conduct a rigorous analysis of the data. At times, another round of experimentation may be the key to unraveling deeper insights and making informed decisions about the product's future.

Adjusting Your Plans

When you have gathered and reviewed data and reflected on what you have learned, you must be able to decide on the next steps. Should you launch a limited version of the product, discard the product idea altogether, or conduct further testing?

An effective product culture is one that can decisively abandon product ideas that lack the desired value. However, this can be more challenging than expected. Here are a few reasons why:

• If your company has a protracted process for approving product ideas—perhaps through an annual planning cycle—there may be a reluctance to discard an idea, particularly if obtaining funding for a new product idea could take months.
• If abandoning a product idea is viewed within your company as indicative of poor product management, there may be a tendency to continue investing in the idea simply to avoid a negative perception.

Do you currently utilize data to inform adjustments to your plans? Where is this data discussed, and who participates in the discussion? Does this lead to actionable changes? Are you willing to forsake product ideas when they no longer align with your objectives? Is the ability to abandon a product concept based on data valued?

Summary

“Learning Fast” and “Failing Fast” encapsulate much more than their names suggest. Throughout this article, we've delved into the nuances of these concepts to shed light on their true meaning. At its heart, “learning fast” and “failing fast” is about rigorously identifying and testing crucial assumptions, actively learning from the outcomes, and nimbly adapting based on those insights. This cycle not only boosts the return on investment but also avoids the risk of sinking funds into unproductive product initiatives that don't yield valuable returns.

[1] Ries, Eric (2011-09-13). The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses (p. 76). Random House, Inc.. Kindle Edition.

Written by: Rita Emmons and Bob Fischer