Teresa Torres’ book Continuous discovery habit is the best product management book that I’ve read to date, full of slick techniques and novel perspectives. Torres provides a fairly structured methodology for how PMs should figure out what to build, and this blog post distills my main lessons learned from reading her book into a quick read.

Opportunity Solution Tree – The Central Artifact

The main thing that Torres uses to organize product discovery is called the opportunity solution tree (”OST”), and it looks like this:

• Business Outcome
  • Product Outcome
    • Opportunity
      • Solution
        • Assumption

The OST is a way of working through the discovery process. It involves moving through the tree from top-to-bottom, starting at the business outcome that you’re trying to achieve and then, through a series of steps, arriving at a validated solution. She calls the OST a “tree” because it’s it follows the structure of a tree data type, where there can be many child nodes for every parent node. In the OST, for every business outcome, there can be multiple associated product outcomes, and for each product outcome, there can be many associated opportunities.

The starting point of the OST is the business outcome that we’re trying to drive. This could be financial, like increasing revenue, or it could be strategic, like increasing market share or increasing customer satisfaction.

After the business outcome has been established, the question then becomes what product outcome must be achieved in order to drive that business outcome. The distinction between a business outcome and a product outcome is useful, because getting users to engage with with your product as desired doesn’t always drive the target business outcome. De-coupling the measurement of the product from that of the business can help to uncover assumptions about how the two are related.

With the business outcomes and the product outcomes defined, the next step is to try to identify different opportunities to drive those outcomes. Opportunities is a broad umbrella term that can refer to customer pain points, needs or desires, or any kind of deficiency that could be addressed in order to drive that outcome. The defining characteristic of an opportunity is that it’s broadly defined enough to be susceptible to multiple solutions.

Speaking of solutions, brainstorming solutions is the next step. In this context, solution is defined as a concrete, implementable idea that could address an opportunity.

For the most promising solutions, the final step is to identify the assumptions that the solution depends upon so that we can test them and validate the solution. In this step, the main focus is to validate what needs to be true in order for our solution to actually drive the outcome that we’re looking for.

One thing worth noting is that the OST is both a process, and a document. As you progress through the process, you create and update the document. It’s a living-breathing document that evolves to reflect your most up-to-date thinking. The OST can be represented either a visual diagram with a new row for each step down the tree, or as a bulleted list with a new level of indentation for each step down the tree.

The next few sections of this post explain in more detail how to carry out each step of this process.

Outcomes

The starting point of the discovery process is to identify the outcomes that we’re seeking to drive. Starting with outcomes instead of solution ideas is important because it implicitly acknowledges that research is required in order to come up with the best solution ideas.

There are 2 types of outcomes to identify at this stage: product outcomes and business outcomes. To illustrate the difference between the two, let’s review a hypothetical example involving a B2B SaaS products that makes money by charging each customer a subscription fee. In terms of businses outcomes, suppose that the company’s top strategic priority is to expand it’s market share. The businses outcome they measure to track their success in that endeavor could be month-over-month growth in monthly subscriptions. As for product outcomes, they might want to track how frequently users engage with the product, and their task completion rate in key workflows, because presumably if customers are using the product successfully and often, then they’re likely to be retained. Even if those product metrics are performing well, customers could still be churning away to a competitor with lower prices. In this case, the metrics on the product outcomes could be pointing in a positive direction, while the metrics on the business outcome could be pointing in another direction. Torres emphasizes that product outcomes and business outcomes are related, but each warrant their own analysis.

Picking What Outcomes to Measure

In picking which outcomes to measure, it’s important to be specific about what you’re measuring and why. Small nuances in the way that a metric is defined can make a meaningful difference. For example, the book reviews an example of an online learning website that was trying to allow students to post course reviews so that prospective students could read them before enrolling. Originally, the team set the outcome as just increasing the raw number of reviews posted on the website. However, they later realized that reviews only make a difference if they are actually seen by users. This led them to explore alternative metric definitions, like “views of courses with at least 1 review”, or “average number of reviews on each course viewed”. By thinking through edge cases and being specific, you can fine tune the metrics being measured to better align with the desired outcomes.

Torres has a couple of useful tips when it comes to picking outcomes.

Her first tip is to be aware of the distinction between lagging indicators and leading indicators. Actual outcomes of interest are often lagging indicators, so it can be useful to explore the data to try to find leading indicators that can give early signals about the direction of lagging indicators.

Her second tip is to exercise caution when using traction metrics that measure feature usage. The risk with focusing on traction metrics is that they pre-suppose that increasing usage of a certain feature is will automatically drive the desired outcome. If the feature is not actually the best solution, then a traction metric can pre-maturely limit the search for potentially better solutions. Traction metrics should only be used when we’re sure that we have the right solution and we’re just trying to optimize it.

Identifying Opportunities to Address an Outcome

With outcomes defined, the next step is to identify opportunities to drive the outcome. To re-iterate, an opportunity could be a pain point, an unmet need, an unfulfilled desire, or any problem that needs to be solved in order to better serve the customer and drive the outcome.

After giving due recognition to common ways of finding opportunities, such as by reviewing behavioral data or customer feedback, Torres introduces two other tools that are particularly interesting: 1) experience maps, and 2) structured customer interviews.

Creating an Experience Map

The concept of an “experience map” warrants more discussion. Essentially, the experience map is a storyboard of how the customer goes about achieving some goal, drawn from the customer’s perspective. The purpose is to provide a deeper understanding of the customer experience. The experience map depicts each possible step of the customer’s experience, including multiple paths towards achieving the goal. For example, suppose that the goal is to order delivery from a restaurant. The diagram could start with searching for a restaurant, and it could include paths for calling in an order, or ordering over a mobile app. Each path could have involve problems that the customer encounters along the way.

To create the map, multiple people from different functional roles within the company should independently make their own version, and then later share it with the group. A diversity of perspectives usually ends up producing a more comprehensive overview of the customer experience than any individual person could create independently. At the end of the exercise, the unique element of every person’s map is merged into a single version. The merged version gives structure to the problem space, provokes discussion, and highlights areas of interest that could be further explored in customer interviews. Frequent customer interviews are the lynchpin of Torres’ discovery process, so the next section reviews her suggestions on that topic in more detail.

Continuous Interviewing

Torres emphasizes that frequent customer interviews are central to a proper discovery process. She says that “The purpose of customer interviewing is not to ask your customers what you should build. Instead, the purpose of an interview is to discover and explore opportunities.”

The recommended structure for an interview is to begin with broad, open-ended questions before delving into more specific topics. These open-ended questions allow customers to focus on what is most important or top of mind for them. This is useful for discovery because it may uncover valuable information about something that you would not have otherwise asked about.

As the interview progresses onto more specific subjects, the recommendation is to ask about concrete examples of people’s past behavior instead of asking about how people would behave in a hypothetical scenario. The drawback of hypotheticals is that people often describe how they’d ideally like to behave, which can differ from how they have actually behaved in the past in similar situations. For example, Torres relays a funny anecdote from a time that she interviewed a customer about how they go about buying new clothes. When asked about what criteria they use to buy a new pair of jeans, the customer claimed that fit was the most important factor for them. Then, when asked about the last time that they bought jeans, the customer revealed that they had ordered a pair online without trying them on first because the price was attractive and they didn’t have time to go to the store. In this way, focusing the questions on concrete past behavior helps to surface more useful information.

At the conclusion of each interview, it’s useful to take notes on what was learned and add any new opportunities discovered into the OST diagram.

Prioritizing Opportunities to Address

After a set of opportunities have been identified, the next step is to create an opportunity-oriented roadmap. As opposed to a solution-oriented roadmap, which represents a commitment to deliver a specific set of features, an opportunity oriented roadmap is a commitment to pursue a certain opportunity. Under this approach, product teams experiment with and iterate on alternative solutions until they’ve addressed the top-priority opportunity. If the first solution that the team delivers doesn’t end up driving it’s target outcome, then the team has the flexibility to try another solution. The opportunity-oriented roadmap emphasizes using iteration and experimentation to drive outcomes, whereas a solution-oriented roadmap emphasizes shipping features.

When sizing opportunities, Torres emphasizes that this is a fundamentally subjective decision, and cautions against using scoring to rank the ideas. Of course it’s useful to consider quantiative factors such as the number of customers impacted, how frequently they are impacted, but at the end of the day, this is a judgement call that involves a variety of factors which shouldn’t be reduced to a score.

Torres recommends considering the following factors:

• Market factors – How will addressing the opportunity impact our position in the market? How does this factor into market trends? What are competitors doing?
• Organizational factors – What’s the political popularity of the idea within the company? Can you garner enough internal support?
• Customer factors – Is this opportunity important to customers?

A final point worth keeping in mind when prioritizing the opportunity space: Most decisions in the software business are reversible. In light of that, we don’t need to wait for perfect data and try to analyze our way into the perfect decision. It’s often better to move forward with an experiment, and learn from experience. If we later learn that the selected opportunity isn’t as effective in driving the intended outcome as anticipated, we can always course-correct and choose to pursue another one.

Generating Solution Ideas

The key to successful solutioning is to develop a large quantity of different solution ideas, particularly for strategic opportunities that differentiate your businsess from competitors.

The main way to get a broad range of solution ideas is to have multiple people from different functional roles all brainstorm alone, and then share their ideas with the group. This prevents groupthink, and ends up producing a broader range of ideas than if everyone brainstorms together. Reserach on brainstorming has shown that first ideas are rarely the best ideas. It’s good to go through multiple cycles of independently brainstorming, then sharing, and then going back to the drawing board solo once again.

For inspiration, it can be useful to look at how other products that are analogous in some way have addressed similar problems. It can also be helpful to think about the needs of different customer segments such as power user vs. first time user, citydweller vs. suburbanite.

Identifying the assumptions in your solution ideas

In order to validate a solution, you don’t need to build the entire solution and then see what happens. The trick is to identify the assumptions that the solution depends on, and then test those assumptions without having ever built the solution.

Torres suggests 2 methods for identifying assumptions. The first is to conduct a pre-mortem. Under this approach, you imagine the different ways that your solution could potentially fail. What solutions were wrong? The second method for identifying assumptions is to draw a story map of the solution, with horizontal axis representing time. For each step of the solution, you make a list of assumptions.

In thinking about assumptions, it’s useful to consider the follwoing different categories:

• desirability (do people want it?)
• usabilty (can people figure out how to use it easily?)
• feasability (is it technically possible to build within schedule and budget constraints?)
• viability (can it support the business?)

Prioritizing which assumptions to test

There are 2 factors which should drive the decision of which assumptions to test. The first is the level of evidence. Assumptions that are supported by weak evidence should be tested first. The second factor is the assumption’s importance to the solution’s success. If the solution has no workaround if the assumption is proven false, then that assumption should be given priority for testing. The main goal of assumption testing is to find the key assumptions that our solution depends upon and make sure that we can find some confirming evidence. This serves to de-risk moving forward with the solution.

To make this exercise successful, it’s useful to enumerate all assumptions in your reasoning about how the solution will drive the ultimate businses outcome. You want to identify assumptions about how the solution will address the target opportunity, but also how the solution will ultimately influence the outcomes of interest. The book reviews an example of why this is important. There is a social media company rolling out a new feature that is intended to achieve the businses outcome of increasing ad revenue. The more time that users spend on the app, the more revenue the company earns for displaying ads, so the relevant product outcome was to increase amount of time using the new feature. The company later learned that even though this new feature was widely-used and successful, it simpyl cannibalized an existing feature and thus didn’t result in a net increase in time engaging with the app. This case study highlights the importance of identifying and testing all assumptions on which the success of a feature rests.

Running assumption tests

We test assumptions because it’s the most efficient way to de-risk a solution idea. Building the full solution idea is often overkill if the goal is just to obtain confirming or disconfirming evidence for one of the underlying assumptions. Also, it’s often the case that several different solution ideas depend on a common assumption, so in those cases, it’s possible to get validation for a set of multiple solution ideas without having to pick one which one to build yet. Finally, another advantage of assumption testing is that you can combine multiple different ways of testing an assumption to get more extensive data and triangulate your conclusion.

To run assumption tests, Torres recommends light-weight methods like one question surveys, smoke-screen tests, and unmoderated usability tests. She reviews a case study where a video streaming company is considering expanding their offering to include the streaming of live sports. This business idea depends on teh assumption that the company’s subscribers actually want to watch sports. In this case, they could validate that assumption by showing a subset of users a 1-question survey like: “when was the last time you watched sports?”, or “which of the following sports have you watched in the last month”? If that 1-question survey provided confirming evidence for the assumption, the company could then further test the assumption through other means. For example, they could run a smoke-screen test on by adding an item on their landing page that appears to be live streaming of a game, but when clicked, simply tells the user that the company is considering adding sports and just looking for feedback on how many users would be interested. The results of these 2 tests could help to figure out whether the company was correct in it’s assumption that their subscribers want to watch sports.

One important caveat about assumption testing is that the goal isn’t to prove that assumptions are true with a capital T. In product discovery, we don’t need the same level of rigor as academic researchers trying to establish new scientific laws. The goal with assumption testing isn’t to eliminate risk, but rather to reduce risk down to an acceptable level and get an early signal that the solution idea is on the right track. If it isn’t, then we can evolve our ideas, and our understanding of the customer, so that they no longer depend on the faulty assumption.

Key Takeaways

Torres book gave me fresh ideas about what the perfect product management workflow would look like. I appreciated the way that she illustrated how one can reason more clearly about product management by breaking down their thought process into small, discrete parts that can be independently evaluated.

Also, the book offered very compelling arguments for why product managers should spend more time understanding the opportunity space as opposed to the solution space. So much of finding the best solutions is a result of properly framing the outcomes that we’re seeking to achieve, and the opportunities to do so.