Modern product teams are simultaneously balancing the speed with which they need to deliver new features and the need to test hypotheses, uncover unmet needs, and gain new insights about their users. So many user research questions, so little time!
Given these competing pressures, how should a product team go about determining how to invest their own limited time as well as their customers’ time? When is it worth investing in time-consuming primary research, and when is it ok to “ship it and measure it”?
At Pendo, we developed a prioritization framework to help us think through these types of decisions.
This user research prioritization framework uses two different axes to evaluate the degree and type of resources that are generally appropriate for any given research question or project. Here’s a breakdown of each axis:
Problem Clarity
The first axis, Problem Clarity, is fairly straightforward. Are you able to–crisply and clearly–articulate the customer’s problem? In order to answer “yes,” there should be reasonable evidence, ideally both qualitative and quantitative, to support the customer problem.
It’s important to note that a common mistake that is made while evaluating against this criteria is to substitute a business problem for a customer problem. While it’s certainly important to identify business problems and goals, in order to achieve those business goals it’s critical to ensure they are aligned with a deep customer pain.
Risk
The second axis, Risk, is specifically referring to the “risk of getting it wrong.” This risk should take into consideration the various costs associated with getting your product decision wrong, including the cost of reengineering and redesign. These might be actual costs (e.g. the cost of building a feature or the cost of losing a customer) or they may be less direct costs, such as the cost of retraining customers. For bigger, more complex research questions it may be necessary to consider the risk of getting it wrong from a legal, pricing/packaging, or product/market fit perspective.
Now that we’ve got the definitions of each axis down, let’s walk through the framework one quadrant at a time.
Research Heavy (Problem Clarity = Low, Risk = High)
This is where the “wicked problems” lie. These research questions might be proactive, for example exploring new product growth opportunities. Or, they might be reactive, such as taking a deep dive into stated reasons for churn and lost accounts. Regardless, the clarity is murky at best and the stakes are high. Solutioning too quickly runs the risk of solving the wrong problem–or no problem at all.
If there’s anywhere to spend valuable primary research time, this is it. Questions in this quadrant ultimately save time by requiring you to slow down and invest in the research. The return on this investment will be experienced subsequently by the entire product team, as they will be able to form better hypotheses and make more accurate assumptions due to the depth of their understanding. At Pendo, we use the product itself as a channel for facilitating research. Our UX team leverages in-app guides and strategically-defined segmentation to poll customers about product ideas and recruit them to participate in qualitative research activities.
Design Heavy (Problem Clarity = High, Risk = High)
When customers (and their problems) are clearly understood, the focus of research can shift toward rapid prototype and design evaluations. The factor that determines how much and what type of research needs to be done is the degree of risk.
In organizations that utilize design systems, an effort to establish a new design pattern would fall into this category. By their very nature, individual design patterns are meant to infiltrate products, streamlining the development process and creating a consistent user experience. Thus, “getting it right” (or at least “pretty right”) is worth the up-front research investment.
Ideally, research in this quadrant exists to empower and inspire designers, providing decision-making confidence at a pace that tracks with their design work. Fortunately, the user research tooling industry now has several products that aim to address this particular need. For example, our UX team links Pendo in-app guides to prototype designs to collect feedback from users.
Research Light (Problem Clarity = Low, Risk = Low)
Research questions in this quadrant are often murky, but don’t present the same types of risks as with the “Research Heavy” quadrant. Deeply understanding the customer problem is important, but since the risks are low, the need to be accurate is diminished.
The goal is to increase problem clarity while minimizing resource investment. Collaborating with your broader team to identify and test hypotheses through a series of rapid research activities will likely yield sufficient problem clarity to move forward to design and solutioning. One key research activity for our team is to target specific segments of our user base with in-app guides that include polls and/or open comment fields. This way, we’re able to collect quantitative and qualitative data to help refine the problem space.
Ship it and Measure (Problem Clarity = High, Risk = Low)
There are times when it is absolutely appropriate to skip primary user research and learn solely through experimentation. When the problem is clearly understood and the risk of getting it wrong is low, the best research approach is to make an informed hypothesis, create a design to test that hypothesis, and measure the results. The important step here is to ensure two things: 1. You can, in fact, measure the results of your hypothesis, and 2. You actually do monitor the data and learn from your experiment. Product analytics tools are essential for the former, and the ability to run experiments and monitor conversions for in-app guides is helpful for the latter.
In the end, no two companies, products, or teams operate the same way. You’ll likely determine what works best for your team through trial and error, but I hope this research prioritization framework helps you get started on the right foot.
