Being “data-driven” is a necessity for just about every role nowadays, whether it’s in engineering, sales, customer success, marketing, or product management. But what does that term really mean to the modern PM? What are its limitations, and what does it get right?
Recently, our colleagues at Pendo spoke to two product professionals (and ProductCraft authors!) about all things data-driven product management as part of their webinar series on the topic. First, they chatted with Manosai Eerabathini, PM at Google, about what being data-driven meant to him and his team. In Part 2 of the webinar series, they discussed strategies for managing both quantitative and qualitative data with Rekha Venkatakrishnan, group PM at Walmart Labs. Below, we’ll share some of the top takeaways from their question-and-answer sessions.
“More” doesn’t necessarily mean “better”
The average PM has a huge amount of data at their fingertips. But is more data really “better” data? Not necessarily — at least according to Manosai. He shared a phrase from another product leader that he felt really encompassed the challenge of having so much data available: “data-rich and insight-poor.” Without proper analysis, data is useless. You need to be able to separate the signal from the noise.
Rekha echoed that sentiment, stating that many PMs try to over-engineer their data because there’s simply so much of it. In that case, you have to resist the temptation to dig into data this way and then that way, and then never really get anywhere. She suggested partnering closely with your data science team and allow them to help you draw actionable conclusions from the raw numbers.
Which metrics are the most important?
The idea of an overarching “North Star” metric is a somewhat controversial one if our recent reader polls are to be believed. For Manosai, one data point doesn’t ever tell the whole story. However, each piece of data is tied to the larger product story. In his own decision-making, he asks, “What is the big question I’m trying to answer?” Then, he considers how the piece of data he’s looking at “ladders up” to that higher-level question and its associated metrics. Personally, he believes thinking of data as a hierarchy and/or funnel is a useful mental model for understanding how different metrics interact with an relate to one another.
In addition, Manosai said that PMs shouldn’t keep their assumptions about what the right North Star metric should be to themselves. As a product person, you likely have an idea of what numbers are most important for evaluating the success of a project. However, your colleagues may have other ideas. Be sure to socialize your thoughts and assumptions about tracking and get everyone on the team aligned to the same goal. Incidentally, this is also a great way to build buy-in and solidarity across different groups.
For Rekha, customer-focused metrics are the most important. No single metric can act as a universal North Star for all product teams, but at the foundation of everything a company does should be the customer. This is true for both B2B and B2C organizations. To make product decisions, Rekha’s team puts more weight on metrics related to customer health than any other KPIs.
Where does qualitative feedback fit in?
According to Manosai and Rekha, quantitative data and qualitative feedback are two halves of the same coin. You can’t build the full story of how users are finding value in your product without both resources. Rekha mentioned the importance of reviews. In our own lives as consumers, we’re constantly reading reviews before making purchase decisions. It’s the same for other types of products as well, including those in the B2B space. Testimonials are particularly powerful pieces of feedback that can act as marketing for your application.
To keep a pulse on customer needs, Manosai suggested keeping a close relationship with your colleagues in UX and sales. Of course, you don’t need to build every feature a sales prospect requests. But having an idea where customers and potential customers feel their needs are still being unmet by your product is useful for future plans and roadmap discussions
Common data mistakes
Both Rekha and Manosai were quick to admit that any PM can make mistakes with data. In fact, they themselves have fallen victim to some of the most common ones. Manosai said that “falling too in-love” with the culture of making data-driven decisions is one mistake he sees frequently. Sometimes, PMs end up using data as a crutch. And where there’s a lack of data (like when you’re dealing with a new product), it can be paralyzing. You simply can’t make the right call looking at numbers alone — you need qualitative feedback and human intuition to understand the full picture.
Another mistake Manosai has noticed is using data as a way to win arguments or end a difficult conversation. For example, the engineering team might turn to their analytics data and say “This is what the data says. It’s the truth.” But user research might say something completely different. Hopefully, the PM can help bridge the gap.
Rekha has seen many PMs make the mistake of using data as “one-time-thing.” For example, they might track a particular metric prior to launching a specific feature, then ignore it post-launch. Instead, she recommends evaluating success constantly and remembering that situations can change over time. Your measure of success might be different a month, a quarter, or a year from now.
Want to hear of Rekha and Manosai’s thoughts on what it means to be a data-driven PM today? You can view Manosai’s webinar here and Rekha’s here. We hope you enjoy the conversations.
