I’d like to share a (hypothetical) story about a product dashboard to illustrate the need to differentiate the noise of product usage data from the signal.
Once, in the world of B2B SaaS, there was a product with a dashboard full of information and charts to help small home repair businesses manage their maintenance and repair work. Users could access all kinds of information—work orders pending, average time to close an order, total billed amounts, and more—that could, in theory, help them do their job more effectively.
From the product metrics, the developers could see that customers were loading the dashboard repeatedly and spending a lot of time on the dashboard. Time-on-page and number of page loads were both very high. From this, they were tempted to conclude that their customers were deriving a lot of value from the dashboard. But were they?
It was time to dig deeper. In this case, one of the charts on the dashboard indicated the dollar amount for past-due work orders (with red column shading). Clicking inside the column shading would bring up an overlay showing the past-due orders. Catching past-due orders and keeping up with collections was very important to the company as a whole and to the individual users, who would be reprimanded if a work order went unpaid.
Path analysis showed that users would visit the dashboard frequently to check the chart and click on the red portion when it appeared. Most work orders never went past due, so although 85% of user sessions included the dashboard page, only 8% included clicking on the red column. In those cases, though, most of the people who viewed the overlay quickly navigated to another page to take action. Use of that past-due overlay was infrequent, but very important to the customers when it did happen.
Was the dashboard really doing its job? Well, sort of. From the overall metrics, the developers might think that there was a lot of user engagement with the dashboard. But most of the actual use of the dashboard was mundane and localized. The users had to catch the overdue payments for work orders, and this was the place they had to check. They weren’t very involved with the dashboard and ignored many of its features. If they were unhappy with it, they might not bother to complain. If they were to give feedback, it would probably be vague and not very descriptive. Not many users were thinking “I’m addicted to this dashboard, it’s so helpful in so many ways!”
“When it comes to measuring user success, less may be more.”
This story shows that it is vital to get past typical session-level metrics and identify the value truly exchanged between your product and customer. In this case, the primary value was the ability to identify past-due accounts and collect money for the company. Investigating that path could lead to new directions for improving user experience and engagement. For example, when are the past-due indicators generated? Could that information be tracked in more detail and notifications issued sooner? Could the process for making payment be easier? Could the action the users were taking when the account was past due be automated?
In this case, the best way to measure whether users are truly getting value from the product might be the average time to close work orders, or the total time it takes the user to follow up on a work order. Metrics like page views and time-on-site didn’t capture how effectively the users were achieving their goals. When it comes to measuring user success, less may be more.
Generalized measures of product use are seductively straightforward and easy to explain, and, in some cases, they can provide insights. But you always have to look past them to see where the real value to the user lies in the product, and question whether evidence really indicates that users are achieving their goals.
John Cutler is a product management and UX consultant. His passions are UX research, evidence-driven product development, and empowering the front line to solve business and customer problems. For more of John’s writing visit his Medium profile or follow him on Twitter. He is honored to team up with longtime friend and editor Katherine Maurer, a freelance editor and poet whose work has appeared in many pretty good literary journals. She is also a graduate student in clinical psychology, and drummer in the band Again is Already.
