The Product Cloud Chapter Book

Introduction: A Cloud is Born

Disruption. It’s an often overwrought word to describe a normal change process. Yet in the recent history of digital products, it’s probably an apt description. In short order, we’ve seen the complexity of digital products increase, the massive growth of available product and user data, and a complete rethinking of how products are purchased and deployed.

Complexity has largely come from the addition of new screens. Digital products aren’t just designed for one device anymore. They have to work across an array of devices and retain context as users “multi-screen” between desktop, tablet, phone, and back again.

Data has become an enormous differentiator in digital products. The best products leverage specific user profiles, context, and behavior to deeply personalize the experience for every visit. To the end user this feels like the product “knows” them, and is adapting to their needs.

Software as a service (SaaS) has become so common that it’s basically the assumed deployment model at this point. The displacement of traditional software licensing and deployment has been swift and sudden. SaaS has made software adoption much easier, but it’s also dramatically reduced customer switching costs.

Taken together it’s easy to see how significantly the software industry has changed. This disruption is impacting how digital products are developed, and it’s giving rise to a whole new set of tools to support the product creation and improvement process. This e-book takes a deep look at how the new world of software is impacting product teams (product management, executive leadership, and user experience professionals), and the new software stack they’re leveraging to get jobs done.

The changing face of product management

No one has felt these shifts more than digital product teams. Product management today is a significantly different profession than it has been historically. The archetype of the turtleneck-clad visionary is giving way to much wonkier, data-driven experimenters. We see this in the way product decisions are being made, and how product teams are quantifying success.

A significant amount of the product management function revolves around decision making-deciding what to build, which features to prioritize, and which stakeholders to satisfy. In the past these decisions have been largely intuitive, or shaped by the most vocal stakeholder. Successful PMs were praised for their intuition and their ability to give customers what they didn’t know they wanted, rather than what they asked for.

Today, we’re seeing a shift to a much more data-driven approach. The growing availability of usage data coupled with broad user feedback is providing input into decision making, and justification for product direction. Questions like “How much is that feature actually used?” or “How many customers will get value from this?” are now a regular part of the discussion.

Often, success to the product team meant shipping product (or shipping product on schedule). The focus was on getting something out the door, with limited attention or follow-up once a product went live. This made some sense in the world of perpetual licenses, but not anymore. Today software isn’t sold once, it’s sold every month, quarter, or year over and over again as users renew their subscription. As a result, product teams are becoming much more focused on continuous delivery of user value.

Products and features that deliver progressively more user value are those considered successful. As a result, the approach to product and feature delivery is changing as well. New features and prototypes are often tested with small samples of users, or in variations, to quickly assess their value before shipping broadly.

The evolving product management toolset

The tools used by the product team have grown and evolved with the role. It wasn’t all that long ago that there were no purpose-built software tools that supported product management functions. Instead generic tools were leveraged-roadmaps were built in Powerpoint, feature backlogs in Excel, and everything from Sharepoint to Post-it notes on a whiteboard were used for project management.

Today, product teams have a much broader array of solutions that they can use. These tools are tailored to product creation functions, and as a result provide much greater utility. Quietly, a sizeable ecosystem of tools across a number of different categories have become available.

As with any nascent software ecosystem, there are a lot of point solutions, and half-defined categories. Are code repositories or issue tracking tools a relevant part of the PM stack? Possibly. As the new discipline of product management continues to evolve and harden the most relevant categories and tools will come into focus.

Towards an integrated product cloud

Integration naturally increases the value of any functional toolset as end-to-end workflows can be orchestrated across systems, with a common, shared data set. The tools supporting the product management role are starting to move towards a more integrated solution. This pattern mirrors what has happened in other functional stacks.

Marketing automation has taken cross-channel advertising and campaigns (along with analytics and attribution) and brought them together in integrated workflows for marketing teams. Large solution ecosystems that support sales activities live on top of, or leverage CRM data. The creation of each of these “clouds” has been driven by increasing complexity and change in the functional roles they support. And, each of these clouds has improved execution and efficiency.

As the product management role evolves, a product creation lifecycle built around data, feedback, iteration, and experimentation is taking hold. Each of the subsequent sections in the e-book will explore in-depth the key workflows that are part of this process. We’ll look at how they help product teams become more value-focused, and how specific tools and integration points are emerging to support them.

Chapter 1: Roadmapping

Authored By Jim Semick
Founder and Chief Strategist

The product roadmap isn’t just a document-it’s a powerful planning, communication, and organizational alignment tool.

• Roadmaps must be deeply informed by product strategy
• The roadmap communicates the product vision, not the feature backlog
• Flexibility and regular change are features (not bugs) of effective roadmaps

Develop a successful product strategy

“When developing a strategy that ultimately leads to a product roadmap, it’s important to identify and articulate the product’s vision and principles—the ‘why.’”

Product managers play a critical role in an organization’s strategic direction. They are at the intersection of many important feedback streams—customers, colleagues, executives, and other constituencies. They are also constantly pulled in different directions by market changes and other external factors. Product managers need to digest this continuous stream of (often conflicting) data wisely, and lay the foundation for the company’s product vision.

Product managers are faced with an ever increasing pace of product cycles. Innovation cycles and market shifts have drastically increased in the last decade and the pace of innovation will just continue to increase. Modern product managers have to adjust their roadmapping process accordingly. Product managers have to think long-term, while also remaining flexible and course correcting quickly (and often). Therefore it is important to implement an agile product management process so that product managers can spend most of their time on what matters most—their organization’s product strategy.

The most important part of the roadmapping process happens before the roadmap is built. Product teams must focus first on setting vision and strategic goals for the product—and, more importantly, getting alignment on these with various stakeholders.

Equally important at this pre-roadmap planning stage is developing a strategy that product teams can clearly articulate and defend. After all, buy-in from various stakeholders (including the executive team) is crucial for the success of the roadmap. That means that product teams need both a high-level vision and evidence or other supporting data to back up their plans.

When developing a strategy that ultimately leads to a product roadmap, it’s important to identify and articulate the product’s vision and principles—the “why.”

Product teams need to spend time before they begin planning the roadmap and determine the product’s mission, and then distill it into a simple statement that stakeholders can understand. This includes product vision, the problems it solves, its target customers, and its value to the marketplace. Documenting this forces the product manager to nail down many of the key items that will inform the roadmap.

The executive team needs to know (and agree with) the plan for the product’s development and updates—because they will ultimately need to sign off on those plans. The development teams need to know what the product team has planned for the product, and why, because they will be responsible for building it. The sales, service, and marketing teams will need to know the what and why as well—so they can articulate the strategy to the market the product.

This strategy-first approach has several benefits:

• It makes it easier to articulate the product vision to any constituency across the company, and ensure that the stakeholders are on the same page before the product manager begins the detailed conversations that follow.
• It makes it easier for the product manager to clearly see the product’s vision, and allows the entire product team throughout the roadmap process to more clearly identify priorities as well as those items that should be set aside because they don’t serve the product vision.

From the product vision, product managers can derive product goals that will in turn influence the initiatives that are on the roadmap. Coming up with product goals is the step that helps the product manager translate the product strategy into an executable plan.

Every organization’s product goals will be different. Teams can develop product-specific, company-oriented, or more generic goals. Here are some examples:

• Competitive differentiation
• Customer delight
• Technical improvements
• Sustain product features
• Improve customer satisfaction

• Increase lifetime value
• Upsell new services
• Reduce churn
• Expand geographically
• Mobile adoption

These goals are general, but can usually be measured and tied back to metrics and Key Performance Indicators (KPIs). It’s these types of goals that will resonate with the product’s stakeholders. Goals are often longer-term initiatives—for example, they might change annually rather than monthly.

Roadmaps communicate the product vision

In most agile product development organizations, the backlog defines the product features for the near term. From the backlog, the development team is (hopefully) aware of what’s coming next, at least for the next few sprints or iterations.

But the backlog in itself is not the roadmap—a product roadmap defines a strategic view of where the product is headed over the mid to long term. The roadmap is tied to the organization’s vision and strategic goals, often for the next 12 or more months. In an agile organization, the roadmap provides guidance rather than a strict project plan.

The roadmap needs to communicate the big picture to the organization—the initiatives that move the needle, expand markets, address competition, and create customer value. That big-picture thinking can’t be distilled in the backlog. It’s challenging to communicate strategy in a list that’s 200 items long, especially to executives and other stakeholders who might not think in terms of iterations or sprints.

Roadmaps are never static

“A roadmap should be agile and treated as a living document—not a fixed plan. Product teams should expect to regularly revisit, discuss, and re-prioritize the roadmap based on new inputs.”

A roadmap should be agile and treated as a living document—not a fixed plan. Product teams should expect to regularly revisit, discuss, and reprioritize the roadmap based on new inputs.

Because the roadmap will inevitably change, it’s important to set the expectation with stakeholders that the roadmap is not a promise. Many of our customers keep the roadmap dates at a monthly or quarterly level, or leave the dates off altogether to avoid setting the impression that features will be delivered by a specific date.

Product managers need to regularly communicate where the product is heading so that everyone is on the same page, especially stakeholders who make final decisions, control the budget, or influence the direction of the company. An agile product roadmap, therefore, should be a visual, easy-to-digest document that stakeholders can understand and that gives perspective to the backlog.

Chapter 2: Design and Prototyping

Authored By Billy Kiely
VP of Product Design

Leading companies are using an agile design approach to deliver customer experiences that are a generation leap over anything that has come before.

• Progressive ideation prevents premature “lock-in” early in the development cycle
• Internal and external feedback loops are both important to more comprehensive customer insights
• Operational maturity reduces design debt and allows for greater scale

“The web and mobile products with the best customer experiences happen by design.”

Not that long ago, design was a pursuit in “making it pretty.” Today, design thinking and practices permeate every step of bringing products to market. Companies like Airbnb, Facebook, Netflix, and Amazon have reshaped entire industries by employing design-forward practices that make them look almost clairvoyant as they release every new version of their product.

As disrupters set the tone, everyone from startups to established Fortune 500 enterprises are now building dynamic, inclusive, scalable design practices. While Agile defines a world of rapid iteration during the development phase, an increasing number of product teams are applying an Agile approach to the design phase too, where they can iterate faster and with even fewer constraints. It’s how the best companies in the world are delivering customer experiences that are a generational leap over anything that came before.

We can simplify design methodology in product teams as three distinct practices: validating the user experience early, collaborating on design cross-functionally, and operationalizing design for scale.

Validate the user experience early

“Investment is the enemy of creativity.”

There’s a temptation for product owners to jump in and start building the product idea they sketched out on a napkin with a customer over drinks. Not a great idea.

The second they start formalizing plans and resources to develop an MVP, they lock themselves into a path that limits their ability to apply divergent thinking. Additionally, the high mental overhead of a formal development project can easily shift thinking to developer resourcing, requirement specs, and sprint planning instead of capturing the opportunity to nail the customer experience up front.

Teams need to give themselves permission to work together on ideation exercises—like storyboarding, mind mapping, and Crazy Eights—and then flesh out ideas with simple low-fidelity prototypes that show how product concepts and screens flow together. What follows are more detailed prototypes that fully articulate the vision—they look, feel, and work in a way that is indistinguishable from the final vision of the product. Every click, tap, and swipe should be as intended.

As prototypes reach higher fidelity, it’s possible to add animations, microinteractions, and hover states—all the little nuances that make the product what it is. This is important for gathering more meaningful feedback and for getting designers and developers in full alignment so that the product that ships is the product that was designed.

All of these steps are critical to ensuring that a single line of code isn’t written until the product direction is validated and clearly articulated.

Collaborate on design cross-functionally

Innovative design is not a discrete service, and it doesn’t happen in a silo. It’s a highly collaborative culture of continuous prototyping and iteration, and it’s dependant on getting buy-in from as many diverse viewpoints as possible, as early as possible.

Diverse internal and external feedback has two major advantages. It takes into consideration more corner-cases of product use, creating a more comprehensive view of the customer, and it helps product teams ship the right product the first time and focus on iteration—not “back to the drawing board” repetition.

Bring customers into the fold. Include developers at every step. Share your plans with marketing and sales. Make sure legal is okay with your approach. Get executive buy-in early. Critical insight can appear from anywhere, at any moment, so look for it earlier. When everyone is aligned on the vision from the start, better products ship faster at the end.

“Great design, like great anything, is a result of inclusiveness and diversity.”

Operationalize design for scale

“Design debt is the silent killer of great products.”

Design debt happens when there is an overabundance of non-reusable and inconsistent styles and conventions—and the interest to pay on that debt is the impossible task of maintaining the mess. Over time, the accumulation of this debt becomes a great weight that slows growth.

Having operational maturity in a design practice reduces this debt and pays huge dividends across the product development lifecycle. It can double the speed of design and iteration and keep the customer experience from looking disjointed. One important step to achieving this level of maturity is the adoption of a design system.

A design system creates a single source of truth, a design language in which everyone is instantly fluent that can quickly and easily be pulled into designs, prototypes, and directly into code. It reduces design debt, accelerates the design process, and builds bridges between teams working in concert to bring products to life. Unfortunately, this is an area in which many product teams underinvest. That underinvestment often leads to systems that aren’t universally adopted and therefore have almost no value. Adoption is truly paramount.

It’s important to consider three audiences when operationalizing design and driving system adoption:

1. Designers—Does it operationalize design in the tools and workflow that designers already use and create a common language that’s easy to align around?
2. Developers—Does it make that design language easily accessible to developers through an API?
3. Other stakeholders—Does it showcase a single source of design truth that’s accessible to every facet of the organization (and even externally)?

A good design system will also gracefully separate these constituents through strong access control, versioning, and data protection in order to keep the integrity of your design language intact.

Designing with empathy

Today’s most disruptive and beloved digital products are created by design teams that consider the customer at every step of their process. They understand their customer’s problem, understand how others understand that problem, and then collaborate on a vision that solves it best.

When everyone is all in on the vision, every decision that follows is informed by a deeply rooted organizational empathy. Once that Why is embraced, their next great product is only a few Whats and Hows away.

Chapter 3: Feature Flagging and Experimentation

Authored By Trevor Stuart
Co-founder and Head of Product

Product teams now roll out updates as a series of running experiments with more and more users invited as functionality hardens and improves.

• Traditional software releases don’t work for digital products today
• Feature flag adoption is growing as companies look to quickly and safely roll out updates
• Best in class feature-flagging platforms tie feature flags to Key Performance Indicators (KPIs) turning them into powerful experimentation systems

Selectively roll out feature iterations and test variations across user segments

Engineering agility has increased by orders of magnitude every five years, almost like Moore’s law. Two decades ago, it took Microsoft two years to ship Windows XP. Since then, the industry norm is shipping software every six months, quarter, month, week, and now every day. The technologies enabling this revolution are well-known: cloud, continuous integration (“CI”), and continuous delivery (“CD”), to name a few. If the trend holds, in another five years, the average engineering team will be doing dozens of daily deploys.

Beyond engineering, agile development has reshaped product management, moving it away from waterfall releases to a faster cadence. Minimum viable features are shipped early and followed by a rapid iteration cycle based on continuous customer feedback.

“Products that were once released every few months are now released every few days.”

Death of the software release

As our software development processes have evolved, we’ve mostly said goodbye to the idea of defined product versions. Many modern product delivery teams are taking this a step further–even the concept of a “product release” is starting to fade. Instead our products are becoming a fluid, rapidly evolving set of features, assembled uniquely for any given user.

Twenty five years ago a new release of a software product was a notable event. This is still the case for a few types of software–operating systems being the most notable–but for most of the software products we use on a daily basis the concept of a version has faded away. What version of Google Maps are you using these days? How about Facebook? Twitter? As far as the end-user is concerned there is no version, just the “current.”

The rise of agile software methodologies and associated practices such as continuous delivery has allowed software teams to tighten up software delivery cycles dramatically. Products that were once released every few months are now released every few days. In parallel, the growing capabilities of the web as a platform has moved many products off of the desktop and into the browser. This has greatly reduced the friction involved in releasing a new version of a product.

Product management no longer must worry about broad sweeping release days–days controlled by engineering with late night release trains, and several on call engineers from across different functions. Instead, for many product delivery organizations, it has become the norm for any change to be managed by feature flags. Product changes are no longer “launched” or “released”—they are incrementally rolled out.

Necessary adoption of feature flags

If you’re not familiar with feature flags, think of it as a way to deploy a piece of code in production while restricting access to only a subset of users. This service is controlled dynamically outside of the code, not requiring engineering resources to make a change.

Feature flags have a widespread number of use cases across both the product and engineering teams. Most specifically, for product management, feature flags provide the ability to:

• Test in production. Feature flags allow teams to perform functional and performance tests directly on production with a subset of customers. This is a secure and performant way of understanding how a new feature will scale with customers.
• Safely roll out new functionality. By having a feature behind a flag, product managers can not only roll it out to subsets of customers, but also can remove it from all customers if it is causing problems to customer experience. This idea of “killing” a feature is better than having to depend on engineering to push an emergency fix or a code rollback.
• Take a measured approach to product management. Rolling out a change involves monitoring the impact of that change. As feature flagging has moved into the domain of product management, product managers have focused their measurement on higher-level business key performance indicators (KPIs) – active users, conversion rates, and business transactions per hour, etc.

Enabled by the feature flag and paired with the telemetry to measure the impact of these features on customer experience, is an emerging practice of product experimentation.

Rise of product experimentation

“It is through experimentation that product management organizations are able to truly measure the impact of their development efforts on customer experience.”

As companies struggle to understand the impact of product development on their customer experience, many have turned to online controlled experiments as the optimal way to rapidly deliver valuable software. These product experimentation platforms provide clarity and help product teams measure the impact of product development on the customer experience.

In an experiment enabled by the use of feature flags, users are randomly assigned to treatment and control groups. The treatment group is given access to a feature; the control is not. Product instrumentation captures metrics (or KPIs) for users and a statistical engine measures any difference in metrics between treatment and control to determine if the feature caused–not just correlated with–a change in the team’s metrics. The change in the team’s metrics, or those of an unrelated team, could be good or bad, intended or unintended. Armed with this data, product and engineering teams can continue the release to more users, iterate on its functionality, or scrap the idea. Thus, only the valuable ideas survive.

Experimentation is not a novel idea. Most popular consumer products–whether Google, Facebook, or Netflix–run experiments regularly. It is through experimentation that product management organizations are able to truly measure the impact of their development efforts on customer experience.

Chapter 4: Onboarding and User Guidance

Authored By Brian Crofts
Chief Product Officer

Today’s digital products need to actively attract, train, and retain their users.

• Rising user expectations and self-service products are driving a B2B onboarding focus
• Tailored onboarding, based on role or preferences, is a powerful personalization tool
• Onboarding isn’t just the first experience. Your product must constantly train users

The ‘Apple Effect’

There once was a time when delightful product experiences were largely contained to the world of consumer software. Here, products were expected to invite new users into their world with the elegance and grace of the concierge at a fine hotel, anticipating unarticulated needs and making you feel special and cared for in the process.

By contrast, the world of B2B software was something else entirely. Users were mostly left to their own devices to find their way through a hostile wilderness of product complexity.

Today, that’s all changed as B2C expectations have crept into all manner of software. Call it the Apple Effect. Every digital experience–at home or at work–is expected to be simple, elegant, and easy to use.

With SaaS now as the dominant software delivery model, customer time to value has become a pressing concern for digital product teams-especially in the enterprise. Whether the product is renewed every month, three months, or every year, there remains a finite window for meeting customer expectations, delivering value, and earning product loyalty and advocacy. This time is further constrained by any setup, deployment, and training that users must complete before a product can be fully used.

Onboarding is becoming a key product requirement

For B2B products, onboarding hasn’t always been part of product management’s responsibilities. It was typically owned by sales, customer success, or specific implementation teams, and much of it took place outside of the product. It was a checklist process that involved handholding through setup, and face-to-face or virtual user training via webinar or video.

The changing consumption and delivery model for digital products has made onboarding much more important. More and more of the customer journey is moving within the product itself. Prospective users are introduced to the product via digital promotion, then sign-up into trial and convert themselves to paying customers. In this model, an organization has no direct contact with prospective customers, and the product experience itself must take a strong hand in onboarding and educating users. User expectations are changing as well. Business users are increasingly expecting the same fast, friendly, experiences that they get with software at home. A delightful first-use experience is a must in the consumer world, and increasingly becoming so in the B2B world as well.

Product teams must now consider how to create product experiences that actively accelerate users’ proficiency, and ensure that users can reach value-creating features without running into roadblocks. Users today have little patience, and will abandon products that can’t be figured out quickly. Customer education needs to become an integral part of the product itself.

Although onboarding is a crucial step in the user journey, education isn’t just important for the first time a user is in a product. Products aren’t static. Product teams are continuously releasing updates, new features, and optimizing to the user experience. Each of the changes needs to be accompanied by the right level of education to ensure that users don’t lose proficiency as they adopt product updates.

Moving onboarding and education into the product

“Make onboarding content and flows a core component of the product experience.”

Many product teams are actively re-evaluating their approach to user onboarding. Rather than treating it as a distinct process outside of the product, they are working to make onboarding content and flows a core component of the product experience. With this approach, when a new user logs into the product or new features are released, relevant training content is pushed directly to the user.

In-app training content can take the form of announcements, embedded video tutorials, links to knowledge base articles or other help content, and step-by-step walkthroughs that take users through specific tasks in the product. Education in-app becomes much more contextual to the user and is more likely to be consumed. However, product teams need to be sure that the content they push into the product is relevant, and not unnecessarily cluttering the user experience.

Personalizing the onboarding experience

Most organizations have enough information and context about their users to tailor the onboarding experience in a way that makes the content as relevant as possible. Basic elements of the user profile such as role, plan level, or customer size can help to determine whether education content will be helpful to a specific user. User behavior and demonstrated proficiency can determine how much training they need when new features are shipped.

New tools make it easier for product teams to leverage this data as a part of their onboarding development. These tools pull user context through integrations with key customer systems such as CRM and marketing automation. They use product usage data to target content based on user behavior. With point and click authoring, they allow product, UX, and customer education managers to create and push personalized training content into the product experience without having to rely on engineering resources.

In-app onboarding increases user engagement

“Increased engagement with onboarding content leads to more proficient users who realize product value at a much quicker pace.”

Insightly is a CRM system focused on the small to medium business market. With a large, mostly self-service customer base they do not have the capacity or resources to manually onboard new users into their system. To help streamline user onboarding, they designed a simple flow to ensure relevant training content was placed directly in front of new users when they come into the product.

First-time users coming into the Insightly platform are presented with a custom modal at the top of the application that directs them to several training videos as well as walkthroughs for common setup features. Before adding this experience, the training videos were only available within a separate help section of the application, and as a result were rarely used. By shifting the help content directly into the product experience for new users, they increased viewership by 1540%, and the average view duration by 40%.

Increased engagement with onboarding content leads to more proficient users who realize product value at a much quicker pace. By leveraging data to power an in-app onboarding and user training experience, product teams can increase customer retention rates, and reduce reliance on customer success and support interventions to make users successful in the product.

Chapter 5: Usage Analytics

Authored By Brian Crofts
Chief Product Officer

We’ve moved from paucity to abundance when it comes to product data. The challenge is to effectively capture and harness it.

• Engineering resources are often a bottleneck when it comes to accessing product data
• Even small measurements can have a significant improvement on user experience
• Usage data isn’t just for the product team, it provides value throughout the organization

Product ‘Math Men’

The speed that product management has moved from data-ignorant to data-informed to data-driven is almost dizzying. What’s driving this rapid change? Simple. It’s the availability of product usage data. There’s a corollary to the ways in which the marketing profession has changed. As more and more data from digital campaigns become available marketers made the proverbial shift from “Mad Men to Math Men”.

Marketing “Math Men” have a much deeper understanding of the effectiveness of their campaigns. Product “Math Men” have a deeper understanding of how their products deliver user value.

User value is delivered one feature at a time. Usage data helps product teams understand which features are used and by whom, which features can be retired, and which ones simply get in the way. Data can help product teams uncover how users navigate through a product, and where they succeed and fail. Data helps product teams drill into specific user segments to understand adoption patterns.

The shift to SaaS has made usage data like this much more readily available. However, accessing it has until recently remained a challenge.

The big analytics tradeoff

By virtue of being hosted rather than installed locally at a customer’s site, SaaS products are a trove of valuable information. Basic information about usage levels, logins, and customer activity are part of the product infrastructure and can be extracted for analysis.

With largely web-based front ends, SaaS products can be instrumented with web tracking tools allowing product teams to augment basic info with additional behavioral data. The problem with accessing data from these sources is that both require (often a lot of) engineering resources.

Accessing internal product data requires engineering teams run custom queries and build reports for anything the product team wants to measure. Web analytics tools generally don’t track anything beyond page views without development teams instrumenting specific events within the product UI.

The reliance on engineering resources forces the product team to constantly make tradeoffs. “Should we pull engineers off of feature development in order to get more data, or forge ahead to deliver more quickly?” It also delays the delivery of information. If a new feature isn’t instrumented for launch, it can sometimes take weeks or even months before engineering has the bandwidth to revisit, leaving product teams in the dark about adoption metrics.

Towards a new model of product analytics

To help address these challenges, a new class of analytics tools are emerging to focus specifically on the data needs of digital product teams. These tools aim to remove dependencies on engineering teams by simplifying and automating data collection, and eliminating code-based definitions for individual element tracking.

These tools calculate and surface metrics designed to answer common product management questions-showing detailed feature usage, comparative usage across features or cohorts, and reports that track the flow of users through a product and the efficiency with which they complete tasks.

Sharing product data across “clouds”

“As usage data becomes more readily accessible, it provides not only a foundation to the product cloud, but direct value to other functional clouds too.”

The value of product data isn’t just realized by the product team. As usage data becomes more readily accessible, it provides not only a foundation to the product cloud, but direct value to other functional clouds too. Usage data helps marketers determine which behaviors of trial or freemium customers are associated with conversion. It shows customer success teams which accounts are highly active, and which are at risk of potential churn.

Integrating usage data with other departmental systems provides these valuable insights, and helps align the organization around product objectives. As visibility into product activity grows, individual teams can begin to see how product metrics influence broader business. This, in turn, helps the product team grow influence with other stakeholders.

Small measurements, big results

Even small measurements, when leveraged, can have a significant impact on the product experience. Workwave, a SaaS platform for field service companies, recently began measuring detailed feature usage across highly trafficked areas of their platform. The data revealed some interesting behavior patterns. For example, they found that users would use the search feature over and over again-as many as six or seven times in a row. This behavior indicated that the search function wasn’t working well for customers.

Using this information, the product team prioritized updates to the search feature. They redesigned the search function, adding a quick search feature that gave predictive results as a user types in the search bar. The re-designed feature reduced the average number of searches in a session down to two-a 70% increase in user efficiency. Clear information about product usage helped drive decisions that markedly improved the product experience. This is data-driven product management.

Chapter 6: Session Replay

Authored By Justin Dilley
Head of Product

There is no substitute for direct user observation: To fully understand the user’s experience, product teams must be able to drill into data, see, and understand the realities of user experiences.

• Session replay can help pinpoint and address product usability issues
• Recorded sessions provide easy scale to user testing and feedback programs
• Dig into trends or anomalies in usage data to find “root cause” behaviors

The devil is in the details

Having the right metrics—quantitative analytics you can turn to and trust—is essential to measuring product success, identifying opportunities, and prioritizing efforts. Things like feature adoption, daily active users, session duration, NPS, churn, and countless other KPIs all work to focus our attention.

But the devil is in the details. While we rely on aggregated insights to guide our decisions, grounding abstract metrics to individual user behaviors can be challenging, time-consuming, and error-prone. As Ronald Coase, British economist and Nobel Prize recipient, famously quipped, “If you torture the data long enough, it must confess.”

Must it be so hard to connect our quantitative analytics to the realities on the ground?

“When it comes to product usage, ground truth is the tactile, tangible reality of user adoption of your product.”

Down to the ground truth

Ground truth refers to information gathered through direct observation. It’s the information you collect through unfiltered, untampered research. It’s the kind of data you get through qualitative insights.

When it comes to product usage, ground truth is the tactile, tangible reality of user adoption of your product. And that makes it powerful.

In order to collect this kind of information, product managers, designers, and UX researchers have defaulted to commissioning user testers, interviewing specific, cherry-picked users, or, in some cases, tapping hapless co-workers for help:

• “Could you play with this new feature?”
• “I want you to try this out. Follow these steps while I observe …”
• Or, “What did you think about the new feature?”

Other approaches include surveys or feedback forms. No matter the approach, getting a qualitative pulse on user behavior has historically been error-prone, highly anecdotal, and terribly imprecise. And because user feedback and testing is so expensive, it’s typically limited in scope to a small sample.

Qualitative insights on tap

“By watching actual users engage, hesitate, click, tap, and scroll, the abstract metric becomes real.”

Advances in product analytics technology have made it possible to collect qualitative data about real user behavior at scale, with incredible precision, and without the bias. Using session replay technology, teams can capture the entire on-screen experiences of real users.

Using session replay technology, a designer can play back all user sessions in which a specific modal on a specific page was clicked. By watching actual users engage, hesitate, click, tap, and scroll, the abstract metric becomes real. A product manager can see when a feature is being used—or overlooked.

Replay technology grounds high-level quantitative analytics to real user experiences.

Play it again, Sam

How is replay helping teams ground analytics to user behavior? Popular business video platform Wistia noticed users couldn’t figure out how to upgrade to a paid offering despite the presence of a prominent blue banner pointing the way. Before replay technology, the Wistia product team had to brainstorm hypotheses and build tests based purely on hunches about what was amiss. By replaying session recordings showing users arriving at the subscription page with the intent to upgrade but ultimately failing, Wistia could observe the friction experienced by customers directly. With this critical qualitative research in hand, the team created a new, faster upgrade channel that quickly outperformed the old approach.

Relatively quickly Wistia found that 60% of their upgrades were coming from their newly created conversion channel—meaning that the majority of their upgrades were coming from changes informed by session replay.

In another instance, TravelPerk used session replay to troubleshoot when a customer left a low NPS score. The team “went to the tape” and replayed the session that led to the NPS score. They watched in horror as the frustrated customer attempted to fill out a form unsuccessfully eight times. Further research revealed that more than a dozen other customers had tripped over the same bug. A quick Jira ticket later and the critical issue was solved.

Replay to the future

In times past, teams had no quick way to understand these nuanced product problems. Now, replay technology is always working in the background, collecting real user experience information that can be researched on demand, whenever required.

Through connecting replay to existing quantitative analytics it’s never been easier for product managers to get the micro-answer to every macroanalytical problem they encounter, which means better products and better customer experiences.

Chapter 7: User Feedback and Sentiment

Authored By Richard White
CEO and Co-Founder

As companies grow, product teams must effectively scale, manage, and organize customer feedback.

• There’s no single perfect feedback method, most teams blend multiple surveys
• Team (and company) access to feedback is critical for successful feedback programs
• Effective feedback programs directly increase customer engagement and satisfaction

Getting beyond 1:1 interviews

When it comes to product feedback, nothing has more credibility to product managers than 1:1 interviews with customers or prospects. This is true whether they’re trying to get insights that inform new capabilities or insights that refine existing ones.

PMs in very early stage companies are especially familiar with the technique, as 1:1 interviews are generally the only source of customer insights they have. Most products (including UserVoice) are born out of a handful of early discussions with the right people.

“Agile development moves fast, and product priorities can shift rapidly, so there is always a higher need for feedback than any team has bandwidth to solicit.”

But for the SaaS PM, particularly those in companies that are actively scaling, relying on 1:1 interviews for their entire feedback diet just isn’t realistic. This goes beyond some of the more obvious issues-they’re time-consuming to set up and administer, data gets stale, it’s hard to share learnings across the product team-and gets to the heart of what it means to be a modern PM.

Agile development moves fast, and product priorities can shift rapidly, so there is always a higher need for feedback than any team has bandwidth to solicit. PMs, especially of freemium or self-service products, have never been more outnumbered by customers, and they’re constantly trying to serve different constituencies in ever more heterogeneous customer bases.

Gaps with existing feedback methods

Thus modern SaaS PMs want to find other sources of feedback beyond 1:1 interviews. Many PMs are gaining a solid understanding of customer needs by deploying a variety of feedback solutions:

1. In-app feedback collection can ask specific, focused questions to customers and can experience higher participation than email since customers are being solicited while engaged with the product.
2. Survey tools can capture more in-depth learnings from customers and, if written well, can enable a product manager to cut the resulting data in many ways to not only prove or disprove their hypothesis but also discover new insights.
3. NPS provides a shorthand for how much customers like the product. Score fluctuations suggest when it’s time to take a closer look at core functionality.
4. Customer Advisory Boards nurture long-term relationships with champions of your product and will work in partnership with your product team to design the future of your product
5. Spreadsheets and other ad-hoc tools allow internal teams like sales or success to collaborate on creating a reference of the product feedback they regularly hear.

These services can support each other to craft a deeper understanding of customer sentiment and needs – for example, a PM can use the insights from in-app feedback to craft a survey that asks the most pressing questions about the product, directed to groups seen as NPS detractors.

It’s the right mentality-committing to take feedback seriously as a means to improve the product already puts the product manager in a fairly elite group. However, there are a few issues to acknowledge:

• Internal teams have good instincts, but it’s hard for the product team to make sense of “lots of people want ___” without knowing who asked for it and exactly what they said.
• Users may not give enough feedback because they’re unaware of how much it’s wanted or how it affects product decisions.
• Teams can make it too hard (“Do you have time for a 10 minute survey?”) and/or users can’t see an obvious value (“What do I get out of filling this out?”).

Even if PMs do manage to overcome these challenges, they often have a data aggregation problem. The product team struggles to keep multiple spreadsheets of feedback in sync or has to get creative to find actionable insights in thousands of pieces of raw feedback.

The next generation of feedback

To tackle these problems, some PMs have created a single repository for all qualitative feedback, whether its an NPS survey, feedback about a beta feature, or a product gap. Ideally, this single repository:

• Is an always-on solution for customers or customer team members to enter feedback
• Allows teams to map feedback to specific product areas, product gaps or roadmap items
• Makes it easy to close the loop with the feedback originator and internal team when there are follow up questions or status changes

There are a few different ways to achieve a single-source feedback database-in Salesforce, via Airtable, or with UserVoice-and the results can solve many of the issues mentioned earlier. For one, product managers can reduce and automate low-value work-one survey found that PMs without a feedback database spend about 20-25% of their time just organizing feedback from inside and outside the organization.

Having such a system gives a PM a comprehensive database of feedback that is representative and credible. With a disorganized feedback mix, companies struggle to consume feedback from a small portion of their customers. With this approach, companies have the potential to map feedback from 50% of customers or more.

This feedback data is useful throughout the entire product development process:

• When investigating problems for the roadmap, the team can easily see and understand which problems most affect specific customer segments
• Ideas generated by management can be quickly validated (or invalidated) via existing feedback
• Once mockups are created, teams have a readily-available database of existing interest to call on for testing

Product teams are also able to create a meaningful analysis of the feedback in aggregate, to answer questions like ‘what are we hearing most from Customer 1?’ or ‘what is the potential revenue impact of choosing feature A over feature B?’. These insights are connected to raw feedback and data from corresponding customers, enabling the product team to dig in and validate potential solutions quickly.

More engaged companies create more engaged customers

Finally (and maybe most importantly), PMs can develop more customer evangelists. In our experience, people are massively impressed when companies close the loop on product feedback, even if it’s months later and the product team has decided not to act on it. Studies have found that only 5% of companies reliably respond to customer feedback. Don’t discount how valuable it is to exceed these decidedly-low expectations.

Moving beyond 1:1 interviews doesn’t mean losing touch with customers. By building a comprehensive feedback system, companies like PracticeFusion, MuddyBoots, and Microsoft have ensured their product organizations are investing in the right features while building stronger internal buy-in and customer understanding.

Conclusion: Leveraging a Product Cloud

“Tools alone won’t magically produce better products, but they can help facilitate and enforce new motions across the product team, and the broader organization.”

Why a product cloud? Functional clouds are the natural outcome of the intersection of software, marketplace changes, and shifting or emerging job roles. A product cloud is coalescing around a set of specific changes taking place for digital product teams. Rather than “why?”, a more important question to ask is “how can or should we leverage the cloud?”

Each of the solution categories described in this e-book focuses on a key stage of the product creation lifecycle. Tools within each category can streamline workflows and help to improve the effectiveness of the product teams and the digital products they deliver. An initial starting point when thinking about how to harness the product cloud is to target areas of the product creation lifecycle for improvement, then identify areas where new tools may be able to help. Tools alone won’t magically produce better products, but they can help facilitate and enforce new motions across the product team, and the broader organization.

Data is the foundation

The common thread across all the parts of the product cloud is data. Data fuels product strategy and planning. It helps to determine target segments for product experiments. It details product adoption (or lack thereof), and how the product experience influences sentiment and customer satisfaction. Whether it’s customer feedback collection, user observation, or aggregated usage analytics, there is no way to effectively leverage a product cloud without a concrete product data strategy.

The power of a functional cloud grows through integration. Although much of the product cloud today exists as slightly separate categories of solutions, over time, these solutions will coalesce and become more tightly integrated. The layer of product data which today is important from a process perspective will become the engine that aligns the product cloud around a common objective: the creation of digital products that users cannot live without.