Not too long ago, I was lucky enough to be invited to a “soft opening” at a local Italian restaurant. It was my first time at this type of event, and I was extremely excited. I could order whatever I wanted: appetizer, entrees, drinks, all to help the restaurant test their readiness for their grand opening, and I wouldn’t have to pay anything. My only obligation was to complete a survey at the end of the dinner. The service wasn’t perfect, the coffee at the end of the meal was terrible, but overall I was happy and it was a positive experience. The restaurant had set the proper expectations and the positives — free food and drink — outweighed any negatives I experienced.
As product managers, we often run the same process with our customers in beta groups or early access cohorts. These are vehicles to get feedback and real-world experience with our products before we open them up to a larger audience. They allow us to iteratively develop products and avoid the pitfalls of a waterfall-driven approach.
While this seems like a simple proposition, in reality, it can be difficult to execute an early-access program that is beneficial to both your company and customers. Success hinges on getting the right set of properly motivated participants, setting clear expectations for the experience and engendering useful feedback and insights.
While early-access programs can take on different forms depending on the scale and objective, I will focus here on high-touch programs involving a small set of participants. For these, I don’t like the terms beta or early access. I prefer to use “Advisory Program,” which better implies the collaborative partnership between product and participant that is critical to success, instead of just describing the state of the software.
Mise en Place
Every good meal starts with having the proper ingredients, or in this case, the right participants.
For a recent program, we were looking for 10-15 users representing a cross-section of our enterprise customers. One mistake we’ve made in the past is being too reliant on the business to nominate participants. While this can be the quickest path to the customer, it can also result in several types of less-than-optimal participants.
The first suboptimal participant is what I would call the “manager” who is the main point of contact for the business. Unfortunately, they are usually not the primary user and thus the feedback they provide can be very limited. They can recruit others that are more active, but their influence over the participants can end up biasing the participation.
The second is the “grumpy customer.” This is a customer that has some level of discontent with the product or service and is nominated with the idea that giving them “new stuff” will make them happier. This is a redirection technique that works successfully with toddlers, but unless the user’s existing issue is going to be directly addressed by the new functionality, it will be very difficult to keep their focus.
The last group is what I like to call the “hand raiser.” They are everyone’s favorite customer. They are very positive, uplifting people that love the product and like to volunteer for everything. The problem with these folks is that they are often too friendly to give critical feedback, and they are early-adopters that are much less risk-averse than the typical user.
What we have found works best to recruit participants, is to start with the usage data and identify users that have been active in the areas related to the new product or feature. Once we have a list of potential candidates we work with the business, usually, the Customer Success team, to further vet them and also ensure that the company is a good fit from the account perspective. We then either reach out directly to them through email and in-product notifications or have our Customer Success team arrange the meetings.
Get Cooking
After you’ve successfully avoided the three types, and managed to build a core group, the next step is to get the participants involved. Setting the correct expectations will be critical.
On our first touchpoint with the participant, we go through how we will work with them in detail and what they can expect. This includes giving them a timeline, communication plan, and a general understanding of delivery cadence. If we want to keep them involved, they will have to feel like the reward is worth the effort. This can be challenging, as the benefit of the new product may be minimal at first.
You will likely need to provide value to them in other ways such as sharing ideas, best practices, and most importantly being a good listener as they share the intricacies and challenges of their day-to-day work. It is always difficult to predict exactly when things will be ready, but as long as you communicate frequently and are as transparent as possible, participants will be reasonably patient.
Thinking back to a restaurant soft opening, there is nothing worse than waiting for the dinner to arrive, but if you know what is going on and perhaps have a few drinks while you wait it makes everything better.
How Was Everything?
Finally, the advisory program is worthless unless you can obtain the feedback and information that you are looking for. You should start by establishing the key assumptions or hypotheses that you want to test and then adjust them over time as you learn more about problems you are trying to solve. We always try to test each hypothesis in different ways, through discussions, prototypes, and monitoring usage of new features. Don’t just ask the participants if they liked the food because the answer is almost always yes.
An advisory program is a powerful mechanism to help you build great products. While it can be a lot of work, it can uncover key insights and build lasting bonds with customers. My wife and I are now regulars at that new Italian restaurant. While they never took my advice on their coffee, it’s still one of my favorites. You likewise probably won’t be able to hit every piece of feedback your advisory council gives, but if you do enough, they’ll be delighted.
