This week on Product Love, I sat down with Jon Noronha, Director of Product at Optimizely, the experimentation platform.
Not surprisingly, we talked a lot about product experimentation – this is not only what Jon does at work, it is really a product management philosophy he lives by. Jon started his career at Microsoft, working on Bing, where he was also really invested in experimenting, mostly in the form of A/B testing. He joined Optimizely when it was still very small, but it was really there that he started thinking about how product managers can bring experimentation into their daily practice. Everything you build should have built-in experiments, Jon says, make that part of your discipline.
Product experimentation lets you quantitatively measure the impact of the things you’re building, and it lets you iterate on a product after launch. Too often, we work too much on launching something, and then we move on because we’re all so indebted to roadmaps. But the fact of the matter is that launch is not the end of the road for a feature, product managers should worry about what Jon calls the last 5% of your work: your feature’s discoverability, adoptability, and onboarding. And those are the places where running experiments can drastically improve your KPIs.
We also talked about why experimentation is important in B2B companies, despite the smaller volume of web traffic B2B companies typically get. Jon points out that in B2B the cost of losing a conversion or a customer is much higher, so B2B companies are starting to come around to testing more things on their product, but also in their marketing and engineering teams.
“Most software engineering is a waste of time, this sounds like dramatics, but it’s the essence of why product management exists as a discipline.”
I asked Jon about this controversial statement of his, and he stood by it. Maybe a “waste of time,” is overstating it, but Jon points out that as a PM there are 100 good ideas that his team could be working on, and he means good ideas. But in any given month they can probably only pull off three or four of those. The opportunity cost is high – for everything you decide to build, you’re letting go of a lot of ideas. As product managers, our job is profoundly important: seeking out what matters and letting go of what’s not.
You might also like
