Are you hiring product managers using the cookie cutter approach? Large companies might think they have a general profile to fulfill, whereas small companies might be searching for their perfect fit product manager. In reality, Jorge Mazal, the director of product at Duolingo, thinks expectations should be set in the middle. Each hire still needs to be crafted with some basic competencies. When I sat down with Jorge last year, we discussed what qualities we look for in product managers, and how Duolingo uses psychology to keep their users engaged.
Jorge has his own approach when it comes to hiring PMs. After identifying problems — or opportunities for change in his organization — he compiles a list of skills that PMs would need in order to solve those problems. He has a list full of eight qualities that have served him well when it comes to scaling his team.
You’ve probably used Duolingo – it’s one of the best language learning apps on the market. As a product leader there, psychology is extremely important. Jorge uses a lot of learning and behavioral science to ensure that people are actually acquiring new knowledge through the app, but are also returning to increase their usage. To Jorge, it’s obvious that a large part of product management is understanding your users so psychology. There are many methods that help users understand their own usage and eventual mastery of your product, as well as psychology in the messaging targeted at them.
Want to know the eight skills that Jorge looks for every candidate? Or see how Duolingo retains their users?
I am now happy to share a lightly edited transcript of our conversation. Whether you prefer audio, or love reading, I hope you enjoy it!
You can check out the original post here and stream the audio version here or subscribe on iTunes today.
Eric Boduch: Well, welcome lovers of product. I’m here today, I had Duolingo with Jorge Mazal. Why don’t we kick this off Jorge by starting with a little overview of your background?
Jorge Mazal: Sure. Thanks for having me. Hi, everyone out there. I’m Jorge Mazal, I’m the director of product here at Duolingo. I’ve done a variety of product management roles before coming here. I was in gaming, I was in health and fitness, productivity apps. Before all of that, I was in nonprofit, in the education sector. When this opportunity came along, I felt like it brought everything together. So, I started to pack my bags and move across the country here to Pittsburgh and have loved it ever since.
EB: Tell us a little for those who don’t know about Duolingo.
JM: Duolingo is the largest education app in the world in terms of installs. I have over 200 million registered users. We are focused on language education. So you can take one of more than 80 courses for free. Spanish, English, whatever. Most of our users are outside the US learning English. But we do have a very large user base in the US as well.
EB: I know my wife used to touch up her French before we went to France on a trip. So, cool.
JM: Yes. It’s a very common use case.
EB: You’ve had a lot of experiences building product teams both here at Duolingo and elsewhere. Can you talk to us about what you’ve learned from your experiences?
JM: Sure. My experiences are probably quite specific to the company size that I have been at. It’s this 100 to 200 size company where product teams are actually fairly new or still evolving. People know they need strong product management and CEOs can’t be involved in everything anymore. But there’s still this kind of growing pains in the organization.
There are a few things that I have found to be really helpful to have in mind. One is that you can’t take a cookie cutter approach to hiring product managers. Those very large companies, I think they have a profile, very small companies they try to find the perfect match for that first PM. This is something in between. You still have to craft each hire. You have a basic baseline of competency you could say. But the needs are still so diverse that you have to have this one at a time picking of each PM. At least that’s what has been my experience.
The way I go about that has been trying to identify in an organization the tasks that are being executed, the problems that are being dealt with and the opportunities that need to be captured. And then from there create a list of skills that I need PMs to have to perform against those tasks, problems, and opportunities. I feel like I’ve come up with a fairly good list that I use when I try to hire PMs. There’s I think about eight of them. One is leadership, another one is strategic thinking, analytics, or analytical ability, the science [inaudible 00:03:00] technical know-how, creativity, user-centeredness and subject matter expertise.
For us, that’s education, that’s learning, that’s language, subscriptions is our subject matter topics that are important for us. And then what I do is that I try to evaluate each PM against each of those areas. What I find is that not every applicant or not any applicant actually meets all of them. It’s about crafting the right team. It’s like a patchwork and you find people that are really good at some things and really good at other things. But as a team, as a product management team, we cover all those areas fairly well, and then we help each other out. That’s getting the right people on the bus is the first thing to build a great product team.
And then it’s about how to make them work together. What I find there is that one of the key things is that just as important as results for a product organization is learning and that really effective product organization are really good at learning and teaching one another. So, everyone learns from everyone’s mistakes and wins.
To do that, I have found that experiment write-ups are obvious things that you need to do. But often those things are kept within teams or just management that they don’t get distributed very broadly. So, what we do is that we have a listserv, and we send those out to everyone. Something that I have found to be helpful is that so often, every couple of months or three months a team should go through all these permanent write-ups that they’ve done and actually do some sort of like almost like a literature review. Where scientists do this where they write papers about other people’s papers. They don’t actually do all the research, and that’s really helpful to see what the preponderance of evidence is.
I think product teams choose the same thing. You go through the last 30, 40 experiments you’ve run, you review all of them and actually do some sort of what have we learned from the last three months of 50 or 60 experiments we run? That’s been super helpful to do that. And then the other component of this is focusing an organization into being really, really user-centered. I’ve been talking for a long time.
EB: No, I think that’s good. Continue on that line. There are two areas I’d love to dig into a little bit more. Let’s start with the list of skills in your building this patchwork. So, talk to me about how you assess someone in each of those different areas. Do you have a methodology for doing it?
JM: Yeah. That goes into the hiring process. And the hiring process for most people it has three phases. There’s some screening, there’s some sort of task and there’s an onsite interview. I have an approach to each of those things. The screening I think, is actually fairly standard, but the task, often the task seemed, for a lot of companies I’ve seen is fairly targeted to just one of those areas as opposed to many of them. So, I crafted a [inaudible 00:06:09] applicants that tries to cover as many of those areas as possible in some shallow area to see if there’s anything that they’re terribly bad at from the get go. Like they’re really bad at data, or they’re really bad at design or something.
And then when they actually come on site, they have set up an interview four or five of those attributes. And the people interviewing are laser focused on that. So, it’s an analytics interview or a design interview or a leadership interview, or a strategy interview. Then when we all get together, everyone understands that their experience just represents one facet of this person and not the whole person. So, not any person can veto or hire someone. Unless they did something terrible on a personal level. But from a skill set, they can say, “Well, this person is not good at strategy.” Or they can’t say this person is not good because it’s just look at that.
So, for analytics, we do some sort of funnel interview. For design, there’s the design challenge that’s fairly self contained and I think quite accessible for people who don’t have a design background, but shows the user centeredness of someone. And then for strategy, we do like a strategy case. Similar as if you were interviewing for a McKinsey or Bain that’s just very similar to that. And then for leadership is really about team fed, it’s about leadership stories, behavioral stories, and things like that.
EB: Well, that sounds really well thought out. I like it. I know when I’m interviewing, I’ll often pick five areas or skills. And I’m like, well, stack rank yourself because no one ever wants to be weak at something. So, here’s five things, stack, rank yourself in there. And then it’s a hard exercise, I think, for a lot of people to really think about how do I want to position this and then you can dig in later and see how honest they were about their stack ranking.
I could see this being as a next step in that stack ranking approach I do. Something I think I’d love to take advantage of. I like the framework you use there. Another thing you talked about was experimentation? Can you talk to us a little bit more about how you integrate experimentation into your work in your organization?
JM: Sure. My first PM job was at Zynga. Zynga is a company that is extremely data-driven, and maybe a little bit to a fault. The only things that matter for them are the things they can measure which is problematic, but measuring all you can is great, right? So, every single change, every single concept we tried to think of how can we A/B test this and use that as a way to prove hypotheses? There’s a difference between proving a hypotheses versus using A/B testing to prove each other wrong. In our organizations, it’s like well, you think this way, I think that way, we just do an A/B test and see who wins. That’s not the best way to do A/B testing, but whether we believe about the user, whether we believe about the experiences we want to create, and does this test actually prove or disprove that broader concept?
EB: Awesome. You mentioned Zynga. For those who don’t know, Zynga is a mobile gaming company that has quite a few really awesome games. Huge company back in the day. Think they’re still doing really well, right.
JM: Yeah, they’re doing good.
EB: How was your work at Zynga affected impacted your approach at Duolingo? You mentioned one area, right?
JM: Yeah. From Zynga, I think I learned things to do and things to not do. I hinted at that earlier, where you want to measure everything, but you also want to consider things that you can’t measure. That’s the area where Zynga falter a little bit and that’s a takeaway that I have taken it very seriously in my career ever since. To care about the things like fun and relationship and brand and things that are hard to capture.
That’s one area. One form that that exists here at Duolingo is referring to our mission. We’re a very mission driven company, we care about the social impact that we have. So, we often bring up when there’s tough decisions to be made, we bring up, okay, what’s more aligned with our mission? That helps us stay focused and something see beyond what the data says.
EB: So, talk to me about that a little bit? Reiterate the mission for Duolingo and how that impacts your decision making?
JM: Yeah. It’s interesting. I don’t think we have formally articulated our mission, but somehow we all understand it. That mission is that we want to provide the aspiration I would say. We want to provide excellent language education for free to the whole world. The reason why we want to do that is because we believe that’s two things. One, it empowers individuals to access economic opportunities, especially people learning English, and two, it generates greater understanding of different people and different cultures, which we believe helps just the world get along better.
Those two things, understanding and [inaudible 00:11:08] feel like can help us be a better world. And we believe in that.
EB: Yeah, that’s very interesting. I think, my experience with Pendo, we’re very core values focused. One of our core values is a maniacal focus on a customer. We’ll often have decisions like, is this good for our customers? We’re built to make product teams lives better, right? We think about this, if we did this, is it great for product teams? Is it making their lives better? Si it improving their life in some specific way? Is it helping our customers?
I can definitely see that there’s a lot of access in North Star having a mission and a series of core values that can be used to frame your decision making.
JM: Yeah, absolutely. Super helpful.
EB: One other thing you talked about when we were chatting earlier, is you’ve done a lot of work about building user frameworks and shared vocabulary about users. Can you take us through that?
JM: Sure. I believe that that’s the job of a PM and of a product organization. That’s probably unique. A lot of companies, they distribute that differently. It could be marketing, it could be design, but I really want PMs to own that because they often own business outcomes. If they own the user experience, and the business outcomes at the same time, I feel like they balance those things well.
What we do is try to make interactions with users and user research, a regular part of PM’s work habits, not just something you do every once in a while, but can you set up recurring interviews every Friday? Or look and send us that data all the time and so forth. Once you have this constant flow of qualitative user data, how do you get those insights, distill those insights for yourself? But more importantly, how do you distill those insights for the entire organization?
Not everyone can spend hours talking to customers. So they need those insights to be summarized and synthesized into frameworks. A framework could be something as simple as a one page table or some sort of user journey or personas or story mapping and so forth. There’s a variety of them, a hierarchy of needs, et cetera. What that does is it helps crystallize your own understanding, but it also gets the entire company and the entire team aligned on what the users’ needs are, which then when it comes time to brainstorming and coming up with solutions, then the whole debate, the whole discussion is in a much higher plane. Because everyone has this knowledge and understanding and allows for much better, faster innovation. That’s something that we do quite a bit.
EB: Awesome. Recently I had Nir Eyal from Hooked on my podcast and we talked a lot about the power of habits, creating habits, getting that engagement and keeping your users engaged. Can you talk about how behavioral psychology and habits have impacted your work as a product leader at Duolingo?
JM: Absolutely, yeah. Psychology in general is super important for us in two fronts, right? One of them is the learning side and making sure that people are actually acquiring new knowledge. And the other side is the engagement side and making sure people come back every day and increase their commitment over time. What we find that’s really interesting is that often those things mix in unexpected ways. Sometimes an experience that’s meant to educate and improves your cognitive ability actually is really hurtful for engagement. Because it’s hard, learning is hard. And the harder you work, the more you learn. But the harder you work the less likely you are to come back some other time.
We use a lot of learning science and behavioral science to try to find the sweet spot for that. For example, something that we noticed in the past is that if we, like I mentioned added harder content, people will drop off. So, we had to keep it easier. Now what we do is that we allow users to drill into different skills at their pleasure. We released this feature called crown levels, where once you do a skill, you can choose to go to the next skill or you can choose to do harder content in the same skill. You can level it up, and that has been incredibly successful for us in terms of improving metrics and engagement and learning as well. That’s one of the examples where using some psychology you can get your cake and eat it too.
More on the behavioral side, some classics that we’ve been doing for some years, streaks are super powerful. They are a way to visualize your mastery of learning, visualize your commitment. They’re also a way to signal to the community that you are someone who belongs in this community, who is a leader in this community. So those are fantastic.
Push notifications are also key. Perfecting the timing and the wording of those messages also has a lot of psychology around that. There are some areas that are important.
EB: Going back to team and product managers, we talked about these eight areas, but what other attributes are most important in product managers? Or is it really you try to encapsulate in those areas?
JM: Those are the eight that I most care about. At least when I screen for product managers. I think once they’re actually on the job, knowledge of the product itself becomes incredibly important. Knowledge of our users as well. I guess maybe one that I didn’t include that I should have included is curiosity. Are you going to just eat up as much information about the product and the users in the market as quickly as you can?
EB: Yeah, and things like empathy would probably fall under user centeredness, right?
JM: Absolutely.
EB: And passion I guess to that extent too.
JM: Yeah.
EB: Looking forward, the craft of product management. Product management is a relatively young discipline, especially when you compare it to engineering or sales or marketing. What trends Do you see in the next few years that will affect the craft?
JM: I think there’s a lot. One of these is obviously AI, machine learning. I imagine A/B testing and experiment write ups and some things like that will evolve quite a bit. I think we’ll have experiences that are less set in stone and more dynamic, that just evolve with users. Where you craft an … I think the job of the PM is to craft the experience that drives the business results. Right now on mobile, that usually means you’re thinking of a UI and UX flow. In the future, it might be you’re thinking of an algorithm a lot more. Some people already think that way. But from our comment, for PMs to be thinking about the experience that algorithm create than the UI. I think that’s a big one.
I think voice is another big area that will change the way PMs think. Where again, moving away from a UI that’s graphical to one that’s purely voice and audio. I think another area that will change is the complexity of the teams themselves. So, traditionally you think of a team that is the engineer and the design, the PM, maybe QA. It’s hard, but it’s manageable for a PM to think through across all those areas. But as technology starts taking on more and more areas of your life, you can imagine those teams becoming more and more complex.
For example, we now have a linguist that are in teams. That changes things a lot, and we might have people who are writers or create stories and things like that. So, that means that PMs need to be sufficiently competent in more and more areas so they can understand this new counterparts and add value to them, to what they do and bring it all together.
EB: Yeah, it’s very interesting. I know recently we published podcasts with John Noronha from Optimizely. So, we’re talking about experimentation which you mentioned is one of the areas that’s going to be increasingly important for PMs. I had a podcast with Kathryn Hume from Integrate AI, who obviously, I guess, maybe not so obviously with .ai. But big into artificial intelligence and machine learning, and that’s a lot of her background.
So, you hit on two things that I saw two. It’s very interesting how that all coalesces together. On Jorge specifically, let’s turn the topic to you. What’s your favorite software product and why is it your favorite? Or your favorite product in general?
JM: Sure. That’s a tough question. I ask that question often to people interview.
EB: I can see that. I’ve done that too.
JM: But I haven’t been asked to me a long time. One of my favorite two products that I have really enjoyed over the years. One is Google Maps. I guess it’s a favorite of a lot of people. That was the reason why I bought my first smartphone. I was moving to Boston and everything was new. If you come from the west to the east, you realize that the streets in the East make absolutely no sense. There’s no-
EB: In Boston particularly.
JM: It’s actually in Boston.
EB: Mine’s now turned to ways for other reasons. As you can imagine, just travel time. But yeah, continue.
JM: Yeah. I bought my first smartphone, we went to Boston with my wife. It really felt like I belonged so much faster than it would have otherwise. Because I could go anywhere I want, I could discover things around me, and it felt really empowering. I think that’s become a theme in the industry where I think something like Airbnb is almost like an evolution of that feeling that I came from Google Maps where you feel at home in new places. I think there’s more happening in that direction.
EB: Awesome. So what about words of wisdom to impart to others in product leadership? If you had to summarize your career to like, “Hey, here’s three or four things you should always think about or always do or not do.”
JM: Sure, words of wisdom. This may sound cliché, but I think never losing sight of the customer needs. It’s so easy once you are in product leadership to worry about what the company needs and our metrics and worry about what are your employees’ needs, and what they want and their goals and internal politics or concerns. Sometimes it’s so easy to get detached from that customer. That really is the most important thing because that’s where the value is being generated. It’s far more important to generate value than to capture value. That would be my word of wisdom.
EB: Awesome. I think that’s a great point. You think about it, it was interesting, I guess. I’ll frame it this way, we did a survey of product leaders, and a lot of it was like because your roadmap and your strategic direction are more driven by customers or competitors. Oddly, I was shocked, we found that of the 300 people you survey, it was mostly B2B companies, and predominantly SAAS. Their direction was more driven by competitors, which I found A, shocking, and B, a horrible idea.
But I like to hear the customer-centric view is very important because you need something that is completely differentiated about your product that makes customers compelled to use it, right?
JM: Yeah, absolutely. It’s so easy to think that you know the pain that customers have. The truth is that you probably know 80% of the pain, and that’s what all your competitors know. But it’s the extra 20% you don’t know where you can make something truly differentiating and that wows and delights the customers. So, knowing these little things you miss is what’s going to make your roadmap 10X better.
EB: Absolutely. So, a final question for you today, three words to describe yourself.
JM: Passionate, mission driven and user centric.
EB: It’s interesting, as I’ve done more and more of these conversations, I always ask people that question. In product leadership, I hear a ton of user-centered or empathy, something that’s very user-focused, like walking in the shoes of their users. And then a lot of passion. Well, thanks. This was great.
JM: Thank you.
You might also like
