BTP: Building a UX Research Team with Mike Oren
Behind the Pixels is a conversational blog series that aims to give a glimpse into the work & life of being a designer at Klaviyo
AH: Alright, welcome Mike! Let’s start with the basics. How long have you been a design researcher? And how long have you been at Klaviyo?
MO: Well, second question is the easier one. I’ve been here at Klaviyo for about a year. For the first one, it will depend on how you count it — if you count since I formerly had a design researcher title, 2012 was the first time. So a decade. But before that, I spent a year as more of a product designer where I was doing the interaction design, sort of doing the visual design — but it was largely coming from a style guide — the prototyping and then all the different research. Before that, there was grad school and my very first exposure to user experience and doing UX research with undergrads was in 2004. I did a research experience for undergraduates where I ended up designing, developing and doing some iterative research sessions and then a longitudinal study on this gesture-driven icon system for a collaborative group where the same set of gestures would open different tools depending on the icon where the user drew the gesture. So it’s basically PowerPoint, but then students could also send their responses back. That’s why that one’s a more complicated answer for how long!
AH: So what you’re saying is it was in your bones from the beginning. It just took a little winding path to get to the official title.
MO: That’s right. You could claim 18 years or you could claim a decade
AH: Depending on who you’re talking to - I love it. All right, well, first things first, when we’re talking about UX research, what does that encompass?
MO: It’s really everything starting from helping identify who potential businesses’ customers are going to be. So if you’re trying to create a new type of product or create a new business, really identifying and going out into the world to try to observe - what are the different areas of friction in people’s lives? Or what are the areas that they want to see improvement? And where are they potentially willing to invest in those solutions? Because there’s lots of different problems people have, but a lot of those aren’t necessarily worth investing in — they’re not willing to get themselves out of their current way of doing things.
So that’s the journey of exploratory stuff, then as we move along the path we move on to concept evaluation where we have this big set [of ideas] and we diverge. We know all these different areas where there are potential problems. We’ll work with designers or if you have a researcher who is a good artist, which most of us are not but there are some who are, then they could design/mock up those concepts, create storyboards, vignettes, and whatever else in order to help articulate potential ways that those different friction areas can be smoothed out and made better for people. We’re then able to start converging into the solutions that will really provide value to people’s lives.
And after we have a sense of the real solutions that are really going to provide value to people’s lives, we diverge once again. Now that we have some sense of a core solution, how does that solution actually need to work in order to be usable? Especially if it’s early on and you don’t have a design system (and even if you do have a design system, there’s still different tools, different layouts that might solve that problem), understanding what’s the right way to engage people and have that efficiency is important. So mocking up multiple solutions then iterating and getting to that final prototype and doing evaluative research to understand how well it’s performing versus other things on the market versus previous designs is all important. This is the ideal flow of going from a blank slate. Or if you have existing solutions, we’re also pulling in quantitative data. At Klaviyo we use Productboard, so looking through Productboard to understand. Listening to Gong recordings with customer support and sales. Viewing flows on Fullstory, etc.
AH: Makes sense. That is a nice lead into my next question — you talked about these diverging paths, all these different opportunities, so the team can obviously be very diverse and contain a lot of different specialties. As a research leader, talk to us about the ideal team makeup and then what our team at Klaviyo looks like currently.
MO: The ideal team makeup is going to vary a little bit from company to company and depend on the different needs. But at Klaviyo… Well first I will say I am a major advocate of mixed methods research. So making sure that no matter what, no matter where you are, you have people who really understand qualitative research deeply as well as people who have a deep sense of quantitative. So we have hired for people who are across the spectrum. We’ve got some folks who are not allergic to numbers per se because again, we believe in mixed methods, but they are much stronger in qualitative research. On the other side of the spectrum, we’ve got people who are really strong in quantitative methods. They’re comfortable with qualitative methods, but they wouldn’t necessarily know how to do some of the more complex qualitative methods — like they wouldn’t necessarily know how to run a diary study, or the right way to do field research, which we don’t do field research so much during a pandemic [laughs] but someday!
At Klaviyo, one of the important things — and I shouldn’t say just Klaviyo this is generally an industry trend - is research teams should not own all of the research. That actually does a disservice to the people that we’re designing for because it’s important that the product managers, designers, even the engineers and data scientists, build as strong of an empathy for the people that they’re building for as possible. So part of a modern research team’s value, and something you have to think about when you’re hiring people, is finding people who are comfortable educating and training others to have these conversations with customers and look at these different types of data and not feel like they have to do it all themselves, but really understand and empower the wider team. That’s something that’s been important as we’ve been building out the team here at Klaviyo — getting people who don’t feel like it has to be done perfectly, because that’s another thing that sometimes researchers get hung up on is they want this perfect model of research. And if you have a perfect model of research, it’s sometimes hard to allow pieces of it to be democratized. Especially if it’s not your sole job to do research, then you’re not going to and you shouldn’t have to care about all the little nuances that someone who’s focusing only on research will have to do.
AH: That’s a nice dovetail into the next question — you all are your own team and less embedded in a single pod, although we are moving towards a hybrid model. So how do you determine where your researchers are going to spend their time or what projects everyone should pursue?
MO: It’s a combination. Number one northstar is the company’s overall goals and objectives. Things that AB and his leadership team are saying are the most important things for Klaviyo right now. Anything tied into those are where we lean in. Anything not tied into those, since we know that they’re still important in order to get work done at Klaviyo, what we end up doing in those cases is understanding what’s the complexity and ambiguity of those different things. Also, what are the different breakdowns of the skills between the product managers and designers in each pod and portfolio. Then we make sure we’re syncing up with the design director and product director within those pillars to prioritize which areas we’re going to lean into and which ones we’re going to, not ignore, but let those pods own a little bit more.
Across the board, there’s recognition that in some pods, the product manager is a little bit better at the interviews and in some, the designer a little bit better and in others, neither have a strength in research. So that’s where we might lean in a little bit more. Sometimes the designer and product manager might have to focus on the near term deliverables, but they also know that they can’t ignore what’s coming up next on the roadmap. We’ll lean into those areas in order to help them get more runway so when they get into that next quarter, it’s not that we’ll have done all the research, but we’ve reduced the ambiguity enough that there aren’t as many risks of putting certain things at the top of the backlog for the developers, designer and product manager.
AH: Yeah, that’s interesting. It’s that relationship between the pod PM, designer and researcher.
MO: I should say some of that is shifting now that we’re getting more researchers on the team. There was a survey done amongst product managers last quarter that asked for more technical support from research. This was always part of the plan, as long as people were finding value in the consulting that research was doing, to get to a better ratio. We’re still not quite at the 1:4, researcher to product pod level, but we’re close enough that we can start to operate at that level. It’s still too many pods for the researcher to do all the work and so the pods still own their research, but it’s also few enough pods that if one or two of those groups have really important things or the designer or product manager feels overwhelmed, the researcher is able to lean in and offer more support.
AH: So we’re working towards that ideal ratio! In the meantime, you guys have put together a lot of resources and opportunities to get feedback. Can you describe what opportunities you all are providing like office hours so people can get help if they don’t have a dedicated researcher?
MO: That is something that we have been experimenting with and we’re working on getting more consistency across all the different pillars and pod areas. We have had weekly office hours that I run for anybody to ask me questions. We also have the Slack channel for anybody to ask asynchronous questions. Adrienne, a researcher within the App Communication pillar, was working through a weekly cadence of meeting with the design lead and then also with the director of product on a weekly or every other week basis.
In other areas, like the area that Katie had been supporting, it was much more of an ad hoc setup. As people had questions, they would reach out to her and then she was largely working with product managers within areas that her research was most directly affecting.
And then we had some researchers whose work didn’t tie to a single pillar. They were only having conversations when we identified that the work was actually going into a pillars area and then we’d go down to that particular pod that would be impacted and have that conversation with them.
What we found is Adrienne’s model works most effectively here at Klaviyo. So these models mentioned will work to different levels of effect depending on each different organization. One exception that doesn’t work is if you’re only working at that top level. You also have to have a similar structure at those lower pod levels. Basically having regular meetings with all the designers and product managers in the pods. But again, too many pods and the researcher will get overwhelmed and won’t get any work of their own done. Too few and they’d be leaning too much into the work… We’re hoping that we’ll get to that right structure now.
From here, we’ll normalize what the expectations are for researchers working within that [system]. This will include not just doing their research, but because research often crosses pillars especially with the nature of Klaivyo, researchers will be leveraging their relationships with other researchers to do cross-product initiatives. We are trying to create more of a spiderweb, hive mind, collective intelligence, whatever you want to call it. This way every researcher should hopefully get to know everything that’s going on.
AH: That’s awesome and totally understand it’s a work in progress. That being said you have taken us from having no research team to having a really robust group of people. What has been the most challenging thing in the last year of building the team?
MO: I’d say making sure that we’re keeping everything equitable amongst the team — you have different size challenges and projects. At the same time, you have different skill sets, you’ve got different philosophies. Anthony (senior researcher) told me when we were hiring people, he was surprised by the diversity of skill sets on the research team because he’s been used to research teams where basically all the researchers were interchangeable cogs, which is very much not my philosophy when building teams. So that’s been somewhat of a challenge — making sure everybody’s aware of [our team’s diversity] and why different people are assigned to projects based on their skill set. I haven’t had anybody complain about it or have any issues with it, but it’s just something that I keep in the back of my mind. On top of that, making sure we keep up that diversity of talent — when I say that I mean the big scope, like different thinking styles and an ability to collaborate and learn from one another because that’s what ultimately makes them the best researchers in tech. You’re going to be exposed to so many different approaches to research, and you learn how they all come at it from different angles and even different levels of rigor. Sometimes you need high levels of rigor. Sometimes you need the quick, dirty answer. And I will admit I am very much a quick, dirty answer type of researcher.
AH: But you’ve built a diverse team to support it. I love it. All right, so now thinking towards the future, what is the biggest upcoming challenge you foresee and then the thing you’re most excited for?
MO: I would say biggest challenge I see is now that we’re starting to align researchers closer to the product team’s org, as well as building a better structure for getting full alignment with research projects, is making sure we don’t lose sight of some of the things that are important to know but people don’t realize they’re important to know until we get the findings. Like, for example, the flows work that was done, it improved flows turning on by 16.5%, and it’s not just that the work wouldn’t have been done but the research to prioritize making that change probably wouldn’t have been prioritized by product at the time. So making sure we don’t lose the ability to do the right thing for customers while at the same time making sure the product org feels like their voices are heard in the prioritization process.
Part of that, too, is making sure that the team continues to feel connected with each other especially across different pillars. Now that we’re adding some structures, some researchers will have their scope of work slightly decrease and then others will see their scope of work slightly shift.
AH: And what are you the most excited about?
MO: Most excited about… Everything with Klaviyo honestly. I think Klaviyo has a lot of different opportunities to expand and grow. And I think it’s an exciting place with fun people! Especially if we start doing international expansion. We can do international research. That would be awesome.
AH: Endless research opportunities! And now finally two questions about advice, what advice would you give to someone who is interested in learning more about or becoming a design researcher?
MO: There’s a reason why the #1 answer researchers provide is “it depends…” context is empress in anything research related. The background you’re coming from, the skills you’ve practiced, etc. can help define what your “shortest” path into research will be as well as where you most need to grow. There’s not one “right way” to do research so much as there’s the right way for what is needed. Increasing your breadth of methods (I created a now horribly in need of updating “cheat sheet” a decade ago) are helpful but so is getting a good understanding of sociological, psychological, anthropological, etc. theories and understanding how to apply those to help interpret what you’re hearing and observing. Also some books to get you started: Observing the User Experience; Understanding Your Users; and Rethinking Users (especially if you’re in a B2B design space)
Research is not just reporting what you hear and observe, but understanding the why people do or say what they say and working closely with Product and Design to turn understanding into action. Like most of design, there is some learning via “apprenticeship” or doing and it’s definitely not something you can master just by reading about.
AH: Very good. Second, for those not lucky enough to have a Mike Oren and team at their company, what advice would you give them to make sure they’re doing their due diligence for research?
MO: The most common mistakes I see designers without a research team making are focusing on the discovery work — this is often seen as the “sexy” type of research, but it’s also the most prone to misinterpretation of data and/or failing to bring value to the business (lots of politics in that space); not doing usability testing (less sexy but the best way to find out if your design is working and easiest to start showing the value of research); and only reporting verbatims (user quotes) vs. the WHY that drives the behavior.
Even when establishing new research teams, the standard operating model is to always make the first initiative a usability test of some type because it’s always actionable and the results are always measurable to show the impact (and value) of research so designers working solo shouldn’t fall so in love with their designs that they skip this vital step to learn how their design could work even better for other humans.
🗒🕵️♀️ — It’s been awesome working with Mike and the research team here at Klaviyo. Good research is an amazing secret sauce to creating world-class prouducts!
Does working with or being a mixed-methods researcher sound fun? We’re always looking for great people to join our team.
