From feedback to feature: How to pivot based on new customer needs

Roadmunk teamed up with Women in Product Chicago for their first virtual discussion — From feedback to future: How to pivot based on new customer needs.

Priya Mathew Badger, SVP of Product at Codeverse, and Maria Pereda, Director of Product Design at Roadmunk, talked about how to use the changing needs in customer feedback to successfully pivot and retain customers.

We’ve pulled out some highlights from the conversation if you’re tight on time but we highly recommend tuning into the full conversation between Priya and Maria — it’s jam-packed with helpful recommendations and insights.

Highlights
(The highlights have been condensed and edited for clarity.)

Priya’s PM background & Codeverse (1:52)

Maria: Can you tell us a little bit more about yourself and how you got into product?

Priya: I started my career in economic consulting, so not in tech at all, but I was always super passionate about technology. I loved all the Google products. I used to submit future requests online to Gmail and Google docs as a user. And then I was lucky enough to join the Google apps team in 2010 when the product was really taking off. It was an exciting time to be there, as we were just starting to monetize and growing to millions of users. I would say one of the best things about that experience was just getting to see lots of different product teams because Google apps is a suite. So, seeing Google docs, Gmail, calendar, all those different teams and how they operate and how they did things, some things similar and some things different in terms of how they manage their products.

After Google I actually went back to business school at Northwestern, and I got a dual degree in the MBA program and the engineering design program there, and it was a really cool way to just build design thinking skills. And I started at eSpark in Chicago afterwards and I love being here in Chicago and part of the community here, especially with the strong Women in Product network we have. And eSpark built a reading, math and writing software that we sold to school districts. Google apps was a B2B product, but it was built on top of a consumer code base eSpark sold to schools and nonprofits. Our end users were young kids in K through eighth. So my first time designing products for kids as young as four and five, which was a lot of fun, but also some new challenges.

And then at Codeverse, we build a platform that teaches kids coding from six to 13. And we’ll talk a lot about that. But Codeverse is a consumer company. So we’re selling to parents who are signing up their kids. And so, I’m still in that education space, but now I’m working on the consumer side, which definitely has some different aspects to the product development, than when you’re working with organizations or businesses. So I have gotten to see some different experiences, be at big companies and smaller startups.

Pivoting from a 100% in-person product to 100% virtual (4:24)

Maria: So up until COVID hit, Codeverse was a 100% based on in-person events and classes, right?

Priya: So our business model was we had physical retail, like brick and mortar studios, that parents would sign their kids up for after school classes, camps, events, birthday parties. And at those, they would be using our online coding platform and the kids would be learning to code with our software. But we felt like that was sort of the consumer behavior. At the time, parents are used to signing their kids up for after school activities. And it was a good business model for us to get started with the company. And so 100% of our revenue was coming from that. Obviously with COVID, we ended up shutting down our studios during that time and really had to rethink the entire business from the ground up in terms of what does the business model look like?

Maria: How did you handle that? How did the entire product process change and how did you handle that pivot?

Priya: I think one of the things that helped us was we have a very strong leadership team and we’re a small company. So we were able to kind of all get in a room and we saw it coming, obviously. You see what’s happening overseas. We had great board members who kind of gave us a heads up, like schools are going to close. You’re probably going to need to close your studios. We talked about it even in our Q1 board meeting and so we were thinking about it in advance. Luckily, we do have an online platform and we knew that we had great assets in terms of our instructors that teach at the studio. So our product, we had already considered was virtual tutoring, one-on-one sessions. And we’d already sort of done a little bit of the groundwork, but this just super accelerated the timeline for that.

I think a lot of the changes came from just that absolute focus on this is the product that makes sense for us during this time. And we all need to go all in on it. And in some ways, we kind of joke that the studios closing was the only way we could have made virtual tutoring successful as a product. Just because it allowed the entire company and all of our resources to be focused on it.

Conducting customer discovery when you can’t do it in-person (6:54)

Maria: Before then, all your feedback collection and understanding insight was based on going in-person and observing and interacting with both the parents and the students, how do you do your customer discovery now?

Priya: Yeah. And that was one of the things that attracted me to Codeverse and I loved about it — was it is really magical to be able to be sitting in a room with your users and just observing them as they’re using it and come kind of in and out of the experience. But I will say that I actually think in some way, doing things virtually has made it easier because we record every session that kids are doing in our online sessions. And so I can go back at any time and watch those recordings and rewind and watch some again, where it was just much harder to do in an in-person setting. And we also started a parent survey where after every session we would survey parents about their experience and their kids’ experience. And that just gave us like a really rich data set that we hadn’t had before.

I think when parents are thinking about the product as the online experience, they’re more comfortable giving that feedback in an online way, versus in the studio, a lot of the feedback came in more anecdotally through conversations they’d have at the front desk or with our staff. The other thing that I’ve done that I think is a little bit new, is joining some Facebook groups, like moms groups and doing kind of social monitoring. There’s just a lot of challenges facing parents right now. And they’re and everyone’s talking about it. And so it’s been interesting to see some pretty direct questions in some of these moms groups about the problem and opportunity that we’re working on.

Handling large amounts of feedback and turning it into something actionable (9:15)

Maria: So now you’re getting tons of information, which compared to before is just the volume sounds incredible. How do you handle that? How do you absorb that and turn it into something actionable?

Priya: I think even in the case that you have a lot of uncertainty, you always need to have some kind of plan and some kind of focus. For us, we kind of came up with what are the objectives that we need to do? And part of it was just transitioning. So how do we get our current customer base to move over to this new platform? And that was our first focus. And then once we got a majority of those customers to move over, then the next focus was how do we attract new customers and how do we grow that customer base? And then now we’re really focused on retention. As well as obviously everything else continues to be important.

I think when you have those focus areas, you’re able to better figure out where’s the feedback that you need to pay attention to. When we were focused on our existing customers, it was a lot of looking at that parent survey, watching those initial videos and figuring out what about this experience feels different than the studio in good ways and in bad ways. And now that we are online, how do we make sure that our experience still feels premium? And how do we make sure that it’s still on brand for what we want, which is the best educational experience for kids learning to code so one example is, when we first launched virtual, we had small group classes because we do classes in the studio. We figured we’d do classes online. We made them pretty small.

They were only three or four kids. But we just saw that, that wasn’t the premium experience when you were online. Because there were kids of multiple ability, levels and ages. In the studio it works because the guide can kind of go along and work with each kid. But in the online setting, it was much more difficult to manage. And so at least in the beginning, we decided that even though it was more expensive for us to do it this way, we would give every studio member a one on one session in exchange for every class they would have had at the studio, which is obviously much more costly operationally.

But I think was worth it just from that goal of really providing customers the value that we believed met our brand. Now later, we ended up kind of launching a virtual camps product, which does have a group setting that has different programming and different techniques that we added in order to manage that group setting. And then now we’re kind of going through that same process of watching videos from camp and trying to figure out where are the things that we need to troubleshoot and improve on.

Getting alignment on implementing feedback (12:15)

Maria: So that’s interesting, you’re talking about this bet, these premium experiences that operationally are much more expensive. How do you bring along the rest of your company in a bet like this? How do you have the confidence that maybe this is the right move to make?

Priya: Some of it is trial and error because we had a mix. We had one on one classes and we had group classes. And so we knew that the one on one sessions were getting amazing feedback and the group sessions were a little bit mixed. And so some of it was just bringing that customer feedback, those recordings, those quotes from parent feedback to the light of the leadership team. And there are other people who are hearing that feedback, not necessarily from the product perspective, but from the people who are working in customer service for parents, or the sales team who might be following up and trying to convert that customer. So it was coming from multiple angles as well. But I think the real thing was just being okay making a big bet and knowing that you don’t know if you’re right.

One of the things our CEO said was we’re kind of being faced with a series of somewhat impossible to make decisions. At the end of the day, we just want to be able to look back and be proud about the decisions that we’ve made and feel like they reflected the values of our company. And so that was always a guiding function for us. One of the things we did was we closed a week or two before the state home order, which we could’ve made money off of the studios being in session, especially cause parents were desperate. One school was closed. So there was a gap between when schools closed and when organizations like us would have been required to close. But we felt like the right thing to do was to close and protect our staff and protect the kids and just move faster to virtual. And that’s what we did.

The other decision we made was not to lay anyone off, which required some really tough budget and financing questions around, like, how do we make the space for this? And how do we find a business model that will work that can support keeping everyone on staff? So there were a lot of decisions that I would say we didn’t know that they were going to work out, but they reflected the values and the brand of the company and they felt like the right thing to do. I think they were also all things that you can reverse and change over time. At some point you just have to go with it.

Priya’s advice for collecting user feedback for the first time (18:02)

Maria: What advice would you give to a company that wants to start getting customer feedback, but they don’t know how to do it?

Priya: That’s a good question. I think not trying to over-engineer it and just try to get something quick and dirty out there. For example, like we just made a Google form and manually texted it to every parent after their session. And then did an email blast and it was great for getting awesome feedback. Now that we’re trying to scale the program, we built in a survey model into our database and we’re going to have nice popups that come up after your session. But to just start getting that feedback, you don’t need a lot. You can just use the tools that exist for you. And then the other thing I would say is, most customers want to talk to you and give you feedback. Pretty much any parent I’ve ever reached out to.

The other thing I would say is, you don’t necessarily need to talk to people who are already using your product in order to get feedback. I’ve gotten really great feedback from people and Women in Product for example, just asking, “Hey, are there moms with kids in this age range that would be willing to talk to me about this?” And sometimes talking to people who don’t use your product yet, but are kind of in the market or in the persona is really helpful. Friends and family or professional networks can be really great grounds for that. Our company doesn’t have any formal process for doing discovery work.

I don’t think you have to necessarily ask for permission or get a budget. I think it’s all about just getting out there and figuring out where’s a good place to start.

Quantitative vs. qualitative feedback (20:50)

Maria: What would you say to companies that say, “we’re data driven, we don’t need to talk to people because we have all this data.”

Priya: We look at the data all the time about what is the usage and the actual behaviors of kids and parents on the platform. And that is another benefit of having everything happening online, is that we can really like tie all the activities together. I always think about it as, the data tells you what happened, but talking to users tells you why it happened. And it can be really hard. If I look at the data and I can see Oh, it looks like kids are engaging with this type of activity, but nobody’s doing this other type of activity.

I know that that happened, but I don’t know why it happened. And so that’s where I think talking to users and observing them gives you more insight into that. And then I also think sometimes things just don’t always show up in the data or they will show up, but they will show up much later than when you want to know about it. And especially considering how quickly we want to be able to react to issues when we’re building this totally new business from the ground up and we have a lot of pressure for it to succeed, because it’s entirely replacing the revenue that we lost. So one example is, we were kind of using a compilation of tools to get these one-on-one sessions off the ground.

So we use Zoom, we use Calendly and then we had our own internal systems and it did work at the beginning. So the data would point you to the direction of this is all fine. But if you look at the customer feedback, they were just these kind of hints and rumblings of I wish this was integrated into the product, or Zoom doesn’t work for me, or I’m concerned about Zoom privacy, or it was hard to get set up. And so even though customers often rate us a five out of five on their experience, we could still see these comments coming up frequently. And then at the same time, it was also really costly for us on the operation side to just pull all these tools together and make them all work. So we ended up investing in virtual video integration.

So now we have video integration right into our platform when kids log into our coding platform, they see the video of the guide in the corner and we don’t have to use Calendly or Zoom anymore, which we just rolled out last week. But if we had only been looking at the data, we would have probably not seen that until customers were starting to cancel or seeing it in retention and then we’d have to dig in later. So that’s something that you can get early signals on what your users’ needs are.

Prioritizing feedback (23:57)

Maria: I want to get back to prioritizing customer feedback. Do you use any frameworks specifically or how do you manage to decide what moves the needle, what doesn’t move the needle. what do we invest in, what do we not invest in?

Priya: One of the hardest things you have to do as a product leader is make those tradeoffs. I always say the best framework to me is starting with really clear company goals and strategy. If you have that, then the product needs sort of uncover themselves. Because you’ve leaned into what are the most important things for us to accomplish as a company? And then what’s our strategy for how we get there? And then from that you can derive, okay, based on these goals, what does the product need to do? Or how does it need to change to be able to meet those goals? Obviously as you’re doing the discovery and the process of building that solution changes over time. But you at least know what goals and what you’re aiming for.

And I find like that framework works really well because it just provides a filter and focus. The challenge if you use another framework, people will weigh the impact versus the effort or something like that. And you have some kind of matrix for that. I think it logically makes sense. The problem is you could have an infinite amount of ideas and you’re going through this exercise that feels almost mathematical about ranking all of these different ideas. But at the end of the day, you don’t really know the effort until you know the solution, which you don’t know until you start the discovery process and you actually start building it. There’s just too many things to actually be able to come up with a rank order for all of those. You need to have the company goals and strategy side in order to filter down that list.

Rapid fire questions

(26:55) What’s the one piece of advice you would give a product manager or leader who’s trying to navigate how to adapt to this time?

(27:58) If you could give Priya from February a piece of advice, what would that be?

(30:45) If you had to recommend a book to anyone, it can be a product management book, it can be a design thinking book — what would it be?

Speaker bios

Priya is the SVP of Product at Codeverse, a Chicago startup working to teach a billion kids to code through its interactive coding platform and studios designed for 6–13 year olds. Priya previously led the product and design teams at eSpark Learning which delivered personalized learning to kids in K-8. Prior to eSpark, Priya worked at Google on the Google Apps team. A glutton for learning, she graduated from Dev Bootcamp and received a dual degree MBA and Masters in Engineering Management from Northwestern.

Maria is the Director of Product Design at Roadmunk, a SaaS product that enables companies to visualize their strategy and align as an organization. She is building Roadmunk’s first design practice and laying the foundation for a user research program to guide future efforts. Maria has a strong track record of setting up design practices from the ground up, where she focuses on building foundations for research, strategic design thinking focused on user goals and business outcomes and creating strong partnerships with product and engineering.

Prior to Roadmunk, Maria served as Senior Director of Product Design at Schoology, a leading Edtech company in the K-12 US market. Maria established Schoology’s first design practice and elevated design culture within the organization by making numerous improvements in process throughout the product development life cycle. Past experiences include Group Director of UX at Critical Mass, leading work for the Citi Global accounts in Toronto and New York. She’s also held design leadership positions at GE, where she helped establish a design Center of Excellence, and Fry, an e-commerce startup later acquired by Oracle.

In 2018, she took an oath to never buy clothes again, and has been making her own since, with variable success.

About Women in Product Chicago

Women in Product Chicago is an organization started several years ago out of the Bay Area devoted to increasing diversity and inclusion in product roles. The Chicago organization started over 2 years ago now and they’ve been going strong ever since thanks to their strong local community.

They would love to hear from you. They’re looking for ways to help their community continue to feel connected. And they’re always looking for speakers and new event ideas. Find them on womenpm.org, click on the Chicago community and sign up for their weekly newsletter — or email them at chicagowip@gmail.com for more info.

About Roadmunk

Roadmunk is an end-to-end roadmapping tool. With the new feedback and idea management suite: centralize customer feedback, create a customer-validated idea backlog, systematically surface high-impact ideas and promote ideas to your roadmap.

Try a roadmap template, which include technology roadmaps, scrum roadmaps, agile roadmaps and lots more. Choose from 35+ ready-to-use roadmap templates and product roadmap examples: https://roadmunk.com/roadmap-templates.