Scaling Your Research Operations — A Case Study with Zsombor Varnagy-Toth, UX Researcher at Emarsys
Founded in 2000, Emarsys began as an ESP built to provide an email solution for the European market. From there, it rapidly evolved into the global marketing platform it is today. From being the first to address deliverability issues in Germany, to standing at the forefront of artificial intelligence marketing globally, Emarsys always seeks to deliver easy-to-use solutions for clients. We spoke to Zsombor about how he and his team us EnjoyHQ to manage and scale their research operations — let’s dive in:
Sofia: Can you tell us a little bit about Emarsys? What kind of research do you do?
Zsombor: Emarsys is a global company with 15 offices and more than 800 employees around the world. Our product development centers are in Budapest and Vienna. In terms of size, we have around 150 developers, the majority of them working in the Budapest office.
In terms of the product organization, we just went through a major restructuring a couple of months ago. Now we have six product areas — each area consists of five to six different product teams. Each Product team has three to six developers, and each area has its own product managers. The managers are not tightly coupled with development teams, but they oversee the whole area’s projects and they divide responsibilities between themselves.
The UX team is now structured in a way that each area has a small UX team. For example, in my area, there are four of us — two designers and two researchers. The UX team is made up of 17 people, roughly half are designers and half are researchers. If designers do research, it’s basic usability testing. But most of the research is done by the UX researchers. And what we do is mostly qualitative research, interviews with our customers and usability testing. The product managers also do some research — they mostly interview customers, but it’s not often that they do research.
Product managers use EnjoyHQ, but first I wanted to onboard the UX research team and designers so that they have access to all of the research we’re doing. We try to be very collaborative about research — whenever we discover an insight, we share it with the relevant development team, and they sometimes also show up to research sessions, as well. We try to make sure that they get as much exposure to research as possible.
In the future, we would like to involve or tap into the knowledge of the commercial organization. We have a huge commercial organization — there’s a lot of people in sales and customer success. I think roughly half of the company is within those parts of the organization. So far we have opened up an email address that they can use to send feedback directly to EnjoyHQ. I’m very happy about the new Slack integration because that’s also something they can use to post feedback directly into EnjoyHQ.
Before adopting EnjoyHQ, one of our main concerns is that we would lose feedback from the commercial organization. Some of their feedback would reach the product organization, especially if there was a major issue or a customer has a serious complaint, but smaller things that are important and happening every day would fall through the cracks. We can’t expect people in the commercial organization to open a ticket, try to find out who’s the product manager for that part of the product, and somehow transmit the feedback.
Sofia: Can you tell me a little bit about the problem that you were trying to solve before you found EnjoyHQ?
Zsombor: The very first attempt to try to solve the problem happened almost three years ago. I had been doing a lot of qualitative sessions and I wanted a way to recollect the relevant pieces a couple of months later. The research topics that we run usually span across many months, and they will come up one year later because all of the problems are very complex.
First I tried to solve the problem in Excel, then I tried Reframer. At that point I didn’t really know about the term “research repository”. Then at a conference one of my colleagues met someone from one of the new product management tools with roadmapping features, and he was really excited about it so we introduced it to the company. Two of the researchers started using it. The plan was to put research there and let product managers take advantage of all the product management features, like planning roadmaps and prioritizing different customer needs. But it turned out that there was a lot more informal random stuff that went into those product decisions, so they ended up not using the product management related features. It turned into a research repository, but it was not a perfect research repository for our purposes because it was difficult to search for certain things. Also, you can’t really share the research results or the set of results with anybody who doesn’t have access to that tool.
Sofia: Would it be fair to say that your main goal was to make your research more shareable and usable for other teams?
Zsombor: Making research shareable was definitely the main priority. But also really making use of the earlier research as well. We wanted to get more mileage out of our previous research.
Sofia: Could you tell us a little bit about the process you used to evaluate research repository tools?
Zsombor: Our process wasn’t completely random because we had already had the product roadmapping tool I mentioned before for awhile. We treated the selection of a new tool and the evaluation process as a design project — first, we talked with some of the product managers and some of the researchers about their needs. A couple of my colleagues sat down with these people and did regular user interviews trying to find out what they needed in a repository.
After a couple of interviews, we started to gather a picture of the user needs here. Then we started to ask ourselves: what are our use cases? We knew we wanted to make a change, and then we started looking at different products. We tried to be very considerate about the decision because we didn’t want to just replace a tool for all these people — we really wanted to make something better.
First, we tried to imagine what we would like to have ideally, and only then did we look for different tools. And after we’d seen what was out there, our expectations were shifted. We realized there are a lot of things that we never even thought about. We started to compare all the different tools according to the needs that we assumed we would have.
Some of the tools that we considered are tools that the company is already paying for, like Microsoft, Confluence, Airtable, Productboard and JIRA. We came up with a rating system that helped us compare all the different aspects of the different products, and we tried to prioritize them. And that’s how we had a good idea that EnjoyHQ was the best out of all of these tools, but then we wanted to see it in action to make sure it works as we expected it to work. And then it did!
Sofia: Did you have any pushback from other team members or stakeholders? Anything that created some friction in that process of trying to find a better solution?
Zsombor: Yeah, the budget was a question at one point in time, but we had an advantage in this because our old platform had raised their price and was offering half of the value that EnjoyHQ offers so it was easy to sell to the financial decision makers that this is a better decision financially.
Our security and legal team have strict processes that I have to work through; we have an internal policy that we only put data in any tool if we must and if there is a very good reason, and that’s something that we needed to sell them. We’re working with legal and security to remedy this so that we can have the data in there, and EnjoyHQ has been very supportive throughout this process.
Sofia: What would you say are the main benefits you get from EnjoyHQ?
Zsombor: What I like most about it is surfacing of the research, so the search functionality and finding all the previously quoted or unquoted topics. This is very powerful. As an example, we had a discovery project on a topic that none of us has researched before. But when I went into the research repository, there were already more than 100 different research articles mentioning that topic in one way or the other. This really boosts the speed of our research. And EnjoyHQ is unquestionably leading the field in terms of surfacing existing insights with their elaborated Search functionality. That’s when I realized, “Okay, EnjoyHQ is going to be very, very useful.” With the search capabilities of EnjoyHQ, I think finding past research is a lot easier and more comprehensive, and we’re finding more of the pieces that need to be there for a good research start.
Sharing results with the team is also important. We are very happy that summaries can be shared with a link. That’s awesome. That’s exactly what we need to share results with the team. We’re also trying to take advantage of the inbox feature, so we set up EnjoyHQ in a way that each area has their own project so they can have their own inbox. All the product managers and UX researchers get notified when something comes into their inbox. They don’t need to be there every day to see what is in the inbox — they just get notified and it really helps them stay on top of things.
We are just at the beginning of working towards our goal of involving the commercial organization. We had a little experiment with the solution consultants team — they work with customers on implementation, and we provided an email address they can use to send feedback directly to EnjoyHQ.
Sofia: What would be your advice to research teams trying to implement a research repository?
Zsombor: My advice would be to start trying to implement a repository early so that that knowledge accumulates. Because the real benefits come after a while when you have a lot of research data in there. Something that caused a lot of headaches for us is the standardization of the coding/classifying feedback. Each team had its own system of storing and classifying data, which made it more difficult to find things that were not coming from our team, but coming from other researchers. They used different taxonomy. They would refer to things by different names so that caused problems. Try to come up with some kind of consistent coding/classification system. Even if it’s not the final system, at least some kind of company-wide agreement on that will help. That’s one thing we also like very much about EnjoyHQ, that you have tag and property management so we can change our minds after we have seen it working.
