Mailwave — How we are designing a new email engagement tool
--
Our Palo Alto tools team is responsible for generating and incubating new ideas for the broader SAP tools, and delivering them effectively. In early 2016, we tasked ourselves to develop a tool that’s worth our time and resources, as well as manifest our help others shine principle.
This post will talk about one of the tools we’ve been developing for the past few months, to do exactly that: help our fellow SAP-ers do great things. That tool is Mailwave, a web-based app that helps employees measure the success of their mass emails. Here we’ll dig into how we came up with the idea for it, how we developed it, and what we’ve learned from it.
Where Mailwave came from
As a small team, we believe in solving the right problem before solving it right. For that, we did two rounds of experiments to scout out problems to solve — a.k.a. design sprints.
In the first round, we followed a short version of the continuous design thinking process. Our team talked to product owners from all over the company, to test out some product concepts with them and see what kinds of value they were looking for. It was successful in that it showed us that our first product hypothesis was not strong enough to pursue.
In the second round, we took a slightly different approach and mixed the lean UX process with the d.thinking process. This led us to Mailwave.
At the beginning of the second round, we realized that we need to reflect deeper on our own experiences inside the company to make sure we’re orbiting the right problem space. Our team members generated ideas based on the burning needs of product teams.
One idea stood out amongst others: A communication tool to track the engagement success of mass emails inside the company. For the past two years, the team has been using an external email campaign tool for one of its projects, and we wondered why we didn’t have a similar tool to measure the success of internal emails. Such a tool could provide valuable insights for communications teams as well as casual newsletter owners when sending out mass emails.
We decided to go with a lean UX methodology and work on concepts both on design and technical sides, and talk to users to validate our prototypes.
Create a hypothesis, prototype, and test with users
Our first hypothesis is all about communication effectiveness. Employees need to see the impact of their internal mass emails. They can then assess the effectiveness of their communications or even define new performance metrics for them. To achieve this, users need an internal communication tool.
Another pillar that surfaced during our discussions was integration with existing communication workflows. We didn’t want to add yet another tool to our colleagues’ already crowded tool landscape. Our second hypothesis is therefore about integration. Internal employees need a tool that works with their existing communication workflows.
As a team, we have a hunch that users want more than open and click rates, and need insights to enhance their communications. Our scaffolding into the engagement space also showed that measuring emails requires us to work with big data and get into the machine learning space. This helped us to form our final hypothesis. Users want to leverage learning from past emails to enhance future emails.
These hypotheses shaped our scoping efforts quite a bit. With limited resources, we were also aware of the technical complexity of such a communication tool. Scoping of the first version of our concept was critical to maximize our time commitment. Our priority was clear: We wanted to make sure we successfully measure the engagement of mass emails. This led us to focus on the sending and tracking of emails and intentionally opt out of the drafting of emails via an editor or templates. With our focus set, we created the following key path scenario:
A user first sends a final draft of the mass email to the tool through their regular email client. We respond with a magic link. When the user clicks the magic link, she adds the recipients, tests the email, and launches it. She then measures the open rates, click rates, and other metrics through the stats page.
By starting the launching flow through the user’s existing email client (such as Microsoft Outlook), we wanted to prime the user that they don’t have to change their existing workflow, and that they are responsible for their own content creation. To test these hypotheses, we created a series of prototypes from low to high fidelity and tested them with users.
Crafting the minimum viable product (MVP)
Having worked on a series of iterations of the early concept design, we slowly worked our way to a working version of the tool. In a way, this version is the MVP of our long-term vision. We wanted to make sure that users can send emails and measure the basics. We also wanted to make sure that they found the workflow easy and see enough value to use or switch to our tool. We tested this version with 5 different teams who already send mass emails to their communities. The concept resonated with them well. There was an untapped need among teams to include the effectiveness of their communications in their performance metrics. Some teams with external facing outreach were more aware of such a need. Moreover, having seen the measurement basics such as open and click rates, more possibilities opened up during our feedback conversations, such as “best time to send” and “best subject matter to choose from”. This strengthened our hypothesis for providing more insights through machine learning capabilities.
Pushback for integration hypothesis, discovery of the need for an immersive onboarding
During our iteration sprints and while talking to users, we observed that our second hypothesis (integration with existing workflows) was proved to be working. However, there was a friction in the integration, especially for new users trying to launch a new mailwave. Users need to leave the tool for their email client and return to the tool after clicking a “magic link”. Even though users appreciate the fact that they can continue to use their existing email templates and workings, there was still some kind of mental model mismatch. This back and forth might be familiar to some users from conventional login page behaviors, but it wasn’t natural for most of them.
We took this friction as an opportunity to design an immersive onboarding experience. All team members workshopped the possible onboarding scenarios. In our discussions, we identified two behavioral personas. The first likes to figure things out without relying on any guidance. The second likes to be handheld. We realized that we need to serve both so we provide two entry points on our landing page: “Get started” and “Take a test ride”.
Let’s talk a bit about the second entry point. In our discussions on how to onboard, we realized the importance of learning by doing, and so we handhold our user through their first mailwave experience. This handholding could be like a test ride. We crafted the test ride meticulously, from its flow to the selection of wording. At the end of the iteration sprint, we tested it with users and made final refinements for implementation.
Iterating on the navigation
Our MVP version made us more confident about the tool and its overall workflow. With our early evangelists using the tool and sending more and more mailwaves, we realized the limitations of the existing navigation. As designer and developer pair, we sketched alternatives to the table list view under the mailwave. The earlier version of the sidebar navigation came out of that effort. Over a couple of months, we worked on two more iterations of the sidebar navigation.
Interim design with sidebar navigation
The evolution of the design happened over short rounds of design sprints between designers and developers. The latest sidebar navigation with a hamburger menu underwent rigorous discussion and user testing. The underlying need for hiding and showing the navigation was to provide a more focused view to users, especially when they are checking the performance of a particular mailwave. In testing with users however, we learned that they want to see their previous mailwaves in the background. This led us to interpret the hamburger menu and apply the collapse behavior only for the folder navigation.
Where are we in the journey and what’s next?
Mailwave’s growth inside the company has been impressive. Our colleagues have launched around 500 email campaigns that have reached around 200K recipients. Several of our key communications departments are now using Mailwave in their workflows. Our technical team is already creating a more scalable infrastructure for the product using cloud services. Meanwhile, as designers, we have been working on a design guideline to streamline future design implementation efforts.
Our design and development cycles have so far helped us to validate the three main hypotheses that we developed at the beginning of our journey. The adoption rate for Mailwave inside the company showed that our initial hypothesis around the need for measuring the success of mass emails is valid and urgent. Our hypothesis of integration in existing workflows was on point. However, it required us to revisit our workflow and design a more immersive onboarding experience. Talking to our participants also made it clear that users want to go deeper and leverage insights from past mailwaves.
As a team, our next challenge is to level up our game in our AI capabilities. As our data grows, we’re getting more confident in how to make sense of user behaviors and move into the predictive and more insights-driven experience arena. For instance, a typical scenario is identifying the best time to send an email and the right recipients. We’re also imagining more edge scenarios where we can diagnose user content and make suggestions for future content that fits the audience.
Many thanks to Leslie MacKay and Rachel Reynard for the editing! Big thanks to the SAP Tools Palo Alto team, including (in alphabetical order) Alex Wu, Ben Boeser, David Farr, Dominik Tornow, Jan Lagarden, Leslie MacKay, Max Vogler, and Rachel Reynard for the awesome product design journey.
