UX case study: Stack Overflow email management
If you have an email account, you’ve probably received a product email at some point. Chances are that your experience fell into one of these buckets:
- The email was helpful and relevant. Managing your emails was dead simple.
- The email wasn’t really helpful and only sort of relevant. It was mostly easy to opt-in or out of emails, but some settings were vague or hidden behind a login wall.
- The email was irrelevant, even misleading. It was difficult to opt-out; in fact, you’re still receiving emails about this product after hitting “unsubscribe from all” countless times.
Stack Overflow’s email experience has hovered around #2 — the few emails we sent were mostly relevant, but management features were scattered and neglected in dark, dusty corners of the product. As we started to send more emails, there was a risk that we’d slide into #3 unless we made some major improvements.
Our team set out to fix this experience, and here’s my take on our process.
This is the second in a two-part series about how we overhauled Stack Overflow’s email management UX.
A quick history of email at Stack Overflow
Up until a year ago, Stack Overflow didn’t send much email. There were a few emails sent to highly active Q&A users, plus emails for basic things like resetting your password or verifying your email address. The assumption was that users didn’t want or need emails from us. If they wanted to hear from us, they could visit the site.
This assumption was true for some users — particularly highly active Q&A users, who are already visiting the site regularly to moderate or participate in question threads. However, this is a small, expert slice of users. The majority of users visit irregularly to get answers, find a job, or learn about an interesting topic — and might miss content that would otherwise be interesting or beneficial to them.
In fact, as we dug into metrics and user feedback, we found that users responded pretty positively to (relevant) email. Our assumptions about email started to shift and broaden, and we began lining up more email projects across the product.
As more emails were sent, experience problems — particularly around email management — became increasingly apparent. To understand these problems, let’s back up and talk about the structure of Stack Overflow.
Stack Overflow is a product for developers. It’s composed of our flagship Q&A feature, Jobs (a job board for programming & programming-related jobs), a business entity, and Stack Exchange (a network of 150+ community sites on topics ranging from web security to cooking).
The product serves over 5 million users with varied tasks including (but not limited to):
- Answering someone else’s programming question
- Finding an answer to a cooking question
- Finding a job
- Building a new community around a topic like Pastafarianism.
Depending on the tasks and sites that a user engaged with, they’d need to visit three or more pages on disparate sites to manage their emails. Some of these pages were behind a login wall, and none were mobile-friendly. It was hard for users to know what they were subscribed to (or unsubscribed from). Some settings were synced across all sites, and others were not.
There were also major infrastructure problems that made improvements to our email sending and managing system costly to implement and difficult to measure, such as poor tracking, limited A/B testing capabilities, and a fragile deployment system.
Defining goals
The UX goal was simple: create an honest, clear, and easy-to-use email management system for users regardless of:
- What tasks they were completing;
- What sites they were using;
- Whether they were logged in or not; and
- What device they were using.
Side note: implementing a more robust email sending & tracking system was a significant part of this project, but this case study will focus on the above UX goal only.
Conducting preliminary research
We started with an audit of our existing emails: what they said, where they came from, what conditions triggered them, and how they were grouped. This helped us broadly understand use cases and problem areas.
We then conducted an analysis of email management solutions on other sites.
The goal was to learn from standards and insights already gained by other companies with similar needs. When analyzing these experiences, we asked questions like:
- What do the best experiences have in common?
- What experiences reflect or contradict our principles?
- What are the paths per use case (eg. logged in vs. logged out), device, and task? How, if at all, do they differ?
We then cobbled together this research, existing data, and past user feedback in order to identify and prioritize pain points.
Defining pain points
Pain Point #1: Managing emails on your phone was really hard. Users attempting to manage their email settings from their phones were simply sent to the desktop site, where they were expected to pinch and zoom in order to change — or even see — their email settings. This presented usability and accessibility problems for a significant subset of users, since our data indicated that a large slice of email engagement occurred on mobile.
This one might seem obvious in a mobile first world, but Stack Overflow has historically centered around Q&A, where users are writing code, reading and writing content, and testing code. These tasks overwhelmingly take place on desktop computers. The introduction of email grew mobile traffic — and consequently, led to more mobile investment.
Pain Point #2: You needed to login to manage or unsubscribe from some emails. If attempting to manage email settings while logged-out — perhaps while using a work computer or phone — users were sometimes blocked by a login wall.
Pain Point #3: Email settings were confusing and inconsistent across sites. In some cases, users needed to visit three or more pages on different domains to manage their emails. Some settings were synced across all 150+ sites, while others were site-specific — and it was hard to know one from the other.
Pain Point #4: Email groups — and which address they sent to — were confusing. For example, users opted-in to the bucket pictured below were eligible to receive emails like beta feature invites, online event reminders, and notifications about their reputation milestones — something that wasn’t immediately clear. In some cases, those emails send to a different address than the one displayed on this page.
Designing and testing the solution
We went through several design iterations before arriving at the solution. Each design iteration surfaced risks and open questions, which we explored through hallway and usability testing. Here’s our solution.
A touch-friendly mobile design. This addresses the difficulty of managing emails on mobile. A reusable toggle UI, which is both touch and desktop-friendly, can reduce implementation cost for future cross-device projects.
One-click unsubscribe and management for all users and emails. Even if you’re logged out, you can control all of your emails easily.
Clear, consistent design across all network sites. This reduces the cognitive load of processing different design patterns when completing similar, or identical tasks. Multi-site vs. site-specific email settings are clearly organized and communicated to the user.
Emails grouped based on common characteristics. We remapped emails and added new groups like Research and Tips & Reminders. Succinct, scannable copy and design makes it clear what you are subscribing and unsubscribing from.
Takeaways
Email management is one factor in our experiences with product emails. There are still plenty of improvements to make as we scale up email efforts — for example, it’s not only important that management is useful and easy, but that the emails themselves are those things and more — relevant, timely, and respectful. This also brings up questions about other communication channels that might be helpful to users, like in-product or mobile notifications.
Thanks for reading! To learn more about Stack Overflow’s email settings project, check out my colleague Ted Goas’s blog post, Designing a Better Email Preference Center. If you have your own stories about email, I’d love to hear. Please leave a comment or email me at donnachoi30@gmail.com.
