Yammer rethinks notifications: How much email is too much?

Ron Blandford, Yammer Product Manager

A (not so) long time ago, Yammer sent quite a few emails.

All right, lots of them — for every single message and reply that made it into someone’s Yammer inbox.

We didn’t think there was a downside to sending more emails if they were important. After all, these are emails from your coworkers about priorities, project milestones, and results. But our decisions on when to email someone were based on data from A/B test results. However, we only tested new emails, each as an isolated notification, for two weeks at a time.

Trust the data, but don’t trust the data.

After a while, people felt overwhelmed and started to complain. Some people turned off email notifications or simply ignored them. Others were engaging with Yammer only from their email inboxes and, by doing so, missed out on much of the discovery, rapid feedback, and lightweight conversation (goodbye, email monologues) that Yammer enables.

And we noticed.

A different approach was in order, and I started by listening, first hearing everyone out to gauge how much (detail and volume) is too much. Early on in this process, it became clear that by sending super optimized email less often, Yammer could more effectively engage its users.

Crafting a new approach wasn’t easy.

As part of a larger effort to improve how we notify users about what’s happening on Yammer, we reconsidered the design and the delivery rules behind the emails we were sending. Alongside our customers’ frustrations, we came up with three big hypotheses towards taking a new approach:

First off, we needed our notifications to match our users’ expectations.

Yammer enables rapid feedback, so triggering emails independently of each other spurs information fatigue. Originally, users would get an email for every conversation and each of its individual messages that landed in their Yammer inbox. This time around, we thought, “If you’re working on Yammer regularly, we should send you fewer emails.”

If you seem to be keeping up with a particular conversation, we should notify you when it develops. Whenever you stop logging in to Yammer to read the developing conversation, the emails should stop too.

We agreed that the emails should reveal what’s most important. And that’s it.

We thought about providing high-level signals (coworkers’ names, faces, etc.) and shortening message content. This way you could quickly decide why you’ve received an email and whether it’s necessary to respond right away.

And ultimately, the notifications should improve the way teams work.

Yammer believes you’re more productive collaborating with a team, not consuming conversations via email. People who predominately add to Yammer conversations through email aren’t fully engaging with their network on Yammer. There’s so much more available in the product!

We changed a lot based on what you said.

We started with our second most delivered email for the first experiment. Here are the changes that went into the treatment:

• Better triaging of the notification
• Explicit mention of why the notification was sent
• Prominent indication of people involved in the conversation
• More distinctive placement of group and network names
• Faster way to identify Yammer as the sender
• Removal of “reply via email” functionality
• A 60-character snippet of message content

The initial response was . . . angry.

For starters, I decided to roll out the new changes to Microsoft employees exclusively. By making early announcements about the impending changes — we tried to soften the blow. (Changing email deliverables by design could abruptly impact some employees’ workflows.) However, it didn’t help much. Some people just didn’t like it, especially the 60-character message body.

More specifically, changes were fraught with concerns about clickbait and content hogging. So we tested another two treatments: both a 200-character snippet and a full-message body. Also, I let the experiment run a full eight weeks just to look into the possibility of false, clickbait success.

People still complained, but the data was great!

We were sending upwards of 3 million of these notifications a day, so achieving statistical significance after a week was possible. But even after 6 weeks, user engagement didn’t drop. In fact, it increased by five percent. That’s 100K more days of engagement per user in our treatment!

Also, after the spike, engagement remained steady during the remainder of the eight week period for both the short and long treatments. But the most encouraging results showed a 50 percent decrease in the delivery volume of the treatment email and an overall 34 percent decrease in volume of all Yammer emails sent.

We’ve now implemented the redesign and realize that lots of people are still unhappy. But considering the results, I couldn’t, in good conscience, not implement the new email experience.

Sometimes the decisions that are best for most people are terrible for some. Just know that we read a thousand plus messages and hear you.