Mentioning Pending Users
Designing, Building, and Testing Features at Yammer
Yammer aims to make workplace communication easier and better than ever before. It becomes challenging when you need to include all of your colleagues in important projects, discussions, and announcements, only to find that some of them have not yet joined Yammer.
With this in mind, we wanted to make it easy to involve people who are not yet on Yammer in conversations. We call them “pending users”. Pending users are people who fell out of the signup funnel, were invited by others but never joined, were added through Active Directory sync but never signed up themselves, and so on. Every app has “pending users”, but the relationship that active users have with pending users may vary.
So we decided to build functionality to allow our active users to mention pending users in Yammer posts. The pending user would then receive an email with the contents of the message and an invitation to join Yammer. To go into detail about the feature, let me explain a bit about our product development approach.
Feature Goal
As mentioned, it’s important that our product helps you get work done with colleagues. Mentioning colleagues helps ensure your message reaches the right person and gets you a faster response. It also grows your Yammer network, which benefits from Metcalfe’s Law — the idea that a network grows in value exponentially as each new member joins. This helps promote all of the other benefits that Yammer has to offer, like serendipitous discovery and more flexible collaboration. By extending these benefits to include pending users, we hoped to improve the experience for everyone.
Product Design
We worked hard to figure out what to change in the product. One variable is how to show pending users in the user typeahead (the dropdown that appears when you start typing a name or @mention) — do we separate out pending users or do we integrate pending users into the list of active users to make the experience seamless?
Another variable is the hovercard for pending users — should we call out that they are not yet on the network and provide their active Yammer colleagues with a chance to invite them?
Caring for our Customers
We realized that some customers may be concerned about seeing people not yet on the Yammer network in the mentioning typeahead. But since the typeahead intentionally only includes trusted members of the company to ensure secure communication, we decided it was important to give users the opportunity to reach out to them through Yammer.
We also wanted to make sure that pending user mentioning didn’t come at the cost of active user mentioning or any other part of the Yammer experience. We kept an eye out for this during our experiment by measuring user behavior. We could tell with statistical significance whether or not mentioning pending users had any negative externalities, such as driving down active user mentions.
Experimentation
We had 4 combinations of the feature along with our experiment control. So we rolled out each version to 20% of the Yammer user base and collected data about what we care about: global metrics like engagement and virality, and the local metrics like user mentioning, which are often correlated with global metrics.
Wow! Data. We all had our own biases and hypotheses about which versions would perform well, which is one of many reasons why we leverage product experiments, and leverage data like what you see above.
Interpreting the Numbers
How does one make sense of all this? The integrated typeahead with the existing user hovercard actually got us a 173% bump in pending user mentions compared to 163% for the integrated typeahead with the pending user hovercard, which ended up being our winner.
Why would we pick a variation that didn’t drive up pending user mentioning the most? Well, consider the experience. In both groups, the user is seeing an integrated typeahead. So, the data suggests that the difference in the hovercards would somehow be responsible for the difference. That seemed implausible, so we trusted that the integrated typeahead was clearly better for pending user mentioning than the separated typeahead, which led to 123% and 106% increase in pending user mentions. Those gains are good, but not as good as we saw with the integrated typeahead.
We also kept an eye out for effects on active user mentioning and posting. We found that people posted about 2% more threads with the integrated typeahead / pending user hovercard version. Also, as you see in the chart, this variation bumped active user mentioning by about 7% as opposed to just 4.6% with the other integrated typeahead or nothing at all with the separated typeahead.
And how do we explain not choosing the variation (separated typeahead, pending user hovercard) that may have yielded about a 2% bump in viral conversions (new users that come through these mentions)? Well, for one, that variation meaningfully underperformed the other variations on other metrics we also care about. But it also may have hurt engagement and active user mentions, and those were costs that we wouldn’t accept for the possible virality bump.
Moving Forward
Mentioning pending users has drastically reduced communication barriers within Yammer, and is available on all networks. We have data that gave us new insights on how people use our product and what they expect. We will be using what we learned from this experiment to continue to design, build, and measure new product ideas that can further break barriers and unlock new workplace communication.
Have you tried the new mentioning pending users feature? Have questions about our development methodology? Leave a note here on Medium to share your product experiences.
If this sounds interesting to you, check out our jobs and discuss this on Hacker News.