Don’t Listen to the HiPPO
Neil McCarthy & Grace Vorreuter
At Yammer, we see a world of workers that are constantly organizing and reorganizing themselves into ephemeral, self-directed groups to get work done. Early on, we decided to build a feature called groups and designed it to support these types of groups. We also expressed our opinion that groups should be default open so they would address the siloing problems of email and other legacy collaboration products. However, sometimes people need to collaborate in private. For this reason, we created the private groups feature and introduced a new problem into our product.
The Problem
When users are having a conversation in a private group and need to involve someone who is not a member of the group, what can they do? They can add the nonmember to the group, but that process is heavy and also forces the group to give the nonmember access to every other conversation in the group. To solve this problem, we needed to give users a way to add nonmembers to individual private group threads.
We designed a feature called Any User / Any Thread that let users add anyone to any private group thread, whether or not the new participant was a member of the private group. We call these users nonmember participants, and we wanted to make sure adding them to a private group thread was as easy as adding a user to an email thread.
We built it and proudly shipped it internally to get some feedback (a method called dogfooding). After about a week, our first pieces of feedback started rolling in and a lot of it was… not positive. A few vocal users were concerned that, because we made the experience for adding nonmembers to threads so low-friction, users would accidentally do it. Since accidentally giving someone permission to see a conversation that they shouldn’t see can be disastrous (think “reply-to-all” mistakes in email), we had to take the feedback seriously. But there was something else troubling about the feedback we were getting: the users reporting it were all Yammer executives.
Research
We immediately went into problem solving mode. There’s an extremely long thread in our Yammer network where we came up with potential solutions to these issues.
We were essentially spinning our wheels, throwing out ideas like, “What about an undo?” and “What about a confirmation step?” All of the solutions we came up with solved singular problems and had significant tradeoffs, e.g. added friction for the user or engineering complexity. What we needed to do was take a step back and understand the real problem, and also establish whether or not this was a real problem, or something only our executives at Yammer, who are extreme outliers, faced. It was clear we had to figure out if the HiPPOs—or Highest Paid Person’s Opinions—were shared by the majority of our users. We needed to understand how normal users would experience this feature. So we made a plan.
We were currently dogfooding the feature in the Microsoft Yammer network. Although we could get some preliminary analytics on the usage, that wouldn’t reveal intention. We decided to do some user research to help us make decisions on what design changes we should make, if any.
Hypothesis
Before we start any project, especially a research project, we come up with a hypothesis. In this case, it was “If users mention non-group members inside of a private group, they usually intend to include that non-group member in the conversation.” So armed with that, we started asking questions.
Method
We interviewed Microsoft employees (who don’t work at Yammer) who had recently used the feature. With the help of analytics, we reached out to internal users with various levels of experience with the feature. We set up Lync meetings where the participants could share their screens and show us relevant Any User / Any Thread conversations.
We designed interview questions that would help us understand the participants’ motivations. We wanted to find out if people were making mistakes (unintentionally adding a non-private group member to a private group conversation), and if people noticed the warning message. We then analyzed the results against our hypothesis.
What We learned
- People did intend to notify the person they mentioned; none of the participants had added the outside group member to the conversation by mistake. Some participants didn’t realize that mentioning behavior didn’t always work in this way — they were surprised this was a new feature!
- Many didn’t know why their private groups were private — it was very encouraging to hear that this feature was actually helping them work more openly and was an easy, natural way to bring people into the conversation.
For the most part, people were not noticing the warning message. One participant told us of a debacle where his colleague did make a mistake, had to delete an entire thread, and went through some anxiety.
What We Shipped
We decided to ship a design with a more prominent warning message. We thought this was a good compromise — even though making a mistake with the feature was rare, it would be very problematic if it happened. It could lead people to lose trust in our product. The new warning message is very apparent, but doesn’t create the added friction of a confirmation step.
Takeaways
Do user research during the dogfooding process. Dogfooding has its own biases. The average user doesn’t provide feedback. During dogfooding, you hear from a vocal minority and from users who already understand your product and are more likely to notice the changes. It’s important to identify (hopefully with the help of analytics) and talk to people who have used the feature — they might not have given feedback on their own or ever even noticed that something was different!
Don’t listen to management. I mean, do… but with a grain of salt. At Yammer, we like to remind ourselves “we are not our users” and Yammer executives are definitely not much like our average users.
Don’t freak out. It can be scary to hear upper management strongly disagreeing with your product decisions. Take the feedback into consideration, take a deep breath, make a hypothesis, and test it. Don’t waste time trying to design a solution to each problem you hear about as you hear about it—you’ll quickly drown in unsolved problems.
Neil McCarthy is a Product Manager and Grace Vorreuter is a User Researcher. They work at Yammer. They are very excited for X-Men: Days of Future Past.
