Testing Yammer’s Signup Flow
Follow Christina Lucey, Product Manager at Yammer, as she A/B tests a shortened version of Yammer’s signup flow.
When we interview Product Management candidates at Yammer, we sometimes ask them how they would change our signup flow and why. The most frequent response we get is that we should remove the steps that seem unnecessary so there will be less friction. Seems like that would be good, so let’s give it a try.
Take the quiz
Want to exercise your Product Management muscles? Take the Shortening the Signup Flow Quiz (est. 30 minutes) before reading the rest of this post.
Christina’s Hypothesis
Christina Lucey proposed removing two steps of the signup flow and replacing them with in-product callouts. She hypothesized that moving steps from the signup flow into the product would more gradually engage new users in the product and increase new user retention.
The idea here is pretty simple. When a person is in our signup flow, they haven’t even seen the product yet. You can imagine how they might be turned off by our request to have them, e.g. upload a profile photo and join some groups. If we ask them to do these things after they get a chance to interact with the product a little bit, they’ll be more likely to stick around. Also, we’ll have fewer steps in the signup flow, so more users will make it through without giving up.
With that hypothesis in mind, here is how Christina structured her test:
Variation 1
Remove the “Upload a profile photo” step of the signup flow and add a big callout to the home page that prompts a user to upload a photo.
Variation 2
Remove the “Join groups” step of the signup flow and add a “Suggested Groups” section to the left navigation links.
Variation 3
Remove both steps of the signup flow and add both in-product features.
Control
The original 4-step signup flow with no new in-product callouts: (1) user fills out some profile information, (2) user follows/invites colleagues, (3) user joins existing groups, (4) user uploads a profile photo.
Test results
Well that didn’t go as expected…
For new users in the test groups, retention was down, less users posted, and less users engaged overall.
What did Christina ship?
Based on the test results, Christina shipped the in-product callout prompting users to upload a photo. Christina didn’t ship the suggested groups module or cut any signup flow steps.
Why didn’t she cut any signup flow steps?
The purpose of cutting signup flow steps was to get more new users through the flow and into the product. Although two of the variations achieved this goal (remove group join and remove both), Christina saw that new user retention was down pretty significantly in both cases. There was a slight improvement in the number of days engaged per user per week, which is an important core metric, but it’s tough to swallow the bad new user retention numbers, a 4% drop in users that posted, and a 34% drop in profile photo uploads.
Explanation: After going through the shorter signup flow, users in the test groups landed in the Yammer product with either no photo, no groups, or neither. They were less invested in Yammer than the control group, less likely to perceive the product as valuable, and thus less likely to post a message, return to the product, or engage with the product at all.
But what about the increase in users that sent invites?
Good question. Christina and an analyst dove deeper into the data to see if the increase in new users resulting from those extra invites was enough to offset the drop in new users retained. They found that shortening the signup flow resulted in less cumulative new users being retained, so she decided not to ship the shortened flow.
Why did she ship the photo upload callout?
The number of existing users that uploaded a profile photo was up 9%, which is what was supposed to happen. Since existing users have already gone through the signup flow, we don’t have to worry about their retention being affected by the shortened signup flow. As an added bonus, there is a (barely statistically significant) boost in the number of people invited by existing users that signed up for Yammer.
There is the troubling matter of the 6% drop in the number of groups created. Christina accepted this tradeoff because more group creation is not necessarily better for a Yammer network, but our intuition tells us that profile photos are good for engagement because they invest users in the product.
Explanation: The photo upload callout is big and steals focus from the other stuff in the left rail, like the group creation control.
Odds and Ends:
What about the metrics/results you didn’t mention?
Either we didn’t think they were that important or their p-values were too high to be considered statistically significant. As I mentioned in the quiz, we generally ignore results with p-values higher than 0.3 and we’re skeptical of results with p-values higher than 0.2. Scientists usually won’t consider anything above a 0.05 significant, so we’re actually being pretty liberal.
Why didn’t Christina ship the Suggested Groups module?
The feature did the opposite of what it was supposed to do, causing an 18.5% drop in group joins. Although there is a slightly interesting increase in the number of users that posted, that’s not nearly enough to justify keeping the feature.
TLDR;
How could reducing the number of signup flow steps to reduce friction be bad? From this experiment, we learned that the signup flow is a really good place to ask users to do things that invest them in the product, even though they haven’t even seen the product yet. It’s counterintuitive, but the behavior of users often is.
Something else I hope this post illustrates is that A/B testing is not just for helping you choose between 41 shades of blue (and crushing designers’ souls in the process). A/B testing is best used to evaluate the results of risky product changes.
If you liked this post, check out my colleague Drew Dillon’s post about why data is not your enemy and why you should not fear it.
