A/B testing sign-up strategies from the Slack App Directory
Have you ever heard of “The Naked Man”? It’s the infamous “move” for convincing others to sleep with you by getting completely naked, as coined by Barney Stinson of How I Met Your Mother, that “works two out of three times”.
We at MailClark don’t agree with the culture of pressuring people into sleeping with us, but we do believe in the pursuit of scientific experimentation and the accepting of challenges. So, we decided to experiment with a “Naked Man” move for our bot by sending some potential users from our Slack listing straight to our authentication page, instead of our website. Like Barney entirely revealing himself to his date, we offered the installation of our email bot to potential users earlier than expected.
Did this “work two out of three times”? Not exactly.
The Test
For three weeks, we set our “Visit site to install” link from the Slack app directory to send 50% of clickers to the mailclark.ai website, as usual, while 50% went directly to our authentication page, without receiving any more information. We hoped to test whether people actually needed to learn more information from our website, or if we could “Naked Man” them into just going with it and adding our integration. We kept track of how many sign-ups we got from people who arrived at the website (Group A — Site Seers) and people who arrived at the authentication page (Group B —Site Skippers) to see which is the more persuasive arrival point.
The Results
The results are quite clear, there is no significant difference in sign-ups between the two arrival points. There was, after 3 weeks, a total difference of 8 sign-ups in favor of landing on the website; a difference of less than 4% between the two strategies’ totals.
The Conclusion
As the internet is not an overly sterile environment, this “experiment” had several uncontrolled variables, meaning its results are not quite worthy for application to the Nobel Foundation. For example, the button in the App Directory is labelled “Visit Site to Install”, giving the impression that clicking it will send one to a website. If it simply said “Install”, perhaps there would have been a different effect. That being said, the results still indicate that whether Slack sends people to a website or an authentication page, there isn’t a real difference in their response.
It’s nice to believe that some minor change could suddenly bring a lot more users to our service, but the truth is that does not normally happen. The best way to attract more users is still by making a good service that people enjoy using and want to share with their friends, not with slick tricks.
So, we tried to “Naked Man” the internet, but it proved us wrong.
