Product lessons learned from our recent iPhone beta test
In part 1 of this post, we walked through some technical learnings as we evolved Tribe of Five from an email based product into an iPhone app.
And now, we’re very excited to share some product related learnings from that same beta test.
This was a big moment for the team because it was the first time that the outside world took our app for a test drive.
Previously, we ran 7 iterations of MVP’s using email as our main product. Each time, we got feedback, learned from it, and improved the Tribe of Five experience as we headed closer towards product-market fit.
We eventually maxed out what email could do though. And because of that, we decided it was time to evolve.
Going into the beta test, we had hypotheses and assumptions about how the newly built app would resonate with our members. We knew how previous beta testers engaged with Tribe of Five when it was an email based MVP.
But at best, it was an educated guess about how these engagement patterns would change now that the product evolved into an iPhone app.
Optimizing to learn
Some parts of the experience worked out much better than expected. Those elicited very positive responses from beta testers! Other parts of the experience behaved as we expected but also exposed unintended edge cases we will need to solve for.
And unfortunately, some parts fell flat on it’s face…
By optimizing for our team to learn, we didn’t get too stressed out by the things that didn’t work. Instead, we focused our energy to understand the “why” behind each so that we could look for small ways to improve the experience and chip away at the problems that our members cared about.
Coming out of the beta test, we gained a number of validated learnings and deeper level hypotheses that have already shaped the next iteration of the app (to be released very soon!).
So without further ado, here are three product learnings we took away from the iPhone app beta test.
One feature that worked much better than expected: push notifications
Going into the iPhone beta test, push notifications made us nervous because they’ve gotten such a bad rap (see Wired article: “Turn off your push notifications. All of Them.”).
Most companies seem to be using push notifications as a way to try and steal our attention throughout the day on things that don’t really matter. Check out this example of dating apps going too far. (See article: “Why is Tinder trying to be my life coach?”)
Even with all of the macro risks associated, we still wanted to use the push notification feature. As a matter of fact, we had strong opinions that it would become a core part of the Tribe of Five user experience.
Why were we so bullish on push notifications?
Fundamentally, we believed we could use this tool, the same one that a number of social media / dating apps are abusing, and turn it into something that would ADD a lot of value for our members.
Hear us out…
For the beta test, we hosted a 30 day reading challenge for groups of five people at a time (we call this a “tribe”). We used push notifications to help each member in a tribe hold one another accountable to read more throughout the 30 days.
The idea (part 1): when you see other people from your tribe check in after they read each day, it makes YOU more likely to find time and read as well
The idea (part 2): each day that you do read, you feel better about yourself because you are making tangible progress on your reading goals
The idea (part 3): by checking in through the Tribe of Five app, it builds a sense of camaraderie between you and the rest of the tribe members since you are working together in the 30 day reading challenge. This is the icing on the cake!
We’re happy to report that push notifications worked much better than expected. We were actually surprised how positively beta testers responded!
In one use case, it helped Andy (beta tester) literally put down his phone and go read his book. This is what he had to say:
“At random times during the day, I’d be on Twitter when I was bored. More than once, I got a notification from someone in my tribe checking in for the day that they read. It made me close Twitter, put my phone down, and go read my book too.”
This is how accountability works at the core!
And it’s just one of the many positive stories we heard about how Tribe of Five push notifications inspired / motivated / prodded beta testers to read more during the 30 day reading challenge.
Obviously, there is a lot more that can be done with push notifications to make the experience even better. But overall, we’re super excited that we were able to leverage the power of accountability to help Andy and others read more consistently through the 30 day reading challenge.
Coming out of the beta test, we feel more confident than ever that push notifications will be a core part of the Tribe of Five experience.
One feature that worked but also had unintended consequences: reading check in streaks
On one hand, check in streaks offered a light form of gamification and gave tribe members one more reason to check in each day.
At a very basic level, when you have a streak going, you want to keep it going. That’s what powers 36k Redditors who subscribe to theXeffect subreddit.
Overall, this feature worked as intended. We received feedback that the check in streak was simple and straight forward to understand.
It was one part competition. That’s because as soon as you open the app, we showed your current reading streak. And we also showed the streak for each of the four other people in the tribe. This added a natural element of competition between the group going through a 30 day reading challenge together.
And it was one part inspiration too. Ayumi (beta tester), said:
“It was inspiring to see someone in my tribe check in for 15 days straight. If they could do it for 15 in a row, then I really had no excuse why I couldn’t do it too!”
On the other hand, adding check in streaks produced edge cases that we had to spend a lot of time to investigate. We found negative side effects that directly impacted usage of the app.
Cheating was one of the edge cases that we had to learn about. Since it was so easy to check in, and streaks were some prominent each time you opened the app, we were curious if people would be tempted to check in even on days when they didn’t read just so they could extend their streak.
In case you don’t believe people care about their online streaks, read this article by Jessica Lawlor about her experience with Timehop. See article: “I Looked at Timehop Every Day For a Year (And Then I Deleted The App).”
If cheating on check ins was an issue, it would present a whole new problem to verify / validate that a person did indeed read each day. To be honest, we don’t want to take Tribe of Five down this path. It doesn’t feel congruent with our values to assume you are guilty until proven innocent.
Luckily, because the streaks weren’t tied to anything of value (prizes, money, online badges, etc), people did not have a strong reason to cheat.
Tiffany (beta tester) said:
“…it didn’t even cross my mind. Checking in when I didn’t read would just be cheating myself at the end of the day.”
For now, we’ll leave this feature alone. We want people to feel like extending a streak is somewhat important. But we don’t want to give our members a reason to cheat on the daily check in process.
We also saw another negative side effect — what happens after a streak gets broken.
A handful of beta testers told us they read BUT forgot to check in during one (or more) of the 30 days. By the time they realized it, their streak was broken. But, to make it worse, they also discovered there was no way to go back and log in the prior check in.
For most people who cared about streaks, this had a negative effect going forward.
In one edge-edge case, Kyle (beta tester) had checked in for 16 days straight. He was straight up kicking ass!
However, on day 17, he read late at night and fell asleep afterwards. He forgot to check in on that day.
After his streak was snapped, his check in’s significantly dropped off for the rest of the 30 day challenge. When we asked him about this, Kyle told us he felt frustrated that he read on day 17 but couldn’t log it. After that, since there was no way to grow a longer streak than he already had, he lost interest in continuing the 30 day reading challenge.
The main lessons we learned here is that gamification can be a powerful feature to drive behavior change. But it’s also introduces edge cases (and edge-edge cases) that can easily cause unwanted friction and pain in the experience.
One feature that fell flat on it’s face: building trust with other people in the tribe
This was a problem that we thought we could solve outside of the app. We didn’t build any meaningful ways for people in a tribe to get to know each other throughout the 30 day reading challenge.
Our hypothesis was that each person in a tribe would just “get it” that the other people in their tribe had the same goal as them to read more. And since everyone had that same common goal, that would establish a basic sense of trust and camaraderie for the tribe to work together and help each other stay accountable.
Unfortunately, this fell way short.
When we asked beta testers why they thought they were paired together into a tribe, they told us it felt entirely random. And since there wasn’t any way to learn more about other people in their tribe, whatever assumption they started out with tended to be the same assumption they carried throughout the 30 days.
Jenny (beta tester) told us she thought the other four people could have been robots! Even though she could see their first name and a profile photo, she had no way to find out more about them.
And she wanted to! At one point, she noticed another person in her tribe was reading a book she read in the past. This could have been an opportunity for her to connect and build rapport with someone in her tribe.
But since she could not communicate with them, she defaulted to her assumption that it might not even be an actual person on the other side. Ouch.
You might be thinking, “why don’t you just build a chat feature”?
On one hand, we do agree that a chat feature could solve this problem. By giving people in a tribe a way to communicate with each other, we could enable more trust and rapport to develop.
But on the other hand, we aren’t ready to take this unstructured approach yet. We want to build a great product that people love. And that means we really need to understand the potential impacts of each design decision on the overall user experience.
There were just too many edge cases that we weren’t ready to consider yet with a chat feature. For instance, what if one person tries to start off a conversation with the group, but no one else responds? What would that do for the dynamics of the tribe across 30 days? Or what if someone wrote something inappropriate in the group chat? The list of edge cases goes on and on…
Eventually, we might build a chat feature. But for now, we think there are more viable approaches to solve this same problem of building trust / rapport / camaraderie in a way that doesn’t introduce weird edge cases that come along with a wide open chat feature.
In our next version, we plan to add user profile pages with social media identities. Within the 30 day challenge, if you click on someone else’s profile, it’ll show you their Twitter, Instagram, or any other social media profiles they choose to add.
Our hope is that this can help solve Jenny’s use case and help her establish a basic level of trust that the other people in her tribe are real people.
Additionally, with social media links, we give people in a tribe an opportunity to identify common interests and values with other tribe members.
We do want people to form deeper relationships. But we don’t think it needs to happen within the Tribe of Five app. With this user profile feature, we plan to give our members a way to find each other outside of the app so they can connect and go from there!
Thank you for taking the time to read about three of our product lessons learned from our recent iPhone beta test.
We hope that some tiny part of this article encourages you take a second pass and think deeper about the features you are building in your own product.
Just as importantly, we hope this reminds you to proactively look for potential side effects and edge cases with each new feature you launch. All of these will impact how your users feel about your product, how likely they are to continue using it, and how many friends they will ultimately refer to it.
