Lessons learned: going from email based MVP’s to an iPhone app

Recently, we completed a family and friends (and friends of friends!) beta test using our new iPhone app. It was a big milestone for us and we wanted to share some learnings.

Before this beta test, we had run 7 prior iterations of Tribe of Five already. But each of those were only email based MVP’s. And this beta test was important because it was the first time that real Tribe members (outside of the team) took the iPhone app out for a test drive!

It was a fun but challenging journey to transform Tribe of Five from an email based MVP to an actual iPhone app.

Our hope is that one of the lessons we learned will resonate and help someone else out there who is going through a similar transition.

The first 7 iterations (some background)

In order to move fast, we heavily leaned on principles from The Lean Startup. And that means optimizing to learn. We knew that the absolute fastest way to learn is by running experiments to understand how the real world reacts to Tribe of Five.

We decided to use email as our platform to launch and learn.

Why email? Well first of all, everyone has an email address so that meant getting started would be a breeze. Secondly, email is infinitely flexible so that allowed us to test different ideas on the fly. For instance, we could choose to send a beginning of the day reminder, a mid-afternoon progress report, or an end of the day inspirational message if and when there was something we wanted to test out. And no code was required!

Our bet was that email was a suitable platform to flush out high level hypotheses and assumptions about the real product we wanted to build later.

The first couple email MVP’s yielded huge learnings (and conclusively validated our hypothesis that strangers are willing to work together and become accountability buddies). The more we learned and de-risked assumptions, the more it showed us where to tweak the experience next. And with each successive test, we saw signposts that we were heading in the right direction. So we kept tweaking and testing!

But by the 5th, 6th, and 7th iterations of the email based MVP’s, we hit a ceiling of what email could and could not do to close the gap on the “real” Tribe of Five experience we envisioned.

Email is a double edged sword. It was easy to work with and to change on the fly. But we all get a lot of email every single day. And that means that some Tribe of Five emails get lost in the inbox, just like other ones from your friends and family do.

Early testers gave us feedback along these lines. And they told us this made them feel like their tribe weren’t as committed and active as they were. Put another way, they didn’t feel a real sense of camaraderie and accountability towards each other.

We felt strongly that the good pressure that comes from accountability was a core part of the experience we wanted to bring out.

Eventually, we saw diminishing returns on each successive email MVP because the feedback kept pointing us towards this same deficiency. And the more we understood the limits, the more we realized we had maxed out email and would now need to evolve the product.

Lesson learned:

• In the beginning of any project, all you start out with are a bunch of hypotheses and assumptions. You think you know what the problem is and the type of people affected. Maybe your guess is spot on, but most likely, it’s not. The only way to validate the idea is by running experiments (aka MVP’s). Each time that you do, you de-risk more unknowns and learn from it.
• Delay writing code for as long as possible. By running 7 iterations over email, we were able to make changes on the fly since there was no code we had to refactor or throw away. Put another way, the cost of change was ZERO and that meant we could make changes whenever we needed. As Paul Graham says, do things that don’t scale!
• We were initially freaked out that people wouldn’t consider using Tribe of Five since there wasn’t an app. But it turns out that hardly anyone cared as long as the “product” (aka our email based MVP) solved a clear problem for them.

Deciding to build the app

We spent a lot of time considering how to move the project forward after the 7th email test. At the end of the day, we’re a small team constrained by time and capacity.

Even though we understood there were high costs to build and maintain an app, we also knew that crossing over into this new medium would give us a whole new playing field to create a better experience.

It gave us access to features like push notifications which we were excited to try out because we had a hunch it could alleviate the problems we kept hearing in our 5th — 7th email based tests.

What to build and what not to build

As we poured over our learnings from the 7 email MVP iterations, we initially sought to combine the “best of the best” ideas all into the app. That meant group chat. And the ability for people to have 1:1 conversations too. And stats. And gameification. And inspirational quotes. And on and on…

We naively told ourselves that since we already validated learnings from 7 prior MVP’s, those features should absolutely make it into the app. Our mindset was that if we didn’t do that, then we’d be clearly be taking a step backwards in the experience.

But as soon we started designing even the low fidelity prototypes, we saw complexities and edge cases that we knew we had to account for. It quickly became apparent that we were taking on too much. We had to start at the CORE of the experience (the daily check in) and get it into a good place before doing anything else.

The byproduct of this realization was that we cut away a lot of the bells and whistles that we thought we needed.

One example of something that didn’t make the cut was a chat feature. As an email based product, it made sense because that feature came for free and it had the potential to help people form deeper connections. But the more that we designed chat, the more edge cases came out of it. And we didn’t have good answers for these edge cases yet.

As we peeled back the reason why we thought we needed chat, it became clear that we wanted a feature that would help tribe members build a sense of camaraderie and togetherness.

But when we defined it that way instead of just saying we wanted a chat feature, other solutions started appearing that would be much easier to build and could potentially solve the same problem.

That decision alone saved us a lot of design headaches and development time too!

We revisited every piece of the design and made tough decisions to trim a bunch of features that we didn’t NEED to build right away (like user profile pages, onboarding experiences, etc) while still allowing real users to test the core of the experience.

Lessons learned:

• Design on paper first. It’s the fastest tool you have and allows you to try out high level concepts without worrying about color, text, beautification, or fancy interactions.
• Seriously test out each part of the design on yourself first. To do this, give yourself a specific scenario to test (ie. “check in using Tribe of Five after reading 40 minutes of a new book”). You’ll quickly realize there are a number of edge cases and error states you’ll need to account for. Even something as simple as adding a new book exposed edge cases that we never considered when Tribe of Five was only an email based product. For instance, now as an app, we had to consider what showed up after a user adds a new book. Should it only be the title, the book cover, or something else? And it made us ask what should happen to the book after a user finishes it? Or they decide they don’t want to read it anymore? The devil is in the details…
• If you aren’t willing to sort through all the edge cases that inevitably present themselves, you aren’t ready to build the feature using code yet. If you do, your users will pay the price and get frustrated each time they hit one of these edge cases. And believe us, they will hit lots of what you initially consider super-edge cases.

Getting the app onto beta tester phones

In order to run the beta test, we had to get the Tribe of Five app into the hands of our testers. We were not ready to submit the app to the App Store though. Simply put, it jut didn’t have enough features to stand on it’s own yet.

Since we optimized for speed of learning, it came at the cost of cutting out onboarding features (we sent a PDF to each beta tester in it’s place), a way add profile photos (we just manually did it for them), a password forgot flow (we had a way to manually reset passwords if anyone needed), and more features that you just expect comes with every app.

To get around this chicken and egg problem, we decided to use Fabric to help distribute the Tribe of Five app.

Fabric has it’s pros and cons, including:

• pro: there is no App Store review process. Whenever you want to make updates and / or fix bugs, you can push out a new version right away. And it’s easy for beta testers to update the app along the way.
• con: there is overhead for each person wants to try out the app. First, you have to send them a special link to download a crash analytics app onto their phone. And in return, the device ID of their iPhone gets sent back to us. We would use that device ID to make a new build and permission that specific user. Finally, after this, Fabric would send out an email that each tester would need to follow to download the actual Tribe of Five app onto the phone.

Lesson learned:

• Map out each and every step your beta testers need to go through to get the app on their phone. Once we enumerated these, we were pretty scared that our beta testers would get stuck along the way. We ended up creating a PDF that had screenshots of each email they would get, and each thing they needed to download. And then we closely monitored each device ID we got back and proactively followed up with beta testers to offer help if they needed.

Actual learnings from the beta test

Ah, yes, that was the whole point why we even ran the beta test in the first place…

Unfortunately, we’re going to leave you on a little bit of a cliffhanger. For now, let’s just say that even though it took a lot of collective brainpower to transform the Tribe of Five experience from an email based MVP into an iPhone app, it was absolutely worth it in order to take the experience to the next level.

On the next post, we’ll share some of the learnings and insights gained from the test results!