We know what our users want…
Part 1: Experiences with several iOS beta testing options
Tldr: Crashlytics Beta is the best solution for internal beta testing available today.
Let’s say you’ve had all of your internal battles to whittle away at the core concept of your app to discover the minimum feature set. Frustrating at times, but ultimately productive and unifying for the team. Let’s say you’ve nailed down the one thing that your app will do better than anyone else. You’ve established your assumptions, focused on quality, and been pragmatic about this first step towards market penetration. What now?
Will users flock to your app and be compelled to share it with their friends and family? Who knows. Short of having a large marketing budget, there are plenty of challenges for any founder to get the word out about their company and to gain traction. The big day arrives, you open up the floodgates, and…nothing. Crickets. Maybe some support from friends and family that know how hard you’ve been working on your app.
Traction is the most important indicator of whether you’re onto something.
Who is going to use it? Why? What job are they recruiting your product to do because it’s the best out there? Traction underlies many of the topics covered in the usual entrepreneur’s reading. Even a little traction can mean a lot. Is there an iceberg there?
What better way to ensure that you are barking up the right tree than to be able to get feedback directly from users as early as possible? Set out to find users that are in your target market and begin to turn them into your appvocates. In the early stages, this is asking for a lot of effort — it takes precious time to try something new and especially to offer thoughtful feedback.
At Proto, we utilize a lean startup approach. A key aspect of this approach to product development is to get the product into the hands of actual users and learn. Learn, learn, learn. In order to incorporate that learning, we need to be able to iterate with as little friction as possible. Until recently, getting an iOS app that was not yet on the App Store distributed to regular users was no easy task.
At last, we’ve landed on a process that works well. If you’ll bear with me I’ll walk you down the path that brought us to where we are today.
We’ll focus on internal users here. You’re ideally solving a pain point that you are personally experiencing and you are the first line of feedback. What delights you? What doesn’t feel quite right? What feels downright clunky?
TESTFLIGHT 1.0 (meh)
I started using TestFlight prior to co-founding Proto and it worked. This was back in 2013, so not long after the dinosaurs roamed the Earth and of course, before the Apple acquisition. The process looked like this:
- Produce an archive.
- Save the IPA for Ad Hoc distribution.
- Upload to TestFlight.
- Select the appropriate internal beta users.
- Notify them by email.
Done. What I later learned was that for many beta users, getting setup with TestFlight was a huge hassle.
First, they had to accept an invitation to the TestFlight team and set up a TestFlight account if they didn’t have one. This then involved installing a certificate on the iOS device they’d be using. TestFlight did an ok job of holding the user’s hand throughout this process…if you were using the native iOS Mail app and Mobile Safari. Use any other email client (the Gmail app, for instance) and the user effectively had a broken experience.
Next the user had to connect their device. Now to be completely fair, this had to happen no matter which solution was being used at the time. To install an app distributed via one of these ad hoc services, we first had to provision each individual device UDID in the Apple Member Center and associate those device identifiers with the binary package via a distribution provisioning profile. You’d then have the user’s device UDID baked in to a build of the app.
Internal users might be willing to endure this type of friction. External users — not so much.
By way of of comparison, I was exposed to Hockey briefly while working on an iOS app for a large organization, though it had been customized for internal distribution and there was an entire team responsible for managing that part of the process. It seemed to work fine there but my gut instinct was that it was overly complex. For smaller scale and earlier stage companies, the simple (and free) options were the best bang for the buck.
We have several clients to whom we distribute iOS builds for review and progress tracking on a regular basis. We went with what we knew and TestFlight seemed to be doing the trick. Then one of our clients challenged us. TestFlight was too onerous and difficult. Wasn’t there an easier way?
HAPPY ACCIDENT
Flurry for crash reporting (notsogood)
It just so happened that we also needed insight into how often that particular app was crashing, if at all (this particular app was already out there on the store, but without an active crash reporting mechanism). Crittercism had been used in the past and Flurry for analytics. Occam’s Razor held that I should use the crash reporting already available in the app. Hence:
Flurry.setCrashReportingEnabled(YES);
What we found was that Flurry left a lot to be desired on the crash reporting front. The feature seemed tacked on as a “me too” and not really part of the core capability for the product offering.
TestFlight crash reporting (DOA)
TestFlight could have been an option for crash reporting as well, but I had mixed experience with TestFlight enabled in a production app. It never quite seemed to work as expected and there were indeed some stability issues encountered along the way. TestFlight was then acquired by Apple and their production solution, FlightPath, discontinued.
Enter Crashlytics
With those options eliminated, I decided to check back in on a promising crash reporting-focused company that I had evaluated earlier in 2013, Crashlytics (now owned by Twitter). Crashlytics stood out to me because the onboarding experience bordered on feeling magical.
Ok, I installed the Mac app. Cool, it’s walking me through setup from the app in the menu bar (and it looks pretty too). What? It detects when I do things in Xcode? Sweet. What? I built my app with the framework and it *knows*? I’m all set up? Super sweet!
After a yearlong hiatus, I got my MBP set up to use the product again, and the onboarding experience still left me delighted. But wait, what’s this new Beta thing? Ad hoc distribution? Neat. Let’s try turning that on. Ok set. To be honest, once it was turned on I kind of forgot about it. The time arrived to cut the next build for that same client that challenged me to find a better solution.
Ready to get an archive produced and distributed via TestFlight, I got started. I always clean up my Xcode environment first — Product menu, option key, Clean Build Folder… (a cool trick if you haven’t come across it yet). Ok, let’s produce our Archive. I patiently wait as the progress bar creeps along to fill up and as usual, Organizer pops up showing me the latest archive to work with — thanks, Apple!
Hey, what’s this? The Crashlytics app in the menu is asking me something… “Archive build successful. Would you like to distribute to testers?” Heck yeah I would! I click Distribute…, add some release notes, enter a few email addresses and voila! Shipped. 5 minutes to build and distribute a beta release.
To be fair (again), I had also tried the TestFlight Mac app but it didn’t feel nearly as polished. The Beta testing email notifications alone are beautifully designed. They literally put your app in the spotlight and the onboarding for beta testers is fairly robust. Not using the native Mail app — check. They’ve got that covered. Install link opens in Chrome for iOS? Check. It just works.
The initial onboarding for users, including provisioning their device UDID, was still cumbersome, but this was a working solution.
TESTFLIGHT REDUX (Apple flavor)
Fast forward to the public release of Xcode 6 and the integration of TestFlight into the Apple empire. I was eager to give the tight integration with the new iTunes Connect portal a shot. I went all in.
iTunes Connect PAIN
We now have the ability to submit a binary to Apple from Organizer and to then turn on pre-release distribution options. We started with internal users. One limitation that we hit with the internal testing capabilities centers on *who* can be set up as a member of the team. This is more a general problem with iTunes Connect rather than a problem with TestFlight specifically.
For us, we work on multiple projects at any given time and iTunes Connect only allows your Apple ID to be associated with a single iTunes Connect account. This is kind of a bummer because it requires some account gymnastics to get to the core capability of provisioning apps that you are actively working on. The Apple developer Member Center does a much better job of managing this — please, Apple, do the same for iTunes Connect.
The second limitation that we hit with internal testers is that those users are restricted to only the most recent build of an app version. So when we’re iterating quickly on internal builds that are being reviewed, tweaked, and put back out, it is not possible to go back to a previous build. It is often useful to refresh the feel for what *was* there vs what *is* there now.
This was a major point against TestFlight for internal testers and the main reason why we reverted to Crashlytics Beta for all internal app iteration.
Use Crashlytics Beta for testing iOS apps with internal users and to establish a tight feedback loop for improving the overall experience before extending the reach of your beta period.
In Part 2, I’ll talk about where we’ve landed on our approach to distributing a pre-release app for external beta user feedback.
Have a great process that you use? Empathize with the pain endured getting users online with a beta app? Need help with your app? Comment below or hit me up on Twitter @sean_m_kirk.
UPDATE: Part 2 is now available here.
