We know what our users want…wait, do we?

Part 2: External iOS beta testing

tl;dr: Apple TestFlight is the best solution for external testing. But be careful out there.

In my previous post, I focused on our process for establishing a feedback loop with in-house users. Once your app has matured through internal testing and iteration, it’s time to get some external users into the mix and test your assumptions.

Crashlytics Beta Learning Curve

We were using Crashlytics Beta for all things beta testing in-house early last year. Naturally it would follow that we’d go with the same tool for external testing. Surely it wouldn’t be too difficult to manage external user access to the app as we prepared to launch. Ever tried herding cats?

Here is another dose of iOS beta testing pain. Stop me if you’ve heard this before. We held a friends and family beta testing period for a client using Crashlytics Beta, and the process of provisioning external users was painful.

Team invitations had to be accepted, devices needed to be registered, and device UDIDs needed to be provisioned. Only it was worse because each step was dependent on the previous one and you were completely beholden to each individual user’s schedule and level of responsiveness. Bluck.

Apple TestFlight FTW

As it turns out, Apple and TestFlight have a solution that is *almost* exactly what we iOS developers need for this next phase of learning. I say almost because we have encountered a few chinks in the armor. I’ll come back to that in a moment.

1,000 external users? W00t! A TestFlight app store for external users? YES. You can now invite people by email to become external users and that’s it. They can install your beta app. It is just that simple. The friction is gone. Beta testing is now as easy as installing an app from the app store. The only additional step is to accept the invitation to test from your email.

In order to use the Apple TestFlight with external users, there is a review process to enable an external beta test. I was a bit nervous about this at first but have found it to be relatively painless. The longest that I have had to wait for the external beta review is one day. Most times, especially after the first approval, the process takes mere hours.

But wait, I have to submit for beta review with each update on an external beta? Not to worry, the app will be ready to go in minutes provided that you haven’t made any major changes that require a more comprehensive review.

TestFlight automatically notifies external testers when an update is available so that you can rest assured that your testers will be running the latest version. You can easily see who has installed which version through the iTunes Connect portal as well.

Knowing is Half the Battle

Now for that “almost.” TestFlight external builds expire after 30 days. When they expire, the app will no longer launch.

Let’s say you have an app that’s using Core Data to store data locally and your external user has been using the TestFlight build of your app. If your app is part of a platform that captures all relevant data remotely, this is less of an issue, but for apps that capture sensitive data and strive to keep the data local to the device, this can be a catastrophe.

When the TestFlight build expires, external users will lose access to their data. That sucks. These wonderful people that have agreed to help you and that are excited about your app are suddenly locked out. Why should they have to start over with the official release, when they’ve already been using the app, populating it through their usage? Or worse yet, what if they’ve been populating the official version for months (or years!) and after opting in to provide beta feedback, they suddenly lose access?

What you don’t want your user to do is to delete the app and lose their data. Don’t panic. This is a temporary problem. There is an important step that we have added to the process that can prevent this problem. But it is a workaround, IMHO.

One of the elements of the TestFlight integration with iTunes Connect that excited me was the ability to beta test your app and then submit that same binary to Apple for review. This is exactly what can trip up external users.

When the app is released to the store, the version does not change and so the user’s beta app is never replaced with the app store release. The workaround, then, is to upload one more binary with the build number incremented. Do not update the external TestFlight version to this one. This will ensure that even your external beta testers will see the app store version as an update.

Timelines

But Sean, surely you’ll have that new release live on the app store within 30 days!? And you’re right. As of this writing, the average app review time is 8 (business) days. Nothing like a little trial by fire. As it turns out, you can release the new version on the store and your beta user can continue along merrily using the pre-release beta version until it expires, then install the official version. Just make sure you follow the versioning guidelines above.

Version Gymnastics

Now that it is so much easier to distribute an app before it is available on the store, keeping track of the various versions of the app that you have made available can be a challenge. Often times this will get increasingly complex as new features and functionality are introduced while the current build is pending beta feedback or review for the app store. And you certainly don’t want to end up with a Frankenstein release because of changes piling up.

To help prevent a bloated release, you’ll want to stick to Crashlytics Beta for ad hoc distributions until you have a viable release candidate or are actively engaging external users in a time-constrained beta period. Even better if, after gathering feedback, you can disable the external beta and have beta testers revert to the official app store release.

The reason is not clear until you’ve had external users beta testing a release that you want to get on the store and find yourself ready to move on to completing the current sprint or round of development. You can not add the next version of your app via iTunes Connect until the current version that is being reviewed has been released to the store.

That said, an improvement to the iTunes Connect workflow is that you can now upload the next version’s IPA before you have created the “New Version” of your app in the portal. At least you can stage candidate builds in advance. Even so, the timing of these phases can be a limiting factor to your release cycle and require solid tracking to understand exactly which updates are in the release pipeline and where. Be prepared.

Our Process

We’ve landed on a process in which the first step is to distribute a release candidate to external beta testers via TestFlight. Shortly after, we then submit an app to Apple for review (don’t forget to bump up that build version!).

We take an optimistic approach to gathering beta feedback. As the feedback comes in, we can decide if we should put the brakes on the release that is in beta or if we can incorporate that feedback into the next iteration of the app. This helps to compress the amount of time that a release is in limbo while we continue to make improvements in development, awaiting that all-important external feedback to ensure that we’re not missing the mark.

Our process also serves to minimize the likelihood that we’ll need to go back and work in changes to an older version of the code to then carry forward — I hate having to do this. Sure, git allows us to do this easily but it always feels like moving backwards rather than forwards, and has a way of stifling enthusiasm about the improvements that have already been made for the next release.

iOS Beta Testing Strategy

Not ideal, but overall Apple’s TestFlight external testing is a leap in the right direction. And so without further ado, here is a summary of our approach to beta testing iOS apps and tightening up that user feedback loop:

Internal testing: Crashlytics Beta

External testing: Apple TestFlight

Please do share your own experiences beta testing on iOS in the comments or hit me up on the Twitters — @sean_m_kirk. Horror stories and glorious triumphs equally welcome!