iTunes TestFlight Beta Testing Takes Off
On February 27 2015, just about a year after Apple purchased the owner of TestFlight, testflightapp.com was finally shuttered. Many iOS developers including myself had trouble letting go of the familiar and easy to use service. Now that the friendly schematic of a prop plane has been replaced with a sober message from Apple, it’s time to check out what Apple has been able to do with their acquisition of TestFlight.
In this post, I'm going to cover the basics of Apple’s TestFlight, help you understand how it differs from the old TestFlight, and walk you through a distribution. I’ll focus on non-enterprise distribution, and I assume you have the Technical role or higher in iTunes Connect.
How Things Have Changed
There are some important things you need to understand about how Apple’s TestFlight differs from the one you knew and loved.
With the original TestFlight, you could distribute to as many devices as you had slots on your developer account — which is to say 100. With Apple’s TestFlight, it’s a little different, and mostly in a good way. There are two distinct ways to distribute:
- Internal Testers — This population is limited to 25 users who must be Users on your iTunes Connect account with the Technical role or higher. Distributing to these users doesn’t require Apple approval of your binary.
- External Testers — You are allowed to define up to 1,000 external testers per app. These users do not have any type of role in your iTunes Connect account. The drawback? Distributing to them requires Apple approval of your binary. (Teaser: as you’ll find below, there are automated approvals for minor updates).
Many developers were aghast when they heard the words ‘Apple review’ associated with beta testing. Depending on your process, it’s possible this really is a deal breaker, and you’ll move to an alternative like HockeyApp or Beta by Crashlytics. Along with providing faster turnaround for devices on your developer account (but not iTC account), they have the added benefit of being cross platform — basically they’re the closest you can get to early 2014 TestFlight.
My opinion, however, is that this new setup is a great thing. It allows you to run much better scaled betas of multiple apps without littering your developer account with device IDs of testers (and yes, running out of slots before your annual purge). And while preliminary reports — including my experience — are of beta reviews taking less than a day, you ideally shouldn’t be pushing updates that quickly at the beta stage. Give those bug reports some time to accumulate! For more rapid distributions with early stake holders, you’ve got the great option of internal test deployment.
Another negative to many is the fact that Apple’s TestFlight only supports test devices running iOS 8. While it may be an effort to push forward adoption of the latest OS, it seems a bad spot to push. Productivity or enterprise tools, for example, would benefit greatly from larger betas but often need to serve a wide audience not known for early adoption. Those working on legacy apps are likely going to look elsewhere for beta distribution.
TestFlight has always been braindead easy for testers and developers alike, but integration with Apple has made this even smoother. Most notable is the fact that you no longer have to mess with UDIDs. You simply invite a tester and Apple sends them an email with a link to download the new native TestFlight app. Once they install it and accept the invite, they’re set up and ready to go. Future updates are also made easier by push notifications that take them right to an Update button.
One last point of distinction is in what an invite really means. In the original TestFlight, you invited testers to your team, and pushed individual builds to them. With Apple, you invite users to test your app. Essentially, everyone with an Apple ID is already prepared to accept an invite, so there’s no concept of registering them with your account.
Now that you have a good feel for what Apple’s TestFlight does, I’ll walk you through the distribution paths for internal and external testers.
Distribution
The first step in distribution looks exactly like app store distribution.
- Create an entry for your app in iTunes Connect, if it doesn't already exist. You can fill in minimal data at this point — just enough that you’re allowed to create the App.
- Create an archive with an app store distribution profile.
- With your archive selected in the organizer, hit Submit, just as if you were deploying to the store. Note: Make sure the profile includes the beta entitlement (listed as beta-reports-active), which it should, as long as you created it after Apple launched iTC TestFlight.
- Hit Submit again to prepare and upload the archive.
Once it’s successfully submitted, you’ll want to head over to iTunes Connect to take a look. From the landing dashboard, select My Apps and you’ll be able to see your app with the Prepare for Submission subtitle below it.
Select your app, then navigate to the Prerelease tab and the Builds tab nested below it. You’ll see your build listed under a section titled Uploaded initially, then under your Version number with a status of Processing. This is typically the case for a few minutes, but it has been known to take a half hour or longer. Once the binary has been processed and passed automated validation, you’ll see something like the screenshot below, with your build number appearing under the associated app version.
Up to this point, nothing here is new unless you haven’t submitted an app since XCode 6 came to be. Hold onto your butts, because that’s all about to change in the next section!
Inviting Testers
Before you can send a build for testing, you need to set up some testers. If not already there, go to your app listing in iTunes Connect, then the Prerelease tab, then the Builds tab one level below. As a friendly blue box has likely informed you, you need to enable TestFlight by flipping the switch labeled TestFlight Beta Testing just to the right of your app version. Do that now — you don’t want to make the blue box angry.
Now you can proceed to set up testers. Go back to the Dashboard of iTunes Connect, and this time select Users and Roles. Select the TestFlight Beta Testers tab and you’ll start off with the Internal users sub-tab. Whether or not you plan to distribute betas to internal testers with TestFlight, it’s a good place to start because it allows you to send invites immediately.
Internal Testers
The list you see minimally contains the Apple ID you are logged into iTunes Connect with. Beyond that, you’ll see up to 25 users that have either Technical, Admin, or Legal roles on your account. If you need to add someone else, check out the iTunes Connect Developer Guide.
If you have the Admin or Legal role, you’ll be able to tick the checkbox by any user you want to add as an Internal tester on your account. If you are Technical — you can only enable yourself. To do that you select your Apple ID from the list, then in the resulting view flip the Internal Tester switch on and finally hit Save.
At this point, you’re ready to push a build to Internal testers. Before doing so, take a quick look at the process for External, as it does vary a bit.
External Testers
To invite an external tester, all you need is an email address and name. The address doesn’t even have to be the one associated with their Apple ID — this all gets taken care of in iOS when they accept.
Still under the TestFlight Beta Testers tab within Users and Roles, go to the External tab. Hit the plus sign circled below:
Enter in as many external users as you want on the resulting form. These users will be associated with your account, and can exceed the 1,000 per app limit. You may want to assign the users to a Group at the same time — this allows you to filter user lists when inviting users to test an app. When finished, click Add and then Add again on the following popup.
Distributing
With all of the setup out of the way, the time has finally come to send out some invites to test your app. Because you can see the results immediately, you’ll start by deploying to yourself as an Internal tester.
Inviting Internal Testers
Head back to My Apps, your app, the prerelease tab, and finally the Internal Testers sub tab. Check the box beside your name (and any internal testers you want to deploy your app to), and then click the Invite button followed by the Invite button in the confirmation. You’ll receive an email that looks something like this:
Depending on whether or not you have native TestFlight iOS app installed, the mail will either take you to the app store to download it, or directly to the TestFlight app. In either case, you’ll eventually end up looking at a cell extremely similar to those seen in the App Store. Likewise, after downloading, you’ll see an Open button for launching your app.
On the Springboard, beta installs are denoted by an orange dot appearing to the left of the app name. This is a nice advantage Apple’s beta implementation has over third party solutions where a tester would need to launch an app and check a version number, if available, to confirm whether they’re running a beta.
As you’ve seen, distributing builds internally is quick and easy. It’s also extremely convenient. Beyond the ease of initial deployment, any time you upload a new build, Internal testers you’ve set up will automatically receive a push notification taking them right to an update button.
Inviting External Testers
There are a few additional steps involved in getting a build to external testers. Go back to the prerelease tab for your build, but this time go to the External Testers sub tab. If you’ve added a tester in the steps above, you just need to hit the plus button seen below, and select Add Existing Testers.
Check any tester or Group of testers you want to invite, and hit the Add button in the upper right, followed by the one in the popup. If you didn’t already have testers on your account, follow similar steps, but select Add New Testers and enter in their information while associating them with this app.
Once added, you’ll see the status of that user change to Added. But don’t get too excited — no emails were sent and no apps installed just yet. The time has finally come to learn about Beta Review.
Head to the Builds tab and you’ll now see a link under External that reads Submit for Beta App Review. Hit that, and you’ll see a form very similar to what you see when submitting an app to the store. The only thing that is really new here is the What to Test form, where you need to enter instructions for your testers — look at these as beta release notes. All other fields, aside from privacy policy URL, Demo Account, and review notes are required and should reflect what you’ll ultimately put on the store (don’t worry, because you can change them later).
Once you’re done there, hit Next and answer the standard Export Compliance questionnaire. Once you’re done, hit Submit, and the app is now in the review queue. This first review has taken less than a business day by my experience and other anecdotal data, but this of course may change. It’s also not clear what Apple does in these reviews — in at least one case, the reviewer didn’t even log into my app. Metadata along with the history of your account may be a bigger impact at this point.
Once you receive beta approval, you’ll see a Send Invites link appear under the External user column of your build listing. Hitting that sends emails to anyone currently in the external tester list for that app, and the process from there is similar to the one for Internal testers. A key difference is that subsequent updates do not get automatically pushed to external testers when processed. In fact, they require another round of approvals.
The good news is that subsequent updates can go through automated review if they are for smaller changes. You simply answer Yes when asked if the update is minor, and (at least in my experience), the approval is instant. No word yet on if / when this option gets throttled, but I would avoid abusing it.
Updates
Much of the update process has been covered above, but there are some additional things to know. In the past, especially if using the old TestFlight, you may not have paid much attention to the Build number in your Target settings under the General tab and Identity seciton.
The old TestFlight would sternly warn you about it if you didn’t increment, but it would also just take care of it for you — putting a unique build version in there to push it through. Not so with Apple, where you need to start paying better attention to the Build number if you weren’t already.
Again, it’s for a good reason — you want reports from your testers to cite a specific version and build number of the app you can tie back to for troubleshooting issues. So, when you are ready to submit an update to a beta build, make sure to increment that build number before archiving.
Wrapping Up
Hopefully you now have a good feel for what Apple’s Beta Testing platform is, and how it differs from the original TestFlight. Clearly, Apple needed something like this, and the initial attempt looks pretty good — it helps that they scooped up the prior leader in beta distribution.
There are some added pains here with automated and manual validation needed before you can distribute betas. The positive is that these steps help you catch validation issues early in the process, and they force you to plan out testing and push new builds at reasonable intervals. The fact that quick updates don’t require manual approval is often overlooked, and makes the process flexible enough to work when things don’t quite go as planned.
I still miss the ease of the old TestFlight, and there are gaps to be filled with tools like HockeyApp or Crashlytics Beta. For instance, I can’t send a really cool concept to my designer, or a demo to a potential client with Apple’s TestFlight. Something else is needed for AdHoc builds now, but what we have for beta testing is great, and I think others will agree when they finally give it a shot.
Follow me on twitter @jefframes
