How we tested iOS 12 Beta

Part 1. Manual testing

Published in

Bumble Tech

8 min readSep 20, 2018

I want to share why it’s important to test betas, how we do it in Badoo and what interesting cases we found during our testing of iOS 12 Beta.

Why do we test betas?

Every September Apple releases the new version of iOS. And every WWDC they boast that iOS users update to the latest version much faster than Android users do.

This happens not only because iOS is genuinely awesome, but also because Apple knows how to force their users to update:

Changing the position of the desired ‘Remind me later’ button in an otherwise identical looking update dialogue so confused users may accidentally tap the update button.
Disallowing downgrades.
Showing the red counter on settings.
And — my favourite — showing the update pop-ups daily.

BTW: There was a very annoying bug in the beta version showing the update alert every time the screen is unlocked or the notification centre is dismissed. However, no update was actually available.

This causes our users to update at the speed of light so we need to be prepared.

As you can see from the chart above, in August 2017 only 1.05% of our users using iOS Beta 11, but by the end of October the figure had risen to around 60%.

Personal example:
In September 2017 I updated my personal device to the brand new iOS 11 and the email application I was using a lot stopped sending pushes. Yes, you read right: “Stopped. Sending. Pushes”. There was never any way of knowing if I’d received any emails without opening the application and checking my inbox.
It was so frustrating, I reported a bug to customer support and they fixed it. It took them 3 weeks!
After the fix, pushes starting coming through again, however the counter on the app icon was not being updated. Rather annoyed, I reported this next bug.
Long story short, I started using a new app instead of the old one (I still use the old one, but it is no longer my main email app).

Mobile software changes so fast and we need to keep up with it. This is why, when a new iOS Beta is released, we need to start testing it almost straight away.

From which beta version do we start testing?

This year we started testing from beta 6. Why not from beta 1, you may ask. Well, it is pointless starting from the very beginning because betas are quite raw. You may encounter various bugs but they will get fixed by Apple early on. We saw this — loads of issues were fixed by Apple between beta 6 and 9.

Example:
We experienced a crash when scrolling in the grid view with video previews. It was reproducible in beta5, however it was fixed by Apple and no longer happened in beta6.

What is the process?

Step 0. Install the beta version on your device.
Have a look at the list of devices that support iOS 12. The minimal requirement for iPhones is iPhone 5s, the top now is iPhone XS Max. So, make sure you order a new device to add to your device collection if you haven’t yet. I would also recommend performing checks using your own most popular device statistics.
Once you’ve selected your devices you can install the beta version. Here is a simple guide how to do it.
Step 1. Start with new features.
Go to www.apple.com/uk/ios/ios-12-preview/ where you’ll find the list of new features. Carefully read what’s there and select the ones that are applicable to your application. In our case, they were notifications, screen time and performance improvements.

Step 2. Manual regression.
Regressions are not usually my cup of tea, but Betas make everything look promising. With freezes and crashes here and there you get this pure startup feel while testing.
Tips:
- Make sure you use the latest released version as a reference (iOS 11.4.1 at the time of writing this article).
- Keep track of all the issues you encounter and pray that most of those will be fixed by the mighty Apple. And, of course, those issues will need to be retested whenever a new beta is released.
- It would be helpful to revisit a list of what was broken in the previous transition from iOS 10 to iOS 11 and verify everything is still working fine. In this section we recommend you pay special attention to payments (Sandbox and other payment providers — both 1-time purchase and subscription), permissions (camera, location, etc.), camera (taking photos and videos, uploading from gallery and previewing), location (online and offline), chat (all types of messages and offline sync) and network (switching between Wifi — mobile Internet).
- Before releasing a new iOS version Apple usually releases the new Xcode version. And of course, you would expect that the version of your app that officially supports iOS 12 to have been built using the latest Xcode. However, you may encounter different bugs in apps built with different Xcode versions. So, make sure you check both.
Step 3. Nonfunctional testing.
We usually use Xcode instruments for non-functional testing. However, if your device has iOS 12 Beta you will have to use Beta instruments to use Activity Monitor, Energy log etc. You can download a Beta version of Xcode here and use instruments as you normally would.

Please note that you need to be in dev program to access Betas and also expect some Beta bugs. For us Total CPU Load and Foreground app activity were sometimes not tracked.

CPU

iOS 12 Beta

iOS 11

Memory

Energy

Our investigation showed there is no significant impact on CPU, Memory and battery usage between 12 Beta and iOS 11.

Step 4. Prepare automation to support beta.
This may include, but not be limited to, installing beta Xcode on agents, updating Facebook WebDriverAgent. More info on this will appear in the next article in this series.
Step 5. Include checking iOS 12 Beta in your testing routine.
Once a Beta is stable we recommend that you include a check on iOS 12 Beta in every testing routine that you do. For us it was new feature, integration and release testing as per our testing flow.

What bugs did we find (both in Badoo and other apps)?

Here are a few worthy mentions:

Crashes on launch — classics. One of the apps that I use for communication with my parents crashed on launch on iOS12. This was fixed by the app developers themselves so after updating to the latest version the bug was no longer reproducible.
Other crashes. While writing this article the application I used for getting the reviews crashed both on resolving comments and on receiving the “Low battery” alert. This time updating the app didn’t help.
Badoo was crashing when attempting to open own profile in SDK version 12.0 because of a misuse of Objective-C enum being initialised with an invalid rawValue. In this case, you get a crash when runtime can’t match it to a valid case in a switch statement. The bug with enum (GIF below) was fixed by our awesome developers.

Freezes when attempting to connect to third-party accounts, i.e. in case of photo upload or verification by a social media account. We had those on Badoo and they were fixed on our side.
‘No photos in gallery’ at attempt to send photo. One of the applications that I use for sending photos to groups of friends has this bug. I am waiting for the fix from their application side. For now, I send photos from the system share dialogue, which I personally find quite inconvenient.
Broken layout for dismissed notifications. One of the apps I use for reading blogs has this problem. Still waiting for the app developers to fix it.
Duplicate permission request on simulator. Well, this one was in our app and our automation team was not pleased. So, we were obliged to fix it so that auto-tests would be both usable and reliable again.
UI bugs with animation, system keyboard. One of the apps that I use for navigation started blinking like crazy when going off track without Internet. Still waiting for the fix from their side. Another banking app has a broken layout after opening livechat from a push notification. Waiting for the fix from the app developers.

Please note, Badoo was tested using app compiled with:

The old Xcode (9) on the latest iOS Beta (ver. 6–12) and GM (iOS 12).
The latest beta Xcode (10) on the latest iOS Beta (ver. 6–12) and GM (iOS 12).
The latest beta Xcode (10) on previous versions of iOS (iOS 11).

Other apps were tested using app compiled with old Xcode (9) in new iOS beta 6–12 (iOS 12).

Some fabulous conclusions

Testing Betas is extremely useful. It helps avoid exposing our precious users to annoying bugs. Some of the bugs were in iOS Beta itself and Apple has already fixed those. But most of the bugs on Badoo were fixed by our own awesome developers. And after retesting we are quite confident that our users will fully enjoy the Badoo app on iOS 12.

It is also quite pleasing to see that other apps are also including fixes for iOS 12 bugs in their new releases. I hope after the official release of iOS 12 all the bugs mentioned above will be fixed, because if they are not discovered and fixed in the beta, they will affect real users once the iOS update is rolled out.

Test betas folks, it will prove rewarding both in terms of self-improvement, as well as from your users’ perspective.