ScheduleOSUpdate: Hitting a Moving Target

Frank Yang

Published in

Fleetsmith

10 min readOct 24, 2018

by Frank Yang, Fleetsmith Product Team

On the surface, the act of triggering an OS update is very simple.

There is a new version of an OS available.
You have a device that needs to be updated.
You trigger the update.
You repeat steps 1–3 for your fleet, or see if there’s a way to do 1–3 in bulk.

However, if your experience has been anything like ours, you’ve come to realize that triggering OS update isn’t always as reliable as it should be.

Last month Fleetsmith launched beta support for iOS and tvOS devices, and as part of this beta support, we’ve extended OS update enforcement feature (and our other core management features) to iOS devices. Our pursuit to uncover the truth behind OS updates was full of traps, false prophets, and red herrings, but we made it through to the other side, sanity (somewhat) intact.

If you’ve ever wondered why pushed OS updates for iOS sometimes work and sometimes don’t, keep reading; join us on this wild ride.

What is it? What is it used for? ScheduleOSUpdate is the MDM command that an MDM provider sends to iOS devices to trigger an OS update, if one is available. That means if you can push an OS update through your MDM provider’s web interface to your iOS fleet, odds are it’s accomplished by sending this command to the device.

The Mobile Device Management (MDM) protocol provides a way for system administrators to send device management commands to managed iOS devices running iOS 4 and later, macOS devices running macOS v10.7 and later, and Apple TV devices running iOS 7 (Apple TV software 6.0) and later. Through the MDM service, an IT administrator can inspect, install, or remove profiles; remove passcodes; and begin secure erase on a managed device.
— MDM Protocol Reference

How does Fleetsmith use it? Fleetsmith allows customers to set enforcement dates for new iOS versions. Once the enforcement date has been configured in Fleetsmith, we automatically trigger the OS update on out-of-date iOS devices using the ScheduleOSUpdate command. For macOS devices, we currently use the startosinstall tool, since it allowed us to deliver a better user experience and it was recommended by Apple in the “What’s New in Managing Apple Devices” talk at WWDC this summer.

To maintain the same experience that we provide with macOS, we need to manage the upgrade with minimal involvement (from the admin and the end user) and still guarantee a successful update. And to achieve that, we needed to understand the nuances of how the command works, what happens on the device depending on its state, and which iOS versions have support.

Chapter 1: Trust the documentation

ScheduleOSUpdate is a one way ticket.

It’s impossible to find an OS update that allows you to revert back to a previous version, and this makes total sense. New security features to an OS are not “simple upgrades” that should be reverted, and to help prevent this, Apple restricts all OS installs to signed versions only. Apple has intentionally designed this for security reasons. If downgrades were possible, an attacker who gains possession of a device could install an older version of iOS and exploit a vulnerability that’s been fixed in the newer version.

For end users using an MDM provider, this is great. The MDM will trigger the OS Update when the time comes, and the device is updated.

However, as an MDM developer, we want to be able to test MDM commands thoroughly, so we can provide the most pleasant experience possible for our customers. Unfortunately, this MDM command isn’t available in Simulator (Radar: 45414745), and without the ability to simulate an upgrade, the only way to test the behavior of ScheduleOSUpdate is to trigger an update on a real device.

And for a command that can only be executed once on each device, we are suddenly left with a discrete, finite number of times we can test the behavior.

Being that each test is irreversible, how do we even set our expectations and architect a solution? The only choice is to trust in Apple’s documentation of how it should work, and cross our fingers trusting that it will.

Buckle up, friends! We have a long way to go, with the documentation as our guide.

Chapter 2: The documentation is incomplete

Bugs happen; this is the natural state of any software product. It’s nigh impossible to ship something that never needs to be updated again, specifically at an OS level. So it’s safe to say that bugs are expected in MDM commands. However, bugs in an MDM command that create an irreversible change in the test device makes things… more difficult, to put it lightly.

Example:

MDM providers can send ScheduleOSUpdate with a few options: Default, DownloadOnly, or InstallASAP. Due to the strong permission system on iOS, the device will not install the update unless the user has entered the passcode for the device (if one exists), so we learned that InstallASAP does not actually force an upgrade immediately.

Now, for more context, ScheduleOSUpdate only returns with a status of Idle, Downloading, UpdateAvailable, or Installing.

Unfortunately, there’s no explicit state for a device where the user has entered a passcode and scheduled the device to update (note: scheduled installs occur between 1AM and 3AM). There is no documentation as to what happens if more commands are received during this time.

For those that are curious, apparently this falls under the status of Installing. However, if you send another ScheduleOSUpdate command at this point, the OS update begins immediately. This behavior is undocumented. (Radar: 45414727).

So how does this affect our testing? Each test becomes a full-fledged clinical trial.

Each test device must first qualify for the test:

Prerequisite #1: Device must be MDM-enrollable to receive the command

On iOS 10.3 and later, supported Software Update commands require supervision but not DEP enrollment. If there is a passcode on the device, a user must enter it to start a software update.
Prior to iOS 10.3, the supervised devices need to be DEP-enrolled and have no passcode. On Shared iPad devices, these update commands are not available when any user is logged in. The MDM server must have the App Installation right to perform these commands.
— MDM Protocol Reference

Prerequisite #2: Device must be running at least iOS version 10.3.

Devices on versions earlier than this do not qualify (unless they are DEP-enrolled… which is unlikely).
Devices on versions earlier than this cannot be upgraded to any other version except the latest version, because only the latest version is signed.

Fun Fact! If you take any device that’s older than 10.3 and plug it into Configurator to enroll it, you will be prompted to update your device FIRST before doing anything else. And remember, you can only update to the latest version, because that’s the only version that’s signed.

Prerequisite #3: Must NOT be version 11.3, 11.4, or 11.4.1, all of which have known bugs with ScheduleOSUpdate.

Examples:

For those keeping score, since 11.3 and 11.4 both have known problems, this means you can only use the command on devices running one of the following iOS versions (11.4.1 being the latest at the time):
- 10.3.x
- 10.11
- 11.2

Each test must be run with carefully documented starting states, outlined test steps, and recorded observations.

The following properties of the device must be documented:

Device model (iPhone model, iPad model)
OS version
Disk space available
Battery charge level
Battery status (connected to power or not)
Network availability (Wi-fi, cell network)
Passcode enabled or not
MDM command sent or not
Recorded MDM response

We originally thought we had enough test devices in our lab to run repeated tests. After going through the qualifications checklist, we found we were left with only 5 devices (all different in some way). Five devices meant we had exactly five end-to-end tests we could run to test this MDM command. We ordered more test devices from Apple, but had no guarantee on which version of iOS would come preinstalled from the factory.

“I’d like 4 iPhones, please, but can you make sure they come pre-installed with 11.1, and 11.2, please? Hold the 11.3s. Oh and no iOS 12.1s, either, please. Out-of-date, but not TOO out-of-date.”
— Me ordering new test devices

And this leads us Chapter 3, a sobering realization:

Chapter 3: Only trust what you (and other developers) can see.

As a result of our journey thus far, we learned to roll with the punches and, more importantly, only trust what we could see.

Observation: Formatting of MDM responses may contradict what is in the documentation, depending on the version.

Example:

The OSUpdateStatus response includes an IsDownloaded key which is a boolean value (according to documentation). Responses from devices on 11.3, however, contain an IsDownloaded key with an Integer value. (Radar: 45414737)

Observation: The documentation only describes behavior for the obvious scenarios, but omits information about other variables that may affect the behavior.

Examples:

After receiving the MDM command, the device begins downloading the update. After the download is complete, the device will inconsistently display a user prompt to update.
Depending on device version, when ScheduleOSUpdate is sent with InstallASAP, the device may never prompt the user to update after downloading said update.
Devices sometimes returns Error response after receiving the MDM command, despite the download successfully starting. (Radar: 45414692)
Battery percentage and charging status on the device may have an effect on whether or not device update is immediate.

The best thing you can do is log the behavior, and hope that you have another device.

We turned to the community to check our sanity two weeks out from our launch date, and we immediately felt validated knowing that others had also had the same experience, and found that ScheduleOSUpdate was ripe with inconsistencies.

To be fair, it’s possible that every bug that we found could (in theory) be fixed with the newest version of iOS 12. But even if they have been fixed, the device must be on iOS 12 and upgrading to iOS 12.0.1, in order to verify correct behavior.

Managing inconsistencies

Are you experiencing inconsistencies with pushing OS updates to iOS devices as well? We’ve learned that your best chances for a successful update is minimize the variables as much as you can. This means:

Make sure your device is plugged in to a power source
Make sure your device has enough space
Identify that it’s not currently on a version with a known ScheduleOSUpdate bug
Push updates during off-hours
Wait for the stars to align

Given the nature of these types of bugs, it’s important to understand that ScheduleOSUpdate bugs are extremely slow to be fixed. Newly reported bugs can only be fixed every other update.

Why is this? Hypothetically, say you uncover a bug with the command during your next upgrade, from 1.0 to 1.1. Note that this is only discovered during the act of updating, so you’re discovering and reporting on a bug on a release that is already one version old. In this situation, if the bug were fixed in the next version, it would be available in 1.2, and (here’s the kicker) there’s no way to verify the fix in production until the next release, because ScheduleOSUpdate can’t be called from 1.2 when there’s nothing to upgrade to. So the only thing to do is wait for the next release, and verify that the fix is applied to 1.2 during its update to 1.3.

So what can we do as an MDM developer? At Fleetsmith, we believe that our customers shouldn’t have to worry about these inconsistencies, and that these OS Upgrade features should just work. As we learn about bugs on Apple’s end, we do our best to add the workarounds natively into our code, so you can rest easy knowing that your devices are secure, regardless of the iOS version.

Chapter 4: So, what can we do about it?

The more Apple is aware of the problems, the more likely these problems will be fixed. MDM providers often have inconsistent behavior due to undocumented behavior, so making sure the MDM documentation is kept updated with known limitations would be a solid start. The fact that these inconsistencies exist is unfortunate, but we want to help surface these issues to Apple so we can help improve the MDM product experience for everyone, regardless of which product you use. If you’d like to help out, please dupe the following radars:

It’s not just iOS that has problems with the MDM commands for OS upgrades, either! If you’re interested in reading about the eccentricities of these commands macOS, check out our friend Victor Vrantchan (groob)’s excellent post over at the MicroMDM blog. And then file radars for those bugs too 🙂

Join the conversation!

At Fleetsmith, the MacAdmins community has been an invaluable resource to us, and we want to give back what we’ve learned. (Shoutout to the #mdmdev channel for always having some enlightening discussions.) We’ll be sharing what we learn as much as we can; hopefully it will be useful for you, no matter what MDM solution you’re using. Come find us in the #fleetsmith MacAdmins Slack channel! We’re friendly folk 🙂

In the meantime, we are currently in the process of revamping our entire Help Center for Fleetsmith users to consume, so be sure to let us know what parts you’d like a little more help on!

References (in alphabetical order)

Originally published at blog.fleetsmith.com on October 24, 2018.