Testing In The Land Of Wearables: Taming The Wild, Wild West
These days you can’t throw a stick on the internet without hitting a think-piece about wearables. Every day a new story appears about how they’re changing everything from fashion to finance or what the implications are for policy, performance, and privacy.
What you don’t hear much about, however, is the process behind getting the new, consumer-friendly devices to market.
When a new technology emerges, new standards for testing have to be established. Unlike testing a website, for example, which has a clear and well-established set of standards that must be met, testing for wearables is relatively new with much looser, more poorly-defined parameters.
Wearables are a bold new frontier and without clear guidelines and steps to follow, that frontier can feel like the wild, wild west.
Which is why, on a recent project, our Testing expert Sahil Gera took note of some key areas of testing for smartwatches, to help provide some insights to fellow testing enthusiasts who find themselves in the wilds of wearables.
Good luck and happy testing!
Everything is New: The 3 Big Challenges of Testing Wearables
The first challenge to overcome when testing new wearable devices is learning how to use them. Give yourself the opportunity to familiarize with a new device and its general operations, and time to properly research and learn the ins and outs of operating the new wearable. It will make the rest of your testing much, much easier.
As Sahil notes, “Installing an application, for example, has to be researched and understood. The new tech means there are lots of different technical things that you simply do not know about yet.”
Build the solid foundation you’ll be working from as step 1 and everything else will go much, much smoother.
The second challenge with wearables, especially at this stage, is that because every device is new, the tech associated with them is still being developed. In many — or even most — cases the hardware is being developed in tandem with the software.
Which makes testing a new fitness app on a wearable device tricky. You obviously need the device to be in hand, but with multiple vendors developing the various components of the hardware — and the software that runs the hardware — you have to wait for those elements to line-up before you can test even the basic functions of your app.
“It becomes hard to test when hardware components are unresolved elsewhere,” notes Sahil. And when other elements are still “in dev” as well, it’s hard to know where a bug is occurring originally. “The bugs could come from any number of levels or layers of the end product,” he adds.
The best you can do is ensure that, as a tester, you’ve done your due diligence to make sure that the specs are met on your end so that when you finally do get to test, there are as few surprises as possible.
The third major challenge you’ll face is the most complicated. Because these are new devices that introduce new technology, in most — if not all — cases they are being developed in secret, “which makes testing really challenging and creates barriers to effective testing” according to Sahil.
When a device can’t leave the office, for example, it’s hard to know if it’ll still work when someone is out of range on a hike or or if the screen remains visible in direct sunlight.
Solving for these issues can be tricky and aside from some clever work-arounds — inventive office-chair jury-rigging can help with testing certain sensors, for example — it’s largely dependent on the establishment of trust between you and your client partner. But knowing the potential challenges of being under lock-and-key going in can help you call out the potential issues in advance, and possibly get special allowances to do proper testing and prevent meltdowns toward the end stages of a project.
The Baker’s Dozen: 13 Testing Must-Dos
Wearables pose very real challenges, mostly relating to their newness, when attempting to test apps and software. But, as Sahil notes, “It’s going to be really exciting to see how wearables advance their space and the intersections of technology and personal lives more generally.” Understanding how to test them effectively is going to be a big part of your job if you do any testing at all.
Because the category of wearables is ever-expanding, no one guide will cover all the bases. Testing for smartwatches is just one subset, and will have different parameters than testing for a headband or a ring, for example.
But this should serve as a solid primer and guide for things to consider when conducting wearables testing, especially if you’re working with smartwatches.
With that in mind, here are 13 specific areas to watch out for when testing for wearable devices:
1) Voice activation
One of the big changes — not just with wearables, but across the board in tech — is the need to make sure an app responds to voice commands. “If I say open the app,” says Sahil, “it should open the app.”
Test with different people on your team to see how the app responds to louder and softer tones, higher and lower pitches, different accents. Does it have trouble recognizing women’s voices at distances of greater than 2 feet? You won’t know if you don’t test for it, so start testing for as many types of voices as you can.
The big elements we’ve seen emerge from the rise in wearables are heart-rate sensors and body sensors. Wearable fitness devices, in particular, are more or less useless if these sensors aren’t accurately tracking what they’re meant to track.
“It’s important to be sure that the results that you’re getting are accurate and that accuracy is universal across experiences (sitting vs walking vs running/high-intensity, for example) and across wearing spots (higher on the forearm or lower on wrist),” notes Sahil.
But it’s not just sensors that read our bodies that need testing. In fitness based apps and others, GPS sensors are massively important, too. How does the device work in a city with concrete all around, as compared to in the suburbs? And how does a change in body movement, such as travelling in a car instead of on a bike, alter the GPS tracking?
Wearables, in particular smartwatches with bio-sensing components, will require rigorous testing of the various sensors they have in a variety of environments.
3) Ambient mode
Because they’ll be tracking fairly constantly and the size of the device limits the size of the battery, the use of an ambient mode is at least as important for a smartwatch, if not more so, than it is on mobile devices today.
The goal, naturally, is to display the minimum amount required for users and to have that be designed for battery saving optimization. A resting ambient screen is a key element to the design and testing for it is important.
But it’s not just a resting screen that has to work right. “It needs to be tested for every screen,” says Sahil. That means that whatever screen your device is on — tracking a run, displaying a music choice, actively viewing a weather app, etc — it needs to be optimized for battery saving.
Your testing process should be checking that each and every screen the app has available on the device is doing the most it can using the fewest resources possible.
4) Burn-in protection
One of the carry-overs from more traditional testing will be protecting against pixel burning. It’s an area that is too often overlooked as part of the testing process already, but will be even more important with the limited screen space afforded on wearable smartwatch devices.
“We need to account for pixel burning and to make sure the design accounts for that,” says Sahil. “Checking for the slight movement of a static screen to keep pixel issues at bay and following the guidelines laid out by the vendor/OEM to keep images visible over the duration of the life of the device ensures your users get the full use of their device and the apps you’re building.”
The number of gestures available on wearable devices has the potential to dwarf the gestures we’re already becoming familiar with. A simple wrist flick/angle change on a wearable smartwatch that switches the device from ambient mode to interactive mode, or what happens when you cover the screen or wave a hand across it… there are a multitude of gestures to check.
But even more than the ones that have been designed for, you need to think about the ones that might not have been designed.
Wearables are on a person at all times, and people twist, jolt, writhe, shift, fall and generally move about in sometimes unplanned for and unpredictable ways. What happens when a shake gesture — intentional or otherwise — occurs? Does the device maintain its in-progress tracking? Does the screen hold? Does music automatically pause? Smartwatches, especially, will be susceptible to these unexpected gestures as our hands and arms are constantly in motion.
Testing for a variety of gestures is an essential component of wearables testing.
6) Brightness testing
As with almost any product launch these days, apps and wearable devices will come with their own specific branded colours to be used, and that’s something that will impact testing in a number of ways.
“We need to make sure that when the device is exposed to direct sunlight (especially important for fitness-oriented outdoors people), the people using it can see the information. The colours often cannot be changed because of branding, so other solutions need to be found such as an increased font weight, increased brightness levels or other solutions you’ll need to work out with the designers for when users take the device into challenging situations.”
7) Third-party integrations
A growing area of testing for a while now has been integration with third-party software and hardware. Testing for things like how a music platform interacts with your fitness tracking device, or how third-party headsets interact with that music platform when used with your device, amps up the complication of testing a fair amount.
As Sahil found with devices that are still in development, adding in third-party components can lead to further complications when bugs are found, but identifying their origin is difficult or the bug resides in components that aren’t your domain.
“The complexity of testing is raised, because we’re testing to ensure the apps are syncing with our app properly, but it may be a bug the other app needs to fix. So we need to ensure it’s working for our end-user experience to be enjoyable and functional, but we don’t necessarily have the ability to make that happen on our end. It requires a lot of clear communication and cooperation,” notes Sahil.
Whether you’re integrating with tracking platforms, music apps, external hardware or any other of the myriad options out there, testing for third-party compatibility is a big job that should not be neglected.
8) Bluetooth connectivity
Right now, most wearable devices connect to mobile and are very dependent on that connectivity to work. To do this, they rely on bluetooth connections.
Any testing expert knows that bluetooth is a fickle and tricky mistress, but it may be one of the most important things to test for. A wearable device that cannot easily and effectively sync with the mobile device it’s supposed to connect to nor display information to the user accurately is rendered functionally useless. It may be frustrating, but commit to testing this one right.
9) Companion app & micro app connection
In a similar vein to the bluetooth testing, testing for companion app and micro app connectivity is crucial.
For most wearables, there is one version of the app running on the mobile phone (the companion app) and one version running on the wearable (the micro app). Specific settings in the app may be changeable using only one version of the app — like changing a background colour using the companion app, for example — but will impact the app on the other version.
Proper wearables testing will include a chunk of testing to make sure the two versions are talking to one another correctly and responding to one another so that the user has a seamless experience.
10) Call interruptions and notifications
Because wearables are synced to mobile devices but don’t necessarily have all the functions of a mobile, things like notifications and calls can interrupt or even ruin user experience if we don’t take proper care to ensure they’re being dealt with easily.
Sahil recommends testing protocols that include testing for notifications from synced apps and calls, to ensure “that the app does not crash if calls come in or notifications alert the wearer of some new item”.
Preventing app crashes should be a fairly top priority in your testing, so be sure to include testing for it with wearables.
11) Battery death
No one likes it when their battery dies but hopefully you’ve done everything in your power (effective ambient modes, settings to allow for minimal background apps, etc) to keep that from happening. No matter how much testing you’ve done, however, eventually a user will try to use a device without enough battery power and it will die.
Your job is to find out how that impacts the devices many functions and settings.
What if a user loses battery power in the middle of run, for instance? Preservation of data even when a battery dies is a big issue for most users. It’s especially important to those using wearables for fitness tracking, as is the case with most smartwatches and wrist-wearable devices. But no one likes lost data, fitness tracking or otherwise.
Make it a part of your testing plan to drain a battery and see what happens.
12) Permissions testing
A relatively small but enormously important component of *every single app* built for *every single device in existence*, permissions are kind of a big deal. As each new sensor or tracking component is built, these apps need to be explicitly clear about what they do and they need to give the option to opt out.
Privacy is already one of the biggest concerns users have and wearables — devices designed to collect highly personal data, that are reliant on storing it in what to most users is the mystery-realm of “the cloud” — will amplify this. Test for it effectively.
13) Mechanics of items
And finally, the trickiest one on the list to recommend, because we can’t know yet what it even really means.
In the case of a smart-watch style wearable, you’re going to want to test for things like providing an analog view and ensuring it’s accurate and meets all the standards of the rest of the device. In the case of other wearables… well, only time will tell. But make sure whatever basic functions of the item, the mechanical nuts ’n’ bolts of what’s on offer, are working right.
So there you have it. The 3 major challenges you’re likely to encounter and 13 specific things to plan and watch out for. Get testing!
Have we overlooked something important? Want to add to our baker’s dozen with a tip of your own? Comment below ⬇️ and let us know! And if you found it useful, give it a recommendation 💛