To really understand the usability of a particular digital product it isn’t just necessary to observe the user’s interactions with the screen, but also to observe the subtle, often subconscious, behavioural cues that can communicate so much of the user’s experience.
This is a more technical version of an Article appearing on the POKE website. Part one is a replication of that post. Part two drills down further into the decisions made and provides some extra help for anybody who wants to replicate the setup. Part three is a list of all the hardware and software used, complete with links to purchase on Amazon or elsewhere.
Over at POKE you will also find links to download the testing-sled 3D print files and/or collaborate to improve upon the design through Github
Part One
With a series of Apps in the pipeline and a couple of super secret projects down the road, the UX team at POKE have been turning our attention to usability testing. This fits in with our renewed interest in sprint techniques and iterative methodology but goes further in our belief that testing and showing work-in-progress to an intended audience is something that should be done early and often.
There seems to be a bit of a myth out there that testing is time consuming and expensive. We wanted to challenge the notion that testing somehow requires enormous expense, two way mirrors, eye tracking, white lab coats and a king’s ransom. So we set out to select and develop a suite of tools that we could use to perform usability testing anywhere — from a gym locker room (yes, we did this) to a more formalised setting at POKE towers.
To really understand the usability of a particular digital product it isn’t just necessary to observe the user’s interactions with the screen, but also to observe the subtle, often subconscious, behavioural cues that can communicate so much of the user’s experience. A holistic testing kit should therefore allow recording, not just of the screen, but the user’s hands, facial expressions and of course audio.
There are a plethora of software solutions out there to choose from. However we found that most were overly complicated for the scenarios we had in mind. Many allowed cloud storage of the video recordings and online commenting. Features we felt were unnecessary and actionable more cheaply and effectively elsewhere. None provided the combination of observations we were looking for. So we embarked on a well trodden POKE path. We got hacking.
The recording of a user’s hands allows one to observe off-screen clues such as hesitation or aborted interactions. It also allows observation of screen interactions in the absence of our jailbroken iPhone (see below). Production-wise these recordings are usually achieved by shooting over-the-shoulder of the user or with static, rather awkward and unnatural, setups of phone, camera and tripod. The best approach employs using a sled — essentially a camera rig attached to the device.
Developing our own sled became one of the most interesting aspects of the project, progressing from first prototype, which used existing consumer items from Amazon, to latest; a multi-phone compatible version developed with our new 3D printer. In true hack style we have open-sourced the design on GitHub to share what we think is a pretty cool tool for others to use, and for ongoing collaboration and development with the GitHub community.
Having solved the hands, we then turned our attention to capturing the clearest possible feed of the screen. To record a phone’s screen is fairly straightforward but in order to generate a polished and completely unambiguous output we wanted to go further and generate a precise overlay of the user’s touch interactions. Solutions for this exist but only within the walled gardens of 3rd party apps. We wanted to go free-range across the iOS estate and, after a nervous weekend of iPhone jailbreaking, we found our solution with the rather brilliant TouchPosé+
The final task was to bring all our monitoring together. Mixing together 3 video feeds with a separate audio track while keeping everything synchronised is normally the preserve of pro (read expensive) systems. But some lateral thinking led us to the world of CCTV monitoring software. We combined this with a custom desktop background and some screen capture software which resulted in what we think is a low cost, yet slick, dashboard.
Part Two…
Firstly some recognition that this solution relies heavily on lots of work from lots of people, scavenged from across the web. Apologies if this includes you and you haven’t been credited. It’s more that i’m a terrible researcher and by the time it came to write this article i’d lost all the links to my research material. If this is you, drop a line in the comments and if i remember your article (and i will) i’ll credit you.
I also apologise for the unapologetic iOS bias.
The Sled
Now there are way easier ways to do this. Many people have experimented with Meccano, Velcro and other hacked methods. However we wanted something that in and of itself felt professional. Something that we could use in the most corporate or elitist of environments without anybody turning their noses up at it. We also wanted something robust. No wobbles, tearing velcro, slipping magnets or suction pads. We wanted it to be mechanical and sturdy.
First prototype
Our first prototype involved attaching the camera to an iphone case with cable ties. And, you know, if you are testing with a single device such as the jailbroken iPhone this is the cheapest and the best solution. We used this case http://amzn.to/29l5qmy for the following reasons.
- The enclosed/locked design was particularly strong and wouldn’t pop off
- The silicon interior allowed for tolerance — we could drill holes for cable ties and the case could still lock closed
- At £5 you can’t argue
Second prototype
You know when you use your colleague’s laptop for a second and find yourself all fingers and thumbs for some unknown reason ? (oh, just me? #awkward). Well, the same is true for user testing. If possible it’s much better if the user can use their own, familiar device. This is what lead to prototype mark 2. A multi-device compatible sled for all phones sized between an iPhone 5 and 6+. Obviously using a test subject’s device removes the possibility of using Touchepose+ but this is a kit of parts not one solution. 3D printing isn’t the fastest or cheapest option either. but it has the advantage of being a one hit process with minimal assembly. We toyed with the idea of laser cutting but the assembly would be way more complex. However if anybody wants to take on that challenge head over to Github and let’s collaborate. If you just want to download and print off the 3D files head over to POKE.
The iPhone hack
As mentioned above one of the more terrifying aspects of the project was the iPhone hack. We did this so that we could generate those nice little red dots on the screen monitor. Yes, you can do this in UX_recorder and probably others, but the recording is restricted to URL’s that can be displayed within UX_recorder or requires embedding code in your prototype. We wanted something that would work across all Apps and the wider iOS estate. Most monitoring apps also tie you into an ongoing payment mechanic and cloud storage. They are definitely useful for guerrilla testing but we wanted something that was optimised for more of a lab environment.
If you want to go down the jailbreak route it’s obviously at your own risk! The downside is you may wipe your iPhone. Jailbreaks are also only available for specific iOS releases and generally lag about 1 release behind so if you’ve just updated your phone you may be out of luck. However if you want to go there we found that TaiG was the most helpful source for breaking. Yes, we had to pay but not much, and it was the only mac version that seemed to work. We did look at windows too but entered a hell of having to find not just the right iOS version of the break but also the specific windows version. Once broken TouchePosé+ can be found in Cydia which installs itself with the break. This isn’t going to be a jailbreak tutorial but one tip!… We needed to refresh the Sources before Toucheposé appeared in the BigBoss Source everything else was a bit scary but straightforward. Happy breaking.
Monitoring all the video
The underlying principle here is to take your three video feeds: the phone’s screen, the feed from the sled mounted camera and the feed from the camera which monitors your users expression. Arrange them nicely on the desktop and capture your desktop so as to create a single movie where all the feeds are in synch.
The phone
The phone screen is monitored using quicktime (nb. Monitor, not record. don’t click record!). You are only monitoring the screen, we capture later.
The HUE web cam feeds
Using CCTV software was the cheapest solution we could find and seemed to require virtually no system resources. We could also flip the feed from the sled meaning that the phone doesn’t appear upsidedown! We could find no other easy solutions for this.
Capturing it all
Seeing as quicktime ‘movie recording’ and ‘screen capture’ are different functions you can do this using quicktime but it produces HUGE files that you’ll need to compress as an additional process. Snagit produces way way smaller files and gives you a nice user interface to manage them. One tip… don’t forget to add the audio.
Part Three
Part list for sled
HUE HD (black) USB camera
To record interactions with the device
http://amzn.to/29sHoEw
AmazonBasics USB 2.0 A-Male to A-Female Extension Cable (2 m / 6.5 Feet)
To attach the sled to the computer
http://amzn.to/29qwtxH
Monacor 12.2240 Speaker Foam Sealing Tape
To pad the inside of the sled to hold the phone more securely and prevent scratching
http://amzn.to/29py1aO
Cable ties
To attach the webcam and USB extension cable to the sled
http://amzn.to/29gOfRQ
LOCTITE Powerflex Ultra Control Gel Super Glue 3g
To glue it all together (suitable for ABS 3D print filament)
http://amzn.to/29dcv8R
Part and software list for the rest of the kit
+1 HUE HD (black) USB camera
To record users facial clues
http://amzn.to/29sHoEw
+1 AmazonBasics USB 2.0 A-Male to A-Female Extension Cable
To attach the camera to the computer
http://amzn.to/29qwtxH
AmazonBasics Apple Certified Lightning to USB Cable — 1.8 m (6 ft) — Black
You’re going to need to power and capture the phone screen
http://amzn.to/29qz22M
NEXTANY® Professional Skype Audio Sound Podcast Microphone
To record the audio (this is what we went for but the exact mic isn’t important.
http://amzn.to/29sICzy
Anker 4-Port Ultra-Slim USB 3.0 Data Hub with 10W Power Adapter
You’ll be handling a lot of data transfer and powering a lot of devices. A powered USB hub is going to help.
http://amzn.to/29gQbd2
Bean Bag Cushion Holder for Smartphones
Not essential but this will make everything sit neatly on the table for when your test subjects arrive.
http://amzn.to/29HoSa4
Security Spy
To monitor the feed from the sled camera and the user face camera.
http://bit.ly/25UzLNt
Snagit
To screen capture all the feeds being monitored by Quicktime, Security Spy and add the audio feed from the mic
http://bit.ly/1fY6FH9
Quicktime
To monitor the phone screen
https://support.apple.com/en-gb/HT201066
