Experiments in User Testing: Putting “More with Less” to the Test
Last year at Inforum 2016, I led a very successful user testing experiment: for the first time, Hook & Loop built a pop-up testing lab where I and my fellow IAs ran moderated user tests on our respective products. The lab, a free-standing structure with three testing rooms, sat in the middle of the Hook & Loop area. The walls of the lab were designed by our Brand art director; intrigued by the attractive design, many Inforum attendees came over to check it out and ended up taking our tests. We were so pleased with the whole experience, we couldn’t wait to do it again this year.
I wrote about our process of building the pop-up lab, and I included a short list of things that are and are not necessary to build your own. That list included elements like a solid testing script, a strict time limit, and a clearly marked designated location — but not a beautiful prefab structure covered in typography and hand-drawn wireframes. How ironic, then, that when the plans came through for this year, that was not on the list (important company-wide announcements around our AI platform necessitated a reduced presence for other things). Good thing I had written, in my previous post, a plan for what to do when you can’t go big!
Apart from a lack of an actual lab, I also lacked other IAs. This year, my colleagues were deeply engaged in other work, and I happened to be the only IA able to dedicate time to testing at Inforum. I would not be able to share the work of attracting testers with anyone.
Here’s what I did have:
A table, two chairs, and a cinema display
It wasn’t fancy, but it worked. I whipped up a desktop background advertising the tests, and it turns out that’s all that was needed to tell passersby what I was up to.
Five tests of Infor.com
My team had just launched a few new designs on the site, and a couple other designs were in the prototyping phase. That means I had physical objects to test, and I didn’t have to build anything new in order to prepare for Inforum.
Scripts built around discrete tasks
Each test focused on findability of information. I was able to measure not only how many seconds each task took each participant to complete, but also which paths to each task completion were taken most often. The final question for each test asked how easy the tasks were to complete, based on a Likert Scale. The tangibility of the test measurement removed a lot of ambiguity of interpretation, and therefore made the tests quick to run (5 minutes or less).
Two interns
This was probably the secret weapon: I had two Infor Scholars from the company’s internship program who were willing to help. I knew I was better off running the tests myself, being much more experienced at moderated testing. So, I assigned my helpers the task of “fishing for testers.” Not only did they excel at it, they had fun! I was delighted to overhear exchanges between them, as they scanned the crowd, such as:
“Hm, she’s walking very fast, she has somewhere to be.”
“Not him, he looks lost! I’m gonna go get him!”
The end result
Last year, I ran 24 tests on Infor.com. (The entire lab ran 96 tests total.) This year, my goal was to run each of my five tests with 10 testers. I would have been happy with completing three tests. But instead I had 60 participants, and ran all five tests! That’s almost three times the number of participants that I tested with last year. And the data I collected from those 60 tests is currently being used in the development of each of the 5 designs.
The location of the Hook & Loop area was right in the middle of the main floor this year, which no doubt contributed to the increase in participants. But I could not have done it without our Scholars. The hardest part of guerilla testing is getting people to agree to do it. Once a friendly face greets them and explains how easy it will be, chances are good that they will agree to help.
Also, almost every participant signed up for the Beta Tester Community, our community of customers, partners, and staff who are willing to offer feedback on new designs. Through outreach at events like Inforum, and resources like Infor’s customer User Communities, the BTC has grown to over 450 participants. Now, whenever I have a click test, preference test, card sort, or user survey, I send out a link to my BTC mailing list and within 36 hours I’ve got enough completed tests.
Once, I was fortunate to visit the testing lab at Etsy, and I asked how they had managed to build an in-house lab where they did user testing every day. They told me to start small: whatever I wanted to do, just start with a manageable scope and build from there. Starting small means not only that your project is manageable, but that any disasters will be commensurately small. At worst, it will be a learning experience; at best, it will be a jewel of a success that provides the kernel of future growth.
