Testing on Tap: What We Learned, and How to Run Your Own
Last week, we hosted our very first W̶a̶y̶s̶t̶e̶d̶ ̶U̶s̶a̶b̶i̶l̶i̶t̶y̶ ̶T̶e̶s̶t̶i̶n̶g Testing on Tap event, co-sponsored by Drizly. Maybe you heard about it. Well, we learned a lot and have some advice for running your own.
What exactly was Testing on Tap? Wayfair and Drizly set up tables at Wayfair HQ where we ran usability tests on our guests on a broad range of features — but not before letting them get a head start on the beer and wine in our kitchen. Our goal was two-fold:
- Test a wide range of features with users who weren’t quite sober, put them in a more confused and honest state, and cut to the core of our usability issues
- Host a fun networking event for Boston “tech” folks (designers, engineers and project managers)
We met plenty of good folks, talked a bunch about UX, and found a whole lot of usability issues. From Drizly’s alcohol-delivery app, to Wayfair’s Registry, to Wayfair Next’s upcoming augmented reality features, we found plenty of things to fix or improve upon before they ever reach market.
As you may imagine, we plan on doing this event again soon, and want to make sure people keep coming back (and bring more people with them, like a drunk, creative virus). That’s why we also conducted “research on research” at the event, asking our guests how the event itself could be improved.
Between this direct feedback, our observations, and the general planning process, we learned a lot. Want to conduct a “drunk user testing” session of you own? Here’s our advice:
Get help, internally and externally
Like most creative events, a drunk user testing night is too much for one person, and maybe even one organization, to pull off.
Internally, this was a job that involved our entire consumer insights team (shout out to Shelly McArdle, Kelly McManus, and Prescott Titus from that team for their leadership in organizing the event), all of our design teams, and a good helping of project managers, not mention our amazing facilities and A/V teams. As with any usability testing, having more stakeholders involved mean they’ll better understand the value of testing and how you’re getting your results.
Even though we at Wayfair have the numbers to conduct the tests, we knew we would need help from outside resources to get the event better recognized in the industry and ultimately draw more attendees.
We knew Drizly takes usability testing seriously and have a product that really resonates with our audience for the event. That’s why we were thrilled to co-sponsor the event with them, and I think our attendees were happy to give their feedback on a product that many of them use day-to-day.
Meanwhile, our friends at General Assembly and Creative Mornings Boston were excited to partner on the event, provide fun name tags, and include us in their newsletters. These organizations are leaders in design learning, and we could not have had the presence we had without them.
In the end, we sold out all 200 tickets to the event, and at least half of those can be directly attributed to our partners’ promotion efforts.
Get organized and delegate
If you’re the “showrunner,” learn the arts of organization and delegation, fast.
I recommend starting with a Google Sheet where you can keep track of everything and add collaborators. Make tabs for tables, internal attendees, materials needed, event partners, and general to-dos.
Use the document as a tool to delegate. There’s no reason you need to do everything yourself. Find leaders on different teams to organize and fill out the document, so you’re not spreading yourself too thin.
A small price to pay
A topic of much internal debate, we decided to charge a small amount of money for the event — $5, plus the small Eventbrite fee for transactions.
We considered making the event free, thinking that charging nothing would bring more attendees. It turns out that events with a small ticket price may actually bring more attendees than free ones. According to Eventbrite, charging a small amount of money for an event has little to no effect on registration and results in dramatically higher turn-outs.
In our case, attendance was at ~75%, a number we’re happy with for an event of this size.
All of this said, prepare for exceptions. Consider allowing a limited number of free promo codes for your partners to use, especially if you or they will be attracting certain demographics, such as students.
Incentivize participation
A last-minute idea that worked out for us was giving users a raffle ticket per test participated in, which users could put in a bucket to enter to win a Wayfair gift card. Attendees told us that this encouraged them to visit more tables and stay later than they would have without an incentive.
Keep the lines moving
One common piece of feedback we received was that the lines were too long. In response, we’ll want to do two things next time:
- Have more tests
- Make those tests shorter
We had about 15 tests that went 5–10 minutes each, for 150 attendees. If everyone was in line, those lines would be 10 people deep. Even though the lines are obviously not going to be that full, this is still not ideal.
Next time, we’ll aim to double the number of tests and limit them to no more than 5 minutes, guarding our time with (real or virtual) kitchen timers.
You’ll have to do some math here for what might work for your event, but I urge you to keep this ratio in mind when choosing your attendance goal. Even if it means a smaller event, your guests will be happier and you’ll still get plenty of results.
One other thing we’ll do more of next time — send out some floaters with iPads to keep people in line busy. What would be on these iPads? Surveys, one-click tests, and card-sorting exercises are a few excellent candidates.
Treat your tests like normal usability testing, but shorter
Usability testing should be part of your design process for any product or feature. If you’re like us, traditional usability testing runs 30–60 minutes per user, with a remote room full of observers writing things down on a whiteboard.
This is not the time for traditional usability testing. But the core principles are the same, in that you want to get a general sense for where users are struggling to use your feature, ask “why” a lot, and put on your best poker face when they ask if they’re doing it right.
[This is a good time to point out that it’s of crucial importance that you understand how to conduct usability testing if you’re moderating at this kind of event. If you’re a testing newbie, talk to your colleagues who have experience with it, sit in on usability testing sessions, and read this book. Or this book.]
As mentioned above, we’ll be trying to cut our tests down to less than 5 minutes in the future, and we recommend you do the same. That can be tough for larger features, so you’ll really want to cut to the core of what will give you the most valuable feedback.
We also recommend having a moderator and note-taker for each test you’re conducting.
Opinion is split within our team on screen-recording sessions, but I personally think it’s useful for this event where it’s harder to gather feedback in real time. There are lots of tools and techniques available for screen recording, and I’d recommend Silverback 3.0 for those with Macs.
Remember that most don’t like it hot
Wherever you’re hosting your event, try to imagine the same room packed with a bunch of chatty designers, engineers, and PMs networking and drinking. It’s going to get hot in there. Plan in advance if you can open windows, crank the A/C, etc.
Seems like a silly little thing, but it will make a big difference for your attendees.
Keep the beer (and water) flowing
Keep those taps, fridges, and buckets full. When the beer and wine is gone, the guests tend to go with them.
Also, provide food and clear signs to water for those who need to take a breather. It’s important that your guests remain as hydrated as they need throughout the event.
There are plenty more learnings, but these are the big ones that should help you run a smooth event. If you’d like any more tips, have any questions, or if you’re interested in joining our next event (as an attendee or tester), feel free to reach out to me (Bobby).
