How to Hack User Testing

Alyson Baxter
4 min readMay 5, 2015

This week our company launched a new site and I used Optimizely and Usertesting.com to shrink weeks worth of feedback and testing into 2 days — here’s how we did it.

We were exactly 1 week away from launching our new site — which meant we didn’t have any time to spend on ‘which text works better’ or ‘what color is more appealing’. We focused on just launching on time — so I had to use whatever time I could grab from an engineer wisely.

Setup Optimizely

I am a rare non-technical Product Manager — which can be a disadvantage because I can’t actually jump in on projects and make any changes. Optimizely side-steps this problem by giving me total control over small tweaks without having to grab an engineer to do any work. I worked with the engineering lead on the project and put Optimizely on the site. I then removed any variations optimizely gives you automatically. I wasn’t going to use Optimizely to A/B test because I wanted everyone that visited our site to have the same experience.

Testing

We put our site as-is on usertesting.com to receive our first round of feedback. It was really informative, but also painful. Turns out what we did and what the site was about was not clear at all. Every single tester spent their 10 minutes talking about how confusing our site was. I shared the videos with the rest of the team so they could hear the feedback.

Marketplace Homepage Version 1

We came up with a bunch of ideas for what to test — change the hero text, add a ‘How it Works’ section, add a pop up explaining what subscriptions were, etc. We started small, changing ‘Find’ to ‘Shop’ — hoping that would direct customers a little better. Our goal was to make it clear that this was a site you could buy stuff from — changing find to shop would hopefully make that clear.

Marketplace Homepage Version 2

The results were great — people started to understand that this was a place to buy stuff. Now that we had a better jumping off point we started to notice a few more issues:

  1. 2 of the 3 testers thought we sold hot sauce (we don’t).
  2. All 3 testers could not find our ‘shop categories’ section

Based off of this feedback we made a few more changes. We removed the text, small images, and title for the featured subscription and also updated the navigation bar so that it was easier to see.

Marketplace Homepage Version 3

After making the navigation bar much more obvious, adding a categories section to the bottom of the homepage and adding a little ‘featured subscription’ section to the header — we finally had a great go-live version of the site.

What we learned

  1. It is really easy to over-correct through one round of user testing

After our first round of user testing we were really discouraged that no one could figure out what the site was about. If we had only done one round we would have launched with a giant ‘How it works’ section, a pop up on what subscriptions were and many other fixes that would not have solve the problem. You have to keep making changes and keep testing. One round of testing is never enough.

  1. User testing is hard — but you can’t ignore the feedback

It’s tempting to respond with ‘well they’re not our ideal user’ or ‘these users aren’t prequalified’. That kind of thinking prevents you from being able to learn and make your product better. Ignoring user feedback can easily make you go from great idea to useless product. Even if you’re not going to use some of the feedback you have to be willing to listen to it.

3. User testing helps prevent needless arguments

Arguing on behalf of the user is exhausting and unnecessary. When you’re testing often you don’t have to guess and hope your right — just test it. If there’s a disagreement about what makes the most sense for the user just test it and stop guessing.

4. The product is never finished (and testing never ends)

There isn’t a point where whatever you’re working on is complete. You might stop for a little while and come back to it but the product needs to evolve and constantly testing can help with that evolution. Expectations change, technology changes, markets change — what a user said a year ago might be irrelevant today. You have to be willing to force your product to evolve with your users because it does not happen naturally.

We did 7 rounds of User Testing in 2 days using Optimizely and usertesting.com — including the dev time to make these changes permanent it took about 4 days total. I think we saved around 3+ weeks of dev time and painful real time feedback by going this route. We still have a lot to learn about our site that we’ll gain from real customers but we won’t stop testing.

--

--

Alyson Baxter

Lover of all things tech, podcast enthusiast and sql ninja. I run Product at Cratejoy