Testing a new design thanks to UsabilityHub… with unexpected results

To my surprise, users preferred the old design. But, why?

Recently, I stumbled on an online User Research service called UsabilityHub, which seemed like a great way to get some early design feedback from a random group of people who have never even heard of us.

UsabilityHub offers a number of different options:

• First click — Find out where a user would click to perform an action.
• Five-second test — Test how memorable the information or pictures in your design is.
• Questions — Ask questions without being accompanied by a design.
• Preference test — Find out which design works best out of a number of options.
• Design questions — Ask a number of questions while the user can see the design.
• Navigation test — See how users navigate through a number of pages.

I decided to start with a simple Preference Test, as I wanted to see whether the new design worked better than the old one. Up until now, I was confident that it was a no-brainer.

New (left) vs Old (right)

The first few test participants backed up my prediction and preferred the new design, and went into detail about how the page was laid out.

• “I like the colors chosen and the way the info is provided in a nice simple yet convenient layout. I feel this one gives just the right amount of info on the initial screen while giving the option to view more details for the villas of interest. Thanks.”
• “Prefer this design primarily because I like the colour combination over the other one. This one has more white which I prefer and comes across a much more fresher look. I also like this one because of the layout. I don’t like the two column approach in the other one. This one easier for me to browse.”
• “It feels a lot more modern. The other option looks quite dated.”
• “It looks light fresh and appealing to the eye, I like how the products are displayed, the ability to save and view or compare which is a great feature. I like how there are savings messages. Overall I prefer the layout and the cleanness of the site.”
• “The layout is more eye-appealing & it’s easier to distinguish the various sections and what they are. However, it would be good to incorporate from the other option: the advanced search option, one or two interior photos, and the “perfectly chilled” tagline.”
• “It looks slick, modern and luxurious. The design is clear and easy to scan.”

So far so good!

However, this early triumph was very short-lived, as the results took an unexpected turn.

new (top) vs old (bottom)

72% of the test participants preferred the old design

It seemed very odd that a design that’s five years old, was the preferred choice. However, when I began reading through the comments about why they chose the old design, there was a pattern emerging.

Here are some of the comments from users that preferred the old design:

• This one showcases the interior space which is often overlooked but is important for any stay.
• The picture is the epitome of chill!
• The fact that the 1st picture you see is this beautiful room with an ocean view is amazing.
• The image at the top is better as well as it is more inviting. It really gives a nice chilled out vibe to what is on offer here.
• This image attracted me more over the other image. This image immediately gives me the look and feel of what the villa looks inside and a taste of the outside.
• I like the 1st picture you see is of one of the rooms. The colors are very soothing. The other pictures are very eye-catching.
• The top photo starts with a picture of one of the suites and a welcome statement to provide more information about the Villas. This makes me feel welcomed and gives me an idea about the style of the suites.
• I like this design better because the first picture draws my attention to how lovely the villa actually is. The view of the ocean is mindboggling. It is a nice introduction by Sarah and Michael.
• it is immediately obvious that this is a villa website. I like being able to see inside one of the villas. gives an instant feel of the quality of the accommodation a user would be likely to find online

Almost every user based their decision on the first image they saw “above the fold”.

It’s clear that people do indeed judge books by their covers, and the content above the fold really does matter. What also seems apparent by some of the other comments, is that some users didn’t scroll past the fold (the bottom of the screen, but not the bottom of the page). The image on the old website takes up the majority of the screen, so it feels as though 25% of the results can, and probably should be disregarded because of this.

Interestingly, there were only a few participants from the UK, but the vast majority of them preferred the new design. The rest of the users were American, and all of them bar one preferred the old design. What can I learn from this? Absolutely nothing, as the website isn’t exclusive to the UK and should appeal to everyone.

Next steps?

So based on the comments I’ve seen, I’ve decided to make the following changes:

• Replace the main image with the one in the old design. Will likely need to adjust the search overlay to fit better.
• Include some interior photos on the property cards.
• Reduce the padding in the promo sections.

UsabilityHub is missing one thing: interests

It was certainly nice being able to choose what kind of user I wanted to target by providing an age range, location, and even income… but one thing I felt lacking was the ability for a user to specify their interests. It’s all very well and good asking a user to put themselves in the shoes of someone who might be interested in your service or product, but this asks the user to make an opinion on behalf of someone they think would like the site, and in some cases you might be targeting a very niche market.

Facebook do interest targeting really well, allowing you to target Ads at people who are interested in certain topics or activities etc.

If UsabilityHub were to implement something similar, I think we’d get much more accurate results.