“Swiss people have an aversion to maps”, and other things I learned at ProductTank London

The topic for this month’s ProductTank London, the meetup for product people, was user research. Speaking were a trio of experts: a product manager for a user testing service; the CEO of a conversion optimisation agency; and the Head of User Research & Design at the Home Office.

Here’s a few things I learned from the night.

Anything is better than nothing

Morag McLaren, from WhatUsersDo.com, began by presenting several important points.

What stood out from these is that any form of user research is better than none. There are many tools and techniques with a low barrier to entry that we can call upon. This includes face-to-face or guerilla interviews — going out and talking to people. You can knock up some paper prototypes, or send out a survey. Even using Google analytics to answer an hypothesis is beneficial.

Don’t measure happiness with an A/B test

The testing method and output should match, so make sure you’re using the right tool and metric for the job.

All three speakers reiterated that they always find out something new, whatever question it is they ask. Even a negative result to your hypothesis is beneficial — maybe even more so that a positive outcome, and infinitely better than learning nothing at all.

Know your stakeholders and what motivates them

This terrific table from Morag splits examples of outcomes into three categories: Head, Heart and Wallet. By knowing their motivation, you can ask the right type of question to get stakeholder buy-in. It also guides you on the type of techniques to use and data you want to collect. Stakeholders motivated by the heart may be looking for more qualitative conclusions, while financially-minded stakeholders will be wanting hard numbers. They may not be affected by emotion.

This point was backed up by Stephen Pavlovich, of Conversion.com, and Katy Arnold of the Home Office. They say that actionable data is always useful. It helps empower you to challenge your stakeholders based on research. Katy went so far as to say she would be worried if the stakeholder and research went without contest. Get them on side, but challenge their opinions.

Gain rapid insight

Pick a thing to test. Make a change to it. Measure the results. Show the conclusion. Keep it simple.

There is no need to waste resource fully implementing and integrating a feature before testing it. Get your MVP or a prototype together and A/B test it. In every sprint, the Home Office tests with 5 users, including at least one with access needs. This allows them to come across issues sooner, without wasting time developing a product that then needs changing.

Using techniques such as ‘fake doors’ to gauge interest can be used. This involves offering an option which has no functionality behind it. This was used at the Guardian to decide if a popular feature on their app would be useful on their web platform.

This particular technique isn’t only useful for software products. Stephen told how LostMyName offered a new gift wrap option, without being able to fulfil it. Producing and distributing the new product would incur high cost. The experiment revealed that the cost was not necessary, as there wasn’t any interest.

Experimentation lets us peek at the results

Stephen is an expert in experimentation. He believes that, “in a fast changing world, testing is our unfair advantage”. Speaking to people, using these testing methods, and gaining valid, actionable and rapid data enables you to figure out where to focus your efforts and invest your time and money. You are then able to allocate your resource where it will make the most impact.

The hypothesis framework used by Conversion.com is a great way to frame your research, and I’m looking forward to giving it a go.

Go to places where users are

The Home Office has a large remit, which includes immigration. Their users are from all over the world. Katy explained how they had problems with their E-Visa Waiver service used by countries in the Arab world. The issue was crucial: the online form wasn’t suited to the name conventions used in those countries. Processing errors caused by this were resulting in people being refused entry the UK. Before revamping the service, researchers travelled to these countries to meet with users.

You might not have to go as far as Kuwait or UAE — just go to where your users are.

and finally…

Swiss people have an aversion to maps

Conversion.com were working with a Swiss property search site when they discovered that it didn’t use maps. Was it because the Swiss had an aversion to maps? It was actually because the team hadn’t got around to developing the functionality. A standard feature on this type of service, you would expect it would improve the conversion. So when Stephen’s team ran an A/B test with an MVP, they found the maps didn’t actually make any difference to the conversion rate. Using this example in the hypothesis framework, we can conclude that Swiss people don’t like maps after all.

User research should be at every stage of the product development cycle — from discovery, development, release, and beyond. Running small tests, to gain rapid insight, ensure that you are putting your resources to the best use. Using the right combination of tools and techniques can help get your stakeholders onside, and your product is solving problems rather than creating them. We should all be doing more, or at least some, user research.