User research

Part 4

The thing we tested

Get feedback fast

In Part 3 we got our designs to a shareable state and threw em on the web.

Now to get feedback from users as soon as possible.

If you want to make something that people want you gotta be guided by insights from your users, rather than the blob of biases that is your own brain.

But but but

There were a bunch of perceived blockers to doing this for our furlough calculator:

1. It’s such a simple site, is it even worth bothering?
2. Everyone’s on lockdown, how am I going to find people to test it on? I’m never going to convince anyone I’m at Y Combinator from my flat in North London.
3. Everyone I know who’s been furloughed works for the same company as me and knows me. Their insights are going to be biased.

While each of these points is based on something true, none of them is a good reason to block me from getting insights.

Better to just try something than be blocked.

Just try something

Though everyone I know who’s been furloughed is biased, their insights are still somewhat valuable. As long as I keep their bias in mind I can move forwards.

I threw out a bunch of messages like this to a few furloughed pals from work:

Within 10 minutes, a fishy bit 🎣

While having video chat I asked them to share their screen and navigate to the site, writing down quotes on a Google Doc and asking probing questions now and then.

It always blows my mind how much useful stuff comes out of user research for even the simplest things. We uncovered loads of little problems.

It also blows my mind how just trying stuff unlocks so much other stuff you never expected.

After our little session, my pal shared a blog post with me on user research, written by our user research sensei at work, Samantha Davies, and I learned lots.

What I did wrong

Turns out there were a few things clearly wrong with my research and some clear ways to improve:

I asked probing questions

Every time my participant was silent for a while, I tried to hold the silence for a bit to see if they’d come out with anything.

If they still didn’t speak, I asked something like “what were you thinking just then?”. I also couldn’t resist asking if features I’d left out of the build but had designed would be useful to them.

As the blog post puts it:

If we need to ask a user questions to understand what they’re thinking, we’re asking them to share their perceptions and opinions. And because opinions can be so changeable, this introduces subjectivity and inconsistency.

Gotcha. Let’s focus on behaviors next time.

I didn’t have a clear idea of the behaviors to look out for

I had a Google Doc open and took quotes, but didn’t really know which behaviors I should have been paying attention to beforehand.

That said, I think that the act of doing this sort of research helps to clarify in your own mind what’s important to pay attention to. I now have a much better idea of that than I did before.

Going forward, I put together a “behavioral analysis matrix” as the blog post recommends, filling it in with the behaviors I wanted to look out for.

I’m a fancy user researcher

I really like how it’s organized by assumptions, rather than by participant, which was how I was about to organize my doc.

It really makes you focus on the problem, and having different participant’s quotes on the same problem side-by-side is super useful when coming up with solutions.

My poor little Google Doc was thusly expunged, 57 minutes after coming into this world ☠️

I didn’t think of testing on mobile

As I was on my laptop, it didn’t occur to me to not test on a laptop. It was weird to notice that blind spot, as I’d been designing for mobile all along. Perhaps having just slipped off my coding mittens I was still in laptop mode 🤷‍♂️

According to the small amounts of data on Google Analytics, my users prefer mobile.

Going forward I should test on both mobile and desktop, checking my analytics to see if that changes once more data comes in.

🤔

I still don’t have a clear idea of what to do if the participant isn’t giving me any signal on the assumption without asking probing questions.

Perhaps I can hold the silence for a while to give them a chance to encounter the assumption themselves, and only after that ask open questions? Or perhaps best to leave the assumption empty, move on and get more participants?

Anywho that’s no blocker to getting more insights, and I’m sure we’ll figure it out in due course.

Insights

After observing 5 participants, I went through my matrix to arrive at some insights. These’ll be used to inform the next design:

Assumption 1: Participants won’t hesitate to put salary in input

1. Placeholder text looks like it’s a real value, which means it’s not clear to the user if they should first delete it or not
2. It wasn’t clear to all users that hitting “enter” or clicking outside the input box was required to update the values

Assumption 2: Participants won’t hesitate to parse the breakdown

1. People don’t instantly understand how we arrive at “base furlough pay”, but do figure it out after a bit of thinking
2. The breakdown values look the same as the placeholder text, which means that people don’t know whether to take it as real output or not
3. It’s not immediately clear that the output is monthly pay (as opposed to yearly)
4. The numbers themselves are a little hard to parse

Assumption 3: Participants won't encounter validation errors

1. Inputs on Microsoft Edge seem to behave differently, specifically the default cursor position

Assumption 4: Participants wouldn't ask for more information

1. It’s not immediately clear if you’ve hit the maximum furlough amount, it’s assumed that you know it’s £2500
2. Some would like student loans and pension contributions included in the breakdown
3. I gave “student loan” a capital ‘S’ for no reason 🤦‍♂️

Overall I’m pretty chuffed with the amount of feedback I got from this small amount of research.

Despite the biases in my group of participants (all aged 20–40, all work at a tech company, most are colleagues) I’ve got way more confidence that the next thing I build will be a step in the right direction 💪

Next up, we go stumbling into search engine optimization in Part 5.

If you read past the tapir, chances are you read the whole thing. If you dig it, give us a few claps. It’ll help others find this too.