Don’t (only) listen to users
Why listening is just not enough and observation is so important.
10 years ago I attended a short course on user experience delivered by the amazing Tomer Sharon. Back then he was working at Google as a user experience researcher and was sharing tips about how to run UX researches. He told a story about researches he held to check how people are using their emails. At the time most business users were subscribed to outlook (I assume the numbers have shifted since). Tomer interviewed different users on their email habits. One of those user stories stuck with me, let’s call her Sheryl for the sake of this story. When he asked Sheryl if she had any issues with outlook she replied that she was using it for years and was very happy with the service. He later observed her when she was writing and replying to emails. He noticed this strange flow - when Sheryl wanted to send an email she first navigated to the “Sent” folder and started compiling her message from there.
I remember this story as a turning point in how I addressed user experience. I highly value interviews and asking people what they want, what works for them, and what they wish was different. However, this story showed me we don’t always know to say what works and does not work for us, and even if we do know, we are not always sure how to put it in words. Sheryl really did not have problems with outlook, or at least did not think she had them. She wanted to send an email, so she turned to the sent folder. She found a workaround to use the product and this was not how it was intended to be used. This hack could (probably) only be discovered by observing her interaction with her inbox.
People don’t always use products in the way they were intended. Viagra was designed to lower blood pressure, SMS was developed by mobile operators to notify customers about network issues, Coca Cola was first launched as a substitute for morphine and there are many more such examples, some more famous than others.
Not only do people use products differently than intended, they don’t always know they do. These things may not come up in user interviews, as your product users may not even know to tell you that this is what they are doing. The data will probably shed some more light on the matter but will also not give away the whole story. If you want to get better acquainted with your product users you will surely talk to them and analyse their usage patterns, but you should also observe how they use your product.
Rebuilding the notification system in Fiverr’s mobile app
A few months ago we decided to test a new direction with the notifications in Fiverr’s mobile app to indicate order-related notifications separate from general notifications. In the process of doing so we changed from a textual list of notification to an iconic representation. The hypothesis we wanted to test was that anchoring notifications to their relevant screens would increase responsiveness and ease of navigation in the app.
- I urge you to read Shashank Sahay piece on designing notifications for apps to learn more about various notification schemas.
Seeing that this is a big change in how the app is structured and how the notifications would be consumed, we created a prototype and started interviewing our app users. We held dozens of interviews and overall the responses were positive. We decided to start an A/B test. The results were positive. Our users were getting more responsive to updates on their orders.
But the comments we got to the live test were also surprising. We’ve been getting customer reviews asking us to bring back the old experience, some of them we could label under “you moved my cheese” but others indicated that our users were actually using the app notifications differently than we originally thought. They were using notifications as their task management system. We noticed that heavy users are often marking their notifications as “unread” so they can back to them later on (same as we all probably do when we mark a slack message or an email as unread).
Given that Fiverr still does not have its own task management system for the sellers on the marketplace, they were using the notifications list as such a priority management list. A task management system surely has different requirements than a notifications system, and our current notification system does not answer all of our sellers‘ needs but they found a way to make it work for them. Enough so, that when we changed it they noticed and did not like it and they were now missing some basic functionalities so it would work for them.
Why did this not come up in interviews? Maybe the ones we spoke with did not use the notifications this way, but maybe they did not even think how this change would even effect this type of usage. There is no replacement for an AB test, and a good one has several iterations as we study from the responses, user behavioural data and responses.
We learned a lot from these post A/B test observation and interviews with users who have experimented at length with the actual product (and not only with a prototype). We are running a few more rounds of improvements, hoping to answer most of the needs that were raised while still maintaining the good results we got so far. We managed to prove our hypotheses but also found out along the way that there are other uses to this feature. We’re not there yet, we keep working on improving and finding better ways to answer our users needs and help them grow. In order to do so we should take into account what our users say that is important to them in the product, but also what they actually do with it.
