Jul 28, 2015
Don’t Listen to Your Users
And other lessons from building a consumer-facing app
Two years ago today, two friends and I sketched out an app, Hitlist, that would alert you when there were cheap flights to places you wanted to go.
Since then Hitlist has helped a quarter million users in 163 countries book over $6m of travel. Almost none of that happened in our first year of existence. We were press darlings, our users tweeted and emailed their love, and the metrics we tracked were telling the right story. But no one was buying. This is how we figured out what we were doing wrong.
Maybe I should give some context. When I was 18, I took my life savings — made from babysitting, odd jobs, and (not weird at all) selling parakeets I bred in my parents’ basement — and bought a plane ticket to Paris. For the next year I worked as a singer and tutor and lived on around $500 a month. I managed to travel to four countries thanks to some creative travel hacks.
I fell in love with travel that year, and its ability to change us and the world around us for the better.
We built Hitlist to make it easier for anyone to find affordable travel. Our app and Chrome extension inspire and enable people to travel more by alerting you when there are cheap flights to your dream destinations.
We comb through millions of flights a day and sort out just the ones that are relevant to you, then send them via email or push, so hopefully you actually take that trip you always dreamed of but never got around to.
When we started working on Hitlist in the summer of 2013, we were three first time founders and had pretty much no idea what we were doing. Our first version was ready to release by November. It featured Tinder-esque interface because who doesn’t love Tinder, right?
We released the app and the reception was fantastic. The Next Web wrote an article about us, and it ended up getting picked up by Lifehacker, and some blog in Greece that resulted in nearly 10k downloads, proving that at least some Greeks are budget-conscious. Then we got this awesome coverage in the New York Times, which looks great in a slide deck and made my parents very happy but wasn’t nearly as useful as The Next Web/Lifehacker coverage (we saw about 500 new users when that article came out and close to 10k between TNW and Lifehacker).
More important than all the press, we were getting fantastic reception from users. We’d get tweets like the above all the time. We got love letters. And some things that made me a little uncomfortable:
It was easy to get caught up in the good news. We poured so much into Hitlist and people were loving it. Right?
The only trouble was: they weren’t coming back. And they weren’t really telling their friends. After our download spike, we were barely growing any more — a couple dozen new registrations a day if we were lucky. And our churn (the rate at which users drop off) was absurdly high. Three months after launch, less than 5% of our registered users were still monthly active users (MAUs). Some smart people we knew told us that what we’d built was a gimmick. And we were closest to the product, and we found ourselves getting bored with it. You should not get bored with your own product. You should love your product so much that it’s all you want to talk about on a first date.
At this point we’d never done any paid marketing. We had one investor push us really hard to start buying Facebook ads to try and drive growth. But other (wiser) advisors pointed out that spending money on growth when we had 95% churn didn’t really make sense. My friend Tarun Mitra put this most memorably:
If you invest in growth before you have retention, you’re renting users, not acquiring them.
So we went back to the drawing board. We decided to completely change the design. The Tinder interface wasn’t right. A city is not going to ‘like’ you back. What kind of UX was going to be more effective for surfacing deals people might actually buy? We moved to a stream of content that’s dynamic, and changes according to your preferences. There’s some cool artificial intelligence behind this which I’m looking forward to developing more. The more you use the app, the more relevant the deals you see become.
We also introduced an ability to filter flights, so you weren’t just getting alerts for cheap flights generally, but for the kinds of flights you wanted. So if you’re trying to plan a 7–14 day beach vacation in August, you can see just the flight prices for that particular search, rather than all-time low fares.
We rolled out 2.0 in August 2014. Here’s what happened.
I got a bunch of nastygrams from users who missed the Tinder interface. We were also seeing a lot less love on Twitter. I tried, unsuccessfully, to pitch the new and improved Hitlist 2.0 to press, but the newspapers and blogs that had generated so much buzz around our initial launch had lost interest.
Then we looked at engagement.
The number of cities people added to their lists went from 41 in the Tinder interface to 5.6. That hurt. We’d counted cities/user as one of our key metrics, because we considered it valuable information on where people were interested in going.
However, when we dove into the data a little bit more, we found that nearly half the deals that people clicked on were never on their hitlists. What people said they wanted wasn’t what they were actually buying. If the point of our app was truly to help people find good travel deals, click through, and buy them, then the number of cities/user wasn’t really a relevant metric.
Meanwhile, our overall conversion/MAU was doing this:
In other words, people are actually taking trips. At this point 10% of our monthly active users are clicking through to booking, some of them several times a month (which is why our total deal clicks/MAU above is closer to 20%).
Equally as important, our retention numbers are completely different. One week after using the app, 42% of users are actively using the app, compared to… actually we have no idea, because we weren’t measuring cohort retention before.
Though it seems obvious now, it took me a long time to realize that there are two things that matter for an early stage consumer app: conversion and retention. User growth is important too, but you can buy new users (plenty of people do, with the amount of venture capital that’s sloshing around these days). But again: if you invest in growth before you have retention, you’re renting users, not acquiring them.
We’re not there yet. We won’t be until Hitlist is on everyone’s phone and is the first place people come when they think of planning and booking travel. But v2.0 took us a step closer, and we’re experimenting, and getting better every day.
So what did we learn? First of all, the user is always right… but sometimes it’s not what they say, but what they do. Of course you need to take user feedback into consideration, but remember that actions speak louder than words.
Secondly, don’t get caught up in what the press thinks. A good article can give you a nice boost in user numbers, but that doesn’t mean those users will stick around.
Most important of all: know the metrics that matter to your company. ‘Engagement’ is a fuzzy term. As we found out, the engagement metrics we initially tracked — cities/user, time in app — weren’t driving the thing that we actually cared about. The 2.0 design was demonstrably better at getting people to actually book travel. And the tweaks we’ve made over time (we’re now on v2.8) have helped increase our retention to the point where it makes sense to start investing in growth.
Speaking of which: please download Hitlist.
If you found this useful, please share it! You can find Hitlist on Twitter @hitlist_app and me at @gillianim. We’re also on AngelList. Visit medium.com/hitlist for company updates and spectacular gifs.
