Lessons learned from a lean experiment
Doing start-ups is emotionally hard, most of the time you fix problems (oh, sorry, grab the opportunities) and deal with failures (oops, acquired experiences). In order to ease the pain, people came up with the Lean Startup approach. The idea is, basically, to minimize the effort needed to test the hypothesis (acquire new knowledge). With 90% of start-ups failing, that means that you minimize the burden of failure and transit to the success-country-land (the rest lean 10%) much quicker. It sounds like a no-brainer but there are many pitfalls. Here is the story of a recent lean experiment we have done with our latest Bergamo Quest app. The app was published just several days ago: Bergamo Quest for Android and Bergamo Quest for iOS.
First of all, I encourage you to try the app and let me know if you’d like to have the same available for your city. Isn’t Bergamo Quest a great pretext to finally visit the city somewhat overshadowed by a buzzing Milan and the so-R&J-romantic Verona?
Inception of Bergamo Quest
While working on our main product Routes.Tips we felt that it would be cool not only to explore new places efficiently, without filtering tons of information available in all forms and sizes (and quality) but to play a travel game while enjoying your holiday as well. In the end of the day, traveling and playing are among the most joyful activities. Wouldn’t that be cool to walk old streets of an ancient city with rich cultural heritage and solve puzzles, while learning the history and culture behind the facades? No more encyclopaedia-style information like ‘lived-worked-died’, but real-life stories, when you have an event pegged to a man and the consequences or reasons why it happened. So, the idea of a Travel Quest game came up. Inspired by our childhood experience of playing the “Treasure hunt” game, romantic Pirates movies about hidden chests of gold, and contemporary entertainment activities like geocaching and orienteering we started to think about a Minimum Viable Product (MVP).
Hypothesis
The initial hypothesis for the MVP was that citizens and tourists would be interested in playing geolocated quest game so much, that they would be willing to part with a couple of dollars. We felt that having two very different audiences (local city dwellers vs tourists) goes against the lean start-up methodology but we weren’t able to resist the temptation to have ourselves in the target audience as well. Another excuse was that the only difference that matters (as we had thought) was the background local knowledge.
- Audience: local city dwellers and tourists
- Action: playing a geolocated quest game
- Price: $3
MVP scope
Probably the hardest part of the lean start-up methodology is to decide what should be there in MVP, and what should be kept outside. We always struggle with this. Everything seems important and critical for testing. Which feature is a showstopper or a game changer, and which is only an amplifier or an unnecessary addition. It seems funny, but the second hardest issue is to decide whether you should abandon MVP and the idea or to tweak it adding some features, changing previously built ones.
In order to keep the scope minimal, we added an additional limitation of 3 weeks’ development time. So, only the things that we could do in 3 weeks’ time should make to the MVP feature set. Getting a little bit ahead of the story, I have to admit that it took us 3 months to launch the product, and 1 month to gather the feedback for the next iterations.
One of the first decisions was to figure out the city where to start and test all the assumptions. It had to be more or less big city with rich cultural heritage and a substantial tourists flow. Italian Bergamo felt like a perfect choice. The city is big enough, with many tourists visiting it every year and hundreds of years of history. Last but not least, it felt overlooked because of its grand neighbors Milan and Verona.
After several hours of heated discussions, we quickly came up with a basic feature set for our MVP:
- A map screen with a set of markers scattered over Bergamo city and a progress indicator (the user’s score or how many puzzles are left)
- A puzzle screen where the user reads the question and provides his answer
- A set of leading messages helping users to get through the quest
- Offline support — tourists might not be willing to use Cellular Data for the app (and locals too)
Things which didn’t make to the MVP:
- Visual design — we intentionally decided to keep UI dead simple in accordance with standard Material Design / Flat design UI elements and dynamics
- Advanced question/answers types — we wanted to stick to the simplest (in implementation) form of answers, free type input. Yet, after very early tests we realized that questions are very hard to answer and users get frustrated because they believe they are right. Funny that even those who had a list of right answers sometimes failed to solve the puzzles. So, we implemented quiz type answer selection and sequential hints.
- Dozens of other genius ideas, like puzzles pronunciation, so, you don’t have to read from the screen, advanced layouts, group playing.
What was completely unexpected and led to a huge delay was the content part. We didn’t expect content authoring to be THAT hard! Crafting words for questions, images for illustrations, reveal content, all those took weeks to polish. Also, it took several business trips to Bergamo. But that was a very rewarding part.
Mid iteration feedback
Somewhere in the middle of the development cycle, we ran a small test on friends and family. We invited some of them to Bergamo and watched them playing the quest. You might have noticed already that after such field tests we decided to ease the questions by them making a multiple-choice type. However, there were other valuable findings as well.
It was crucial to align content (photos, question texts and hints) with puzzle locations. Some puzzles had to be removed completely as on the spot they were irrelevant. Some puzzles had to be tweaked significantly, so we spent weeks on copywriting.
The app was not intuitive. How on earth can two screens be confusing?! Well, they can be. Mid iteration feedback is crucial. Do not skip real life test on friends, family, and loyal users who have features of your target consumers as early as possible (as soon as your prototype could be consumed).
MVP Triage approach
The approach to making MVP feature set decisions was the following:
- Will the project die without this feature?
- Could we test assumptions without this feature?
- Can we develop it in a 3-week timeframe?
- Could we do it later without sacrificing the whole project?
Bergamo Quest technology stack
While it is not essential for the MVP discussion, I’ll just quickly mention the most important tech we used in the project:
- Material design / Flat design UI components — obviously, a significant time saver on UI customization.
- MapBox SDK — nice, very flexible maps with offline maps support. We were even able to style maps to the app colors, so the map feels native.
- MixPanel analytics — our choice for measuring small data (distinct user sessions) opposed to “averages” of big segments.
Evaluating MVP
The most interesting part, the season one finale, is to measure the results. While we’re are just starting to gather the quantitative data, most of the measurements preparation work should have been done earlier. Here is our approach to measuring our assumptions:
- Users complete the quest (solve all tasks) — we have a greeting pop up at the very end of the quest
- Users are willing to buy more puzzles — we have added a fake buy more puzzles for 3 euros call to action in the middle of the quest, also we are going to try making the app paid after a while
Both are measured as events in MixPanel, so we could figure out how many users reached the goal. Also, we have a vast range of qualitative feedback gathering activities planned in Bergamo.
Lessons learned
There were several lessons learned I feel quite important:
- Limiting development time helps a lot to limit the scope of MVP and keeps you focused on the core MVP features.
- Not related to coding part of the project is usually bigger. Preparing content, marketing, and users’ research usually take much more than software development itself.
- Being a good at project management would save you a lot of time. I wasn’t :-(
- Test or friends / family / loyal users as early as possible (as soon as your prototype can be consumed)
That’s it for now. In a month, I’m going to share our assumptions test result and the future plans. Meanwhile, I’d highly appreciate your feedback!
Also, it is interesting to know how you do your lean experiments? What are the hardest parts and how do you overcome the hurdles? I’d love to hear your thoughts.
