When we started Rollout.io, we wanted to help mobile developers (especially iOS developers) deal with the “pain” associated with the long mobile release cycle. More specifically, help mobile developers deal with bugs and quality issues in production. Fixing or changing anything in their mobile app would require them to go through the full release cycle every time even during ‘panic mode’.
The problem can be summarized in a couple of questions: What do mobile developers do when they discover they have a bug or crash in production, and what are the damages?
The typical answer we get is that it’s a nightmare— fix the bug in ‘panic mode’, rush the Q&A process or even skip it altogether, release to the AppStore using the “expedite review” service, wait, answer outraged customers, see the bad reviews, wait some more, try to figure out if you have connections in Apple that can help speed the process, nail-biting, stop all user acquisitions campaigns, more waiting, find Apple review team on Facebook and try to contact them (true story), try to solve the problem from the backend side, and more. If you're lucky, the damages are minor, but in a lot of cases the damages to you mobile company are huge.
The waiting time for a release cycle in this ‘panic mode’ can be anywhere from 24 hours to a few days, and in some cases even weeks. And then of course you still hope that your new release is not buggy as well (since you rushed the QA), otherwise the entire process will happen again. You could say, “it wont happen to me” but it happens when you least expect it, and it happens to big and small companies alike.
Our initial solution to the above problem was to let iOS developers package two versions in the same IPA file and allow the developer to remotely switch between them. The idea was that on each new release you will have your old stable release available to fall back to in case of a catastrophe, without going through the AppStore. Sounds good, no?
We decided to go directly to app companies and ask them about the problem and solution in order to make sure (=validate) that our product worth a buck. We built a question list that we kept updating, and tried talking to who ever we could find — indie developers, service companies, big companies, other SDK companies, etc. Here are part of the companies we talked to: Viber, Fring, Magisto, Moment.me, Foreach, Any.do, Askem, JoyTunes, Ringya, MySuperMarket, Win.com, Bizzabo, AppMyDay, Wibbitz, Yokee, and many many more…
Our customer discovery process had three major flaws which are listed below, eventually we managed to fix these flaws and as a result got invaluable insights that made us change Rollout.io solution completely.
Here are some of our mistakes and lessons we learned:
Key Learning #1
Separate the validation of the problem from the validation of the solution.
At the beginning our validation process mixed solution and problem in the same interview. This made it impossible understanding the feedback — did they just liked the solution, but did not have a real problem or they have a real problem but don’t like the solution, or both?
We wasted a lot of time and resources on the wrong solution. Only after we validated the problem separately from the solution we understood that the problem is very real and painful but our solution is bad (complex, deals with only very narrow use-cases, has some unresolved technical issues,etc), also our morale was fading away since it seemed that the “pain” is not large enough when the actual case was that the solution was not good enough.
Key Learning #2
Validating the problem is NOT asking it directly.
When we started our problem/solution validation process we asked many direct questions, such as: “Have you had production bugs ?”, “How many production bugs did you have ?”, “What was the impact of these bugs in production?”. As much as some of these questions are great they do not give you a clear image. Imagine getting the following answers — “Yes, we had production problems about 4 times a year, the impact was crazy, I would definitely pay at least 200$ a month for a solution”, seems like great validation of the problem, but in reality these type of customers want to please you, they know what you are doing and they do not want to make you feel bad (unintentionally).
The validation has to go deeper, in our case we needed to see how our potential customers deal with quality issues on their mobile app, how critical is mobile in their strategy and what are they doing today to avoid quality issues in production. Questions like — “How do you Q&A ?”, “How many people are on the Q&A team?”, “Which app crash reporter service do you use ?”, “How do you discover you have a production issue?”, “How are you tracking your bugs?”, “Which type of bugs did you have, can you elaborate ?”
After we started asking both type of questions (direct and indirect) we saw that in more than one case the answers would look something like this:
- Have you had production bugs? — Yes.
- How many production bugs did you have? — About 4 times a year.
3. What was the impact of these bugs in production? — The impact was crazy, I would definitely pay at least 200$ a month for a solution.
(Up to here everything seems good)
4. How do you Q&A? — We do it internally, some of us use the app before we send it to the app store.
5. How many people are on the Q&A team ?— no one is dedicated, the developers check.
6. Which app crash reporter service do you use — we do not use any crash reporter, we use the generic Apple crash reporter.
7. How do you discover you have a production issue? — We start getting complaints from users, we see the bad reviews, a friend tell us something is not working, etc.
8. How do you know why the app is crashing? — We do not check all the small issues, if it’s something big and most customers are affected we debug and release a new version ASAP.
As you can learn from the above more in-depth Q&A, the company that answered does not address quality issues as a priority and getting validation to the problem from them is meaningless, even though the answers to the direct questions seemed very helpful and ״validating”
Bottom line: Do not pitch your idea to the customer, try to understand the routine (and find validation to the problem there)
Key Learning #3
When trying to validate the solution, you have to put your potential customers in the position that they need to make a decision
So you validated the problem and sure it’s big and painful, now you are trying to validate the solution, you need somehow to make sure your solution is good enough and that you are in the right direction.
As in lesson #2, many potential customers would not tell you your solution suck, you need to make them make a decision, it can be payment, it can be some other type of commitments — such as a weekly meeting with the development team to start integrating the solution.
If they are not willing to commit to anything, either they do not have the problem or you solution is problematic.
Conclusion
The discovery process is an invaluable process but it needs to be done correctly, otherwise it creates noise and distractions with no real insights in regards to either the problem or the solution. Try to figure out the questions that will give you an answer you're not expecting. Drill down deeper, and get to the core. Don't offer a solution, ask for the customer to describe it for you.
