10 lessons on 3 years with Prayas
Yash and I have come to the conclusion that our original idea with Prayas Analytics is not working, and that the time has come to shut it down. The most basic goal of a startup is to make a product that users really want; it took a long time to realize it, but we’ve failed at doing so.
Our product was a retail analytics tool. It was meant to be the Google Analytics for the real world, using security cameras to collect the data. The story made sense — Amazon has data on how customers behave and can optimize their website; using Prayas, Walmart can do the same. After three years of trying, it’s clear that this isn’t working.
This hasn’t been an easy decision. It’s not one that we’re taking lightly. People have invested in us financially and emotionally for our idea. And we’ve been working on it for three years — even though the first two of those were while we were still in university, we’ve had this idea in our heads for what seems like ever. Letting go is hard. But it is clear that it’s the right decision.
Although the arrival of failure at our doorstep is never welcome, it always comes with deep lessons packed in its suitcases. With the benefit of a few months of hindsight, this post is an attempt to unpack some of those lessons, from the more pedantic business learnings (1–7), to broader musings on life (7–10). And finally, on what comes next.
Lesson 1: Ask questions to a comparable set of customers.
Proving the basic hypothesis of whether people need a startups’ product means having good customer conversations. If many customers say similar things, then it’s a sign that you’ve targeted a narrow enough segment. Unfortunately for this to happen, the customers have to be similar and specific. When we first started with the Hiper Consulting idea (consulting for startups, paid for by the minute), our customer conversations had no specificity to them.
Conversations with customers is key. When we started working on the retail analytics idea, we didn’t even really have conversations with customers — just advisors, professors and friends. We wrongly took their approval as validation of our idea. The only opinion that matters on an idea (at least in the short term) is that of your customers.
A good framework that I’m trying to follow now is to have at least 27 conversations per idea — 20 with potential customers, five with domain experts and two with “gurus”. Domain experts are people who have worked in the industry you’re selling to, or with the type of product you’re building. Gurus are higher level experts like trusted advisors or investors, for macro-level advice on longer term strategy.
The conversations with 20 customers should ideally reveal similar answers. If the answers are all similar, then you know that they are a specific enough group. If the answers are different, find the subsets, and hit 20 conversations in the most relevant subset. The product should change in response to their feedback, so that the process ends with a different product, or a few users for the current product. Here’s an article with a good framework on this process.
Lesson 2: The questions should go deep.
Questions to potential users should be deep enough, such that a negative answer can destroy an entire idea. This is inherently difficult because most of us have an internal resistance to criticism or failure (see lesson 10); but looking for validation rather than criticism can hurt the process of developing a product.
We constantly made this mistake at Prayas. It’s funny how scared of failure we seem to be, or how desperate we were to come off as smart to the person we were talking to. Confirmation bias hurt us; we ignored negative feedback on our idea, and over emphasized the positive feedback.
Apart from just asking questions, it’s important to dive deep into each answer. The pivotal moment when I realized we had done this all wrong was when I read The Mom Test, an excellent book on customer conversations. When we spoke to customers, we always asked them about our idea and what they thought about it. The problem is that no one will ever say an idea is bad — people are too nice. What we should’ve asked instead is whether they felt like they lacked the data we were providing; if so, what did they need it for? How were they trying to collect it right now? This way we could have evaluated the depth of the problem faced.
A good rule would be that the co-founder needs to be able to understand the call and all its dynamics perfectly when she reads the notes/is debriefed about the conversation. The person who had the conversation should be able to answer every question. Ideally each conversation should help prove or disprove a startups assumptions.
Lesson 3: Willingness to pay is the only test.
If all the customer conversations go well, then try selling. If people aren’t willing to buy your product, it means that you haven’t found a hair on fire problem, or that this market is incredibly stingy. The objective behind charging money at the start though isn’t really revenue — but rather it’s to gauge product-market fit.
One problem we faced with our analytics product was that the data was very interesting, but wasn’t actionable. This meant there was engagement, but no ROI. We should’ve taken this as a bad sign, but we ignored it. For a long time, we never really charged money for our product. We were too afraid of losing our users. When we did try to charge them, no one paid.
Lesson 4: The initial group of customers has to be approachable.
We have now learnt that companies should start in a position where the founders can feasibly recruit the initial set of users. In our case, going after large Fortune 200 retailers was a mistake. We lacked the retail network, or the understanding of the industry to make in-roads. Now, both of those can be overcome (indeed, now our retail network is pretty great, thanks to several cold emails to CEOs who actually replied pretty enthusiastically), but it slows down the company significantly.
To prove the basic hypothesis of product-market fit, you need fast feedback. Large enterprises are no good for this, because the user and the buyer are very different, and their identities are hard to gauge at the start if you have no experience with the industry. Furthermore, the sales cycles are generally quite long. Selling to large companies can be sped up if there’s a well defined process. Figuring it out as you go along won’t work. Unfortunately, due to our inexperience, we didn’t have a good process.
Being very young and bereft of a network in the industry makes it very hard to sell large contracts. Indeed, being young in general seems to be a disadvantage in SaaS. It’s probably best for young founders to sell products at lower annual contract values(under $100,000), and to smaller businesses to start off with.
As inexperienced large company salespeople, we were constantly fooled by small signs of progress. We always thought that we were very close to sealing the deal. I remember the day of our first presentation to a pilot customer, which is a large Fortune 200 retailer. They found the data extremely fascinating. They were engaged the whole way through, asked great questions — and immediately my mind jumped to the thought of completing the sale and making this a huge company. Three days later I was texting friends saying that we’re very close to a million dollar contract with them. Three years later — still no contract! Lesson learned in managing expectations.
The reason why we went after large companies from the start was our assumption that large companies will have completely different needs than small ones. I don’t know if this is true anymore; if something is a massive pain point for small companies, then it’s likely that large companies face it too (but are unaware it’s a pain point or don’t care). Small things might change of course, and crossing the chasm is always hard — but there’s no point trying to cross it without first proving if anyone needs the product.
Lesson 5: Tech development second, product-market fit first
There’s no point building tech for something that people don’t want — it just becomes a science project. The ideal situation is to have a solution that you can hack together quickly, one that can provide the illusion of a product. Essentially, this means creating a good front-end, and manually joining the dots on the backend at first.
This is a fast way to test the product. When enough people want it that you can no longer so it manually, then it’s a good time to start building out the technology. I love Paul Graham’s advice on this point — which is to focus on a week-on-week growth number to help prioritize. Will building out feature x help me grow? Is scaling feature y necessary to grow? If not, don’t do it yet.
Technology can also lead to inertia. Founders are loath to give up tech they have built, and so it almost commits a company down a certain path. It’s a case of premature optimization.
I feel good about this part. I think we did this fairly well. I don’t think we ended up getting attached to our tech at any point either. That said…
Lesson 6: Establish what’s possible so that you don’t panic
We made the big mistake of not talking to technical experts in our field. In fiercely technical fields, like artificial intelligence and computer vision, there usually is a manual and hacked together solution. As long as you can confirm that what you’re hacking together is possible to automate in the future, then hacking it together (using people instead of machines) to temporarily test the product is fine.
But we didn’t talk to any experts. So we always made big assumptions about what was possible and what wasn’t. This lead to panic and hesitation when we had feature requests. For instance, one of the first things we should’ve done would have been to talk to an expert in the field of security cameras and NVR system architecture. We could’ve talked to this person about our plans to get all the video from these cameras, and established if they were realistic or not. Instead we just panicked and worried for a long time, alternating between feeling hopeful that it would work and hopeless that it wouldn’t.
Keeping a technical consultant on file (or just a bunch of smart & knowledgeable friends) is a good idea. This expert can verify if what you’re doing is possible to build with technology or not — and if it is possible, no harm in doing it manually for now to test things out.
Lesson 7: Offline analytics is a tool, but not a full solution.
Data on its own isn’t useful; solutions that come out of the data are. However analytics products often fall into the trap of being just a part of the solution.
In our case, after looking at the data, someone had to figure out what to do and then implement the solution. This made the analytics product less useful, and increased barriers to adoption. As we learnt, implementing the solution based on the data is particularly challenging in the offline world; re-configuring the arrangement of shelves in a store is much harder than changing some CSS on a website.
Another problem with our business model was that people didn’t really use data like this already, and so there was no imperfect substitute. Since customers weren’t used to using this data, it took them a lot to figure out how to use it. If they couldn’t figure it out, the data was useless.
As an offline analytics tool, we couldn’t implement the solution on our end. The retailer would have to make the changes in the store themselves. Our customers had to see the data, and then implement the solution based on what they saw. So we weren’t in full control of the process, and this added a barrier to adoption of our product.
If the data product can’t implement the solution itself, then the next steps from the data need to be clear. Providing clear recommendations on what to do for instance. But this is a chicken and egg problem; if you can’t get anyone to get ideas on how to use the data, how do you build a recommendation system? It might have been possible if we had domain expertise in retail, but we lacked it. Our pilot customers just didn’t know what to do with our data. It was very interesting information, but not important.
Finally, offline analytics products have infrastructure issues. In our case, it was dealing with old camera systems and retail stores that typically only had a 1 Mbps connection to power their POS system. We ran the math at some point for one of our pilot customers, a retailer with 1000+ stores across the country. They had over 70 cameras in each store. If we were to get video from all the stores of just this one retailer, it would be thrice the volume of video YouTube receives. I’d calculate how long all of this would take on a set of 1 Mbps connections but I’m scared of numbers with too many zeros.
For this reason, I don’t know if offline analytics products can work. I don’t know if working on another company in this space is a good idea.
Lesson 8: Don’t get caught up in your own mythology.
In general, we didn’t let press or external success get to us — and I’m very proud of that. But we let our internal story about ourselves get to us. How many times in conversations with people did we tell people that the first thing we did when we were looking for startup ideas as sophomores in college was pick up the phone and talk to a hundred Wharton alumni? After a while we forgot that it was an exaggeration. We looked back at our phone call notes and CRM from back in the day — and the truth is, we only ever spoke to 11!
Histories are always reconstructed, but are not always real. We always talked about how it had been a great journey and how we’ve come so far. And emotionally, this is true — we grew up by many more years than we lived. Intellectually, definitely true — our hard skills are sharper, soft skills more rounded. But in terms of customers and progress? It really wasn’t a great journey.
In the end, we made a sum total of $5750 in revenue. None of this was recurring. Our data was never really used for anything, and we didn’t make an impact with our work.
This is hard to admit. It’s frustrating. But it is more frustrating when we think about how much time we spent on this. Each day that passed without revenue or customers made us feel like we had to keep going on, if only to make the last x many days worth it. The sunk cost of all the time spent kept holding us back. Building on the sunk cost was the frustration of not creating real value. Perhaps to soothe our anxiety over this, we told ourselves that we were laying the groundwork for something bigger and that success would come in the future, but it never did.
We had a great story and a clever idea. But maybe if we had been less enthralled by it, we’d have been more open to change.
Lesson 9: Follow the inspiration
This is a principle I seek to live my life by, but at some point I forgot to live it out with our company as well. We were in a rut for ages. There’s a reason why I felt upset and demotivated for a long period. The first year of working on Prayas was certainly very happy, because we were learning so fast. The second year wasn’t as happy, but there were enough small signs of success that it felt good (we were also too foolish to realize that these were false signs).
But to be honest, the last one year was not enjoyable. Some moments were amazing — getting an investment from Y Combinator, seeing brief signs of excitement at customer meetings, each response to a cold email from a retail CEO. And then the times when we were in the flow, optimizing our cold email campaign or building out the technology. But I’ve genuinely enjoyed maybe eight weeks of work in the last one year.
It came down to that feeling of not creating value. And for me personally, the frustration of not being able to succeed at something. There was some amount of impatience born out of this world being new. For instance in college, a few nights of intense studying before an exam equalled decent grades. This was taking months, and there was no success. Furthermore, we were just not growing. It didn’t feel like our product could be useful.
Impostor syndrome kicked in when we felt like we should be successful because we were a Y Combinator company. Now, with the benefit of hindsight this isn’t true — every YC company goes through moments like this, even the most successful ones (and the truth is that even at YC, most companies fail, like us.) But it was hard to see it at the time.
This was all leading to constant burnout. Insomnia and anxiety attacks crept in. I couldn’t work for more than 3 weeks without feeling like I needed time off.
What we should have done is recognized the reasons for the demotivation — a lack of progress, stagnant personal growth and so on. By changing things up to make the startup more exciting for ourselves, we would have been more encouraged and probably built a better product.
Lesson 10: Don’t fear failure.
I sadly think that we did things, or avoided doing things because of a deep rooted fear of failure. Starting with our customer conversations where we constantly looked for compliments and validation, to our fear of disappointing people by putting on a price tag.
Our fear of failure made us avoid tough decisions. When people didn’t buy our product, we assumed that our execution was bad, not the idea itself. We never questioned our basic idea; we assumed that it was sound, and feared being proven wrong enough that we ignored evidence to the contrary.
It’s amazing to me that the idea of a pivot only recently came to us. We never considered it. I think it was a feeling of stubbornness that came from fearing failure — that changing our idea would be failure.
Startups should be nimble enough to change when all signs say so. The Lean Startup describes entrepreneurship as a set of experiments to see if a certain hypothesis is true. To succeed, a startup should test the hypotheses quickly, and change tracks if the results are negative. The most basic hypothesis of them all is whether there is a real need for what the startup is doing. Most signs said that the answer was no for Prayas. But we never acknowledged it because that would mean admitting failure.
The first way to fix this is to not be too attached — to ideas, to tech, or to any decisions we made. Not being afraid of failure is a mentality challenge — and I feel that just by realizing it this time, we will be able to avoid it next time around.
The next is to use external constraints. Use our advisors to judge external signs and what people are saying. And importantly, we need to stick to our deadlines. For instance, if we can’t validate a hypothesis within the deadline to do so, we have to understand that the hypothesis is false. Deadlines and goals need to have consequences.
But lastly, and more philosophically — failure has long been wrongly defined in my head. Failure isn’t trying out an idea only for it to not work out. It isn’t having a hypothesis that’s proven wrong. Failure would be staying in a position where things can only go right because they are easy. Failure would be keeping it safe.
It’s taken time to accept that the original idea did not work out — but that doesn’t mean it was a failure. There’s been personal growth and learning. Even if our impact on the world was limited, the impact on ourselves was tremendous. And I feel we’re much better equipped now for everything in our future.
What’s next?
After a lot of introspection, my co-founder Yash decided to move on from Prayas — a bummer, but a decision that I’m fully supportive of. And given the situation, it felt like the right thing to do was to offer the money we raised back to our investors. Most of them decided to take the offer.
The process of giving money back was possibly the hardest part of the last few months. I have no ill-feelings at all towards our old investors; but the moment I wired the money to them, it felt like the company was truly buried. That said, I’m glad that it was handled very well by all parties involved, and I’m grateful I can still count on all of Prayas’ old investors to be my professional support network.
Through this all, I re-discovered how much I loved entrepreneurship. I think the pivotal moment was when I was trying to convince Prayas’ largest investors to stay in the company, even though they had no idea who my new co-founder is, had barely ever met me in person, and all we had was a vague slide deck on the new idea. This was actually the first time they had seen my Veerappan-like mustache, and they regretted doing this pitch over a video-call instantly. The sheer absurdity of it all is what makes entrepreneurship so enjoyable.
So with the few remaining investors, and a great new co-founder, it’s time for the new avatar of the same old legal entity — MetricBoy! We’re a data science development shop, working on plenty of interesting problems, and having a lot of fun while doing it. If you’ve got any data needs, please reach out — my email ID isn’t hard to guess!
And even better, if you’re a founder that needs a shoulder to cry on, I have two.
Update (as of late 2016):
Yeah, MetricBoy didn’t work out either. But, we’re now working on SimpleMoney! It’s a portfolio tracker for Indian mutual funds, and it tracks your portfolio automatically by reading the statements from your inbox.
