Case Study of a Failed Startup

My sophomore year of college I was a neuroscience major and my plan was to unlock the secrets of the mind and universe. Then I realized that the path of grad school → professor → grants → incremental research wouldn’t be a great way to pursue this goal. At the same time, I read Paul Graham’s essay — How to Start a Startup. I felt a strong sense of:

I could do this!

and

Making a lot of money from a startup and using it to do something good would be a better way of pursuing goal of doing something important.

Throughout sophomore year I read all I could about startups, and about things like finance, marketing, strategy, economics, design, technology etc. The following summer I started learning HTML, CSS, JS and PHP, but didn’t make much progress (I also started a very early version of the website that I’m writing this post about).

The following year I thought about starting the startup during the year, but decided not to. It was a busy year with classes, and a good opportunity to be social. I still spent a decent amount of time thinking about the startup and learning about related stuff.

Finally, the summer after my junior year I began the journey of starting a startup.

The Idea

The image below best explains it. Choosing a college is hard. Applicants have a bunch of questions (leftmost column), but only get answers to a few broad ones (other columns).

Solution: Address this pain point. Answer all the questions that applicants have.

Expand: Grad schools, high schools, coding bootcamps, trade schools. Career “reviews”. Better RateMyProfessor.

Monetization: Choosing a college is an important decision. Wouldn’t you spend $100 if it meant going down the right fork in the road? I think that a successful version of this site would generate a lot of utility for users, and they should be willing to pay for it. There’s also always ads.

Demo

I saw two notable roadblocks:

1. This site only has reviews for 2 of the schools I’m interested in. I’m going somewhere else to do my research.
2. There are only 1–2 answers per question. Those answerers might just be outliers. This isn’t reliable.

Hypothesis: I think that there will be a lot of users if…

1. Enough schools are covered such that the user will be able to find most of the schools they’re looking for on this site.
2. The content is detailed enough that it’s useful, and plentiful enough that it’s trustworthy.

Execution: The biggest component here is to get people to answer questions. I thought there was a decent chance that people would do so voluntarily if I asked them on Facebook and Reddit. But if that didn’t work, I did some rough calculations and was confident that I wouldn’t have to pay that much money to get the answers I’d need.

The Outcome

I could write for hours about this, but I’ll be brief:

• People didn’t answer questions for free.
• Colleges, high schools and student organizations didn’t help me.
• After the first two bullet points failed, I tried incentivizing people, but this too was difficult. I spent a lot of time messing with different structures and amounts of money. It was mostly a failure, but in the end I finally managed to have one successful pilot program where I got a sufficient amount of answers for 3 schools. But now I needed to raise money to pay for the remaining 297 schools.
• Investors weren’t interested in something without traction.
• I actually got a meeting with one investor who said he liked the idea and would be interested in investing if I got some traction. We talked and agreed that it’d be a good idea for me to do a pilot program at the Ivy League schools.
• I paid $1/answer + Facebook advertising + $10 per referral and got 5000 answers at the Ivy League schools. 98% of the answers came from 6 out of the 8 schools though.
• I spent 2 weeks cold calling college counselors saying, “Hey I’ve got free student reviews of Ivy League schools. Check it out, let me know what you think, and sign up here if you want to be notified when I get more schools.” Almost all of them ignored me.

Mistakes

Roughly in order of importance, I’ll describe my mistakes.

I overestimated the likelihood that I successfully raise money.

The way I saw it:

• P(capital) = the chances I successfully raise money.
• P(success | capital) is very high. P(success | capital) depends on P(answers | capital), P(users | answers) and P(revenue | users).
• I had successfully gotten answers at 3 trial schools, so I thought that this was good evidence for P(answers | capital). In retrospect, this was absolutely wishful thinking and overconfidence.
• I thought that P(users | answers) was high because my intuition that users wanted more detailed information was strong. I still believe this.
• As for P(revenue | users), a) my competitors made tens of millions of dollars a year, and b) I thought the mismatch between the value I offered and what I’d charge was high enough that high schools would pay to provide their students access to the content. In retrospect, I think I underestimated the difficulty of selling to bureaucratic organizations. But ultimately, I still think P(revenue | users) is high.
• An investors decision to invest should depend on P(success | capital). To the investor, P(capital) = 1, because the investor would be providing the capital.
• I thought that there was a solid chance that investors would agree that P(success | capital) is high enough to make an investment worthwhile. In my mind, the big question mark was P(capital). I thought that with capital, I’d have a high chance at success.
• I know that not everyone thinks in such a bottom-up fashion, but I thought that since there are a lot of investors out there, chances are I’d find someone who thinks this way.

In reality…

• I should have had a better intuitive sense that people don’t think like this. They depend much more on heuristics/outside view, and the outside view says that an inexperienced, unskilled college kid with no other partners, no connections, and no traction is unlikely to succeed. They weren’t looking as much at the details of my specific situation as I was.
• I should have done more research into what P(capital) really is. I should have found someone to mentor me. I think that I suffered notably from confirmation bias here. I should have given much more consideration to the possibility that P(capital) is low.
• I should have responded more quickly to the trouble I was having raising money. It should have been a sign for me to update my beliefs more than I had.

I underestimated people’s distrust and reluctance to try unfamiliar products.

Self explanatory.

I underestimated the importance of design.

People told me that the design was iffy. I had a strong belief that the value the site offered was mostly determined by the information, and that it’d take a really bad design to detract from the value offering. I didn’t think the design was close to that point. I thought that it had a moderately unprofessional feel, but that it was reasonably usable and the aesthetic quality was alright.

My beliefs (in this context):

I now believe that this is wrong. I think that the graph looks much difference for companies that are in the early stages and are unproven:

I think that users (mostly unconsciously) look for signals of competence.

• Being a real company with real employees is a signal of competence.
• Having lots of users is a signal of competence.
• A good design is a signal of competence.

In the absence of other signals of competence, I think that a bad UI is a signal of incompetence, and that has a huge impact on value. (Not having the .com of your name is also signal of incompetence.)

I didn’t realize this. Partly because I didn’t understand users enough, and partly because I spent too much time winning arguments against weak opponents and not enough time thinking up and arguing against the strongest counter arguments I could find.

I underestimated the difficulty of the coding.

I was bad at coding. I had a vague understanding of HTML/CSS/JS and had started the site right after spending a few weeks learning Rails. I figured:

Oh, I just need to get a few basic things down, and then once I get traction I could outsource the coding.

In reality, the coding was harder than I expected and I spent way too much time trying to hack stuff together.

I was hesitant to make this an individual point because it’s very much related to my overestimate of P(success)… but I still underestimated the amount of code I’d have to write. Looking back, I definitely should have had the sense to consult with someone on how difficult the code would be relative to what my abilities at the time were.

I spent too much time messing with incentive structures.

I spent months trying to find an incentive structure that worked well. I increased the amount I’d incentivize very slowly and experimented in series (one after another). This took way too long. I should have realized that smaller amounts of money weren’t working and I should have experimented in parallel. Doing each of these things would have been a bit more costly, and I was hesitant to spend money, but I think that the time it would have saved me would have made it easily worth it.

I should have been more committed to getting an MVP working.

I spent a lot of time asking people for advice, and “focus on an MVP” was the main thing they told me. I thought that my minimally viable product would have to meet the conditions in my hypothesis above, which were: 1) enough colleges such that a user would expect to find schools he’s looking for, and 2) enough content to be useful. I thought that (1) ~ 300 schools and (2) ~ 10 answers per question. This would have taken a couple hundred thousand dollars to pay for.

When people told me to test a smaller MVP of this, I thought about it, and I didn’t really think there was anything that was both smaller and viable. People were telling me, “try getting answers for less schools and/or less answers per question first”. I thought that this would fail to get users. And I didn’t think that this would invalidate my hypothesis that a sufficient amount of schools and answers per question would get a lot of users.

I interpreted peoples’ insistence on a smaller MVP as (to an extent) blindly following a heuristic that they don’t understand (they really didn’t seem to understand the viable part). I still think that people (to an extent) blindly follow the heuristic and don’t really understand the viable part.

My position now is that:

• There does exist a product that is smaller, yet still viable.
• I didn’t look hard enough for this.
• I was frustrated with peoples’ lack of understanding of MVPs. This lead to my suffering from the reversed stupidity fallacy, which played an embarrassingly large role in my not looking hard enough for a smaller MVP.
• It would have been hard, but if I was pushy enough I probably could have found some local high schools to pay me to collect data for them (in the form of student reviews). This probably should have been my starting point, depending on what the evidence said about P(resources) and P(success | !resources)… because if either of those were high enough I may have been able to skip the MVP step and gone more directly to building a bigger viable product.

I (probably) overestimated the difficulty of dealing with the government.

I didn’t come across particularly strong evidence for this after having failed, it’s just that the conclusion I reach now is different when I reflect on what I know. There were however two experiences I had that added to this:

1. I thought that colleges may provide some help with getting people to answer questions, but they didn’t. Same with high schools and student organizations.
2. I couldn’t even reach any guidance counselors after my pilot program. If I can’t reach them, how would I eventually sell to them?

I overestimated peoples’ desire to help someone starting a startup and trying to do something good.

This will probably sound naive, but I don’t think it is. I thought that people would have some sort of respect for what I’m doing and want to help out. For example, the high school guidance counselors I tried to contact, maybe the default is for them to operate in a bureaucratic way, but I had thought there was a decent chance that they’d think, “Oh cool, look at this kid trying to start a startup, good for him! Hm, he’s actually trying to do something that would be pretty useful. Let me check it out.”

And I when I asked people from my high school to answer questions and spread the word a bit, I thought that a bunch of them would have respect for what I’m doing and want to help out. Almost no one did, which I found to be quite selfish of them. I don’t think my initial expectations were too unreasonable given what I had known, but my beliefs have since been updated.