Be the Einstein of Your Startup

How to use science to test your startup ideas

Published in

The Startup

9 min readAug 26, 2019

This is Part Four of my Real-Time Journal — Starting a Startup, where I document in real(ish) time our attempts to start a startup.

Signup here to get the next journal entry delivered to your inbox

After a few months of brainstorming and refining, Zach and I had an idea we thought was worth pursuing further, Audio Newsletters. Since we’ve read the Lean Startup and learned the lesson of the “just build it” methodology the hard way, we both knew that the next step was to test our idea using science.

So, with clear eyes and full hearts, we began to science.

Let me tell you a story about the greatest founder of the 20th century.

It’s early 1900. Newtonian physics holds unquestioned dominance over the marketplace of physics theories. Its dominance is so complete that it’s even lost the provisional name of “theory” and is instead referred to as “law”. Even to question its accuracy is reason for immediate dismissal and ridicule. After all, what problems had yet appeared in the universe that Newtonian Physics could not solve?

Then along comes a rogue, socially awkward man with weird hair and a revolutionarily different theory. He works like a maniac on the side of his day job as a Swiss patent clerk for hours and hours every day.

The entire industry of professional physicists thinks he’s insane. He can’t even get a job as a university professor.

But eventually, after an incredible demonstration of the power of his new theory to explain something Newtonian physics never could, he’s proved right and goes on to become the most celebrated scientist in history.

The reason why Einstein’s theory of relativity sounds just like a startup story is because science and startups DO THE EXACT SAME THING. They both generate new knowledge about the world, and in the process overthrow the incumbent operating on now outdated knowledge.

But if we want to use science to help us prove (or disprove) our own startup ideas, we need to understand how science works.

Unfortunately, that’s not trivial. There’s a whole branch of philosophy dedicated to it.

Fortunately, there’s a whole branch of philosophy dedicated to it so some really smart people have spent a lot of time thinking about how science works and we can steal their ideas to test our startups.

For my money, the best explanation for how science works comes from Karl Popper (detailed in The Logic of Scientific Discovery).

The basics are this:

Science starts with problems. We recognize that our current best theories of the universe have problems — things they cannot explain.
We try to solve those problems by developing better theories that solve all the same problems as our old theories and some of the new ones.
We then use creativity and logic to develop “severe tests” of our new theory.
If the theory fails, we go back to the drawing board.
If the theory survives, we subject it to even more severe tests. The more severe tests the theory survives, the more likely it is to be valid.

Popper’s inspiration for and favorite example of how science works in this way was, in fact, Einstein’s theory of relativity.

So with Popper and Einstein as our guides, let’s see if Zach and I can come up with a way to test our Audio Newsletter idea using science.

Step One — Starting with Problems

Popper’s theory of science (and theory of life) is that all knowledge is problem-solving. We notice problems in the world and come up with ways to solve them. These problems can often seem trivial at first.

Before Einstein’s theory of relativity, all physicists knew that some minor problems remained unsolved by Newtonian physics. A few planets didn’t follow their predicted orbits as precisely as they ought to. Furthermore, nobody could explain with Newtonian physics how gravity acts at a distance or what happens to particles as they approach the speed of light.

For whatever reason — interest, rejection of authority, drugs — Einstein realized that these problems might not be small.

The same is often true for startups. Big startups can come from seemingly very small problems with products currently available on the market.

As we’ve discussed previously, Zach and I noticed some problems with the way people consume audio content and newsletters.

While the number of hours people spend listening to audio content is going up and up and up, the amount of short-form audio content doesn’t seem to be going up as much.

At the same time, while the number of niche, paid newsletter is rising rapidly, few of these content creators are making audio content.

Finally, while the number of smart speakers is growing rapidly, few of the niche providers create content for smart speakers.

Step Two — Developing Theories

Popper says there’s no guaranteed way to come up with better theories. It requires an act of inspiration, a bolt of lightning, LSD, whatever. But once you have an idea for a better theory, you can evaluate its internal consistency using logic before testing it.

Internal consistency is important because something cannot be both true and false at the same time (seems like “duh” but think about how many times your boss asked you to do contradictory things).

As a trivial example, if one part of my theory requires the number of smart speakers in use to go up and another part requires that they go down, my theory is not internally consistent and I have to start over.

Einstein famously came up with the inspiration for his theory of relativity by imagining “what it would be like to be a beam of light”. He, as far as we know, did not require LSD to think this way.

But after he developed the inspiration for his theory, he spent years working out the details to make sure that it was internally consistent. He even invited a whole new kind of math called tensor calculus to do so.

Zach and I came up with our theory for solving the Audio Newsletter problem through an open discussion (no beams of light or LSD, unfortunately). Then we refined the idea from something vague to a clear value proposition, namely, that niche newsletters need a way to easily create and distribute their audio content to their subscribers across multiple audio platforms.

We used pitch decks to help us check our idea for internal consistency. For instance, if our theory were true, we would expect people to use their smart speakers to listen to news (which they do).

Step Three — Finding What to Test

Severe tests are the crux of Popper’s view of science. However, as he admits, designing severe tests for our theories is difficult. Usually (in fact always), we can’t test our theory directly because of the physical and epistemological limitations of our universe. So we have to use logic to deduce from our main theory, contingent theories that we can subject to severe tests.

Einstein couldn’t find a way to test his theory for years. It took Eddington, a British experimental physicist, to think of a way to test relativity against Newtonian physics using an upcoming eclipse that physicists.

Zach and I didn’t have that kind of time, so we used a technique that Eric Reis recommends called an “assumptions analysis” to help us think through contingent theories that we might test.

To do an assumptions analysis, you think of all the assumptions implicit in your startup idea. These assumptions are contingent theories in Popper’s parlance. For instance, one implicit assumption or contingent theory, for our Audio Newsletter idea is that people want to listen to niche, short-form content. That must be true for our idea to be true.

After coming up with a list of assumptions, you place these assumptions on a cartesian coordinate graph where the x-axis measures how confident you are that the assumption is true and the y-axis measures how critical the assumption is to your startup idea.

The assumptions in the upper left quadrant are the ones that we were both the least confident in and thought were the most critical to our idea. These are the most important assumptions to test.

What Popper would say is these are our weakest (ie boldest) contingent theories and thus are the ones that are easiest to subject to severe tests.

That’s an important point. Bold, revolutionary theories are easier to test and more likely to wrong. That makes sense if you think about it. If you want to start a vet clinic, you’re don’t really have a revolutionary business idea, so it’d be difficult to design a test to prove that you won’t be successful. But if you have an idea for teleportation, it’s probably straightforward what the test should be (although it may be impractical).

Step four — Creating a Severe Test

Now that we know which assumptions, or contingency theories, we need to subject to severe tests.

Designing that test is a problem in its own right and thus requires an act of creation. Popper discusses how we need to be creative and opportunistic when designing our tests.

As mentioned above, Eddington realized an upcoming eclipse would be a good test of Einstein’s theory.

For our assumption — that people want to listen to niche, short-form content — it just so happens that I have a friend who runs a local political newsletter here in Austin.

For our test, we created an Alexa Flash Briefing Skill for his newsletter so we could see how many of his subscribers installed the skill and listened to the update.

This is a severe test. Users would have to install an Alexa skill and listen to his update regularly to it validate our assumption. It was also cheap, fast, and easy.

Step Four — Continual Testing

As Popper is careful to point out, we never conclusively prove or disprove anything in science. We can always create more explanations for why a theory failed or passed a test. The best we can do is hope for more and clearer evidence.

However, just because we don’t know for sure, doesn’t mean we don’t learn anything by testing our theories.

In Einstein’s case, relativity so precisely predicted the results of the eclipse, that the scientific community realized he was on to something and came up with new and creative ways to test his idea. Now we’ve even tested his theory by putting a super-precise atomic clock a hypersonic airplane to see if the clock speed up relative to a clock left on the ground (spoiler alert, it did).

In our case, if 50% of the subscribers had installed the Alexa Flash Briefing and listened to it every day, that would have been a strong piece of evidence in favor of our idea.

However, that’s not what happened. About 2% of subscribers installed the Flash Briefing and none of them listened to it very often.

Of course, the Alexa test, like all tests, is not perfect.

Users might want to listen to a short update about Austin local politics but might not have an Alexa.

Or they might just consume their political news when they’re not home with Alexa.

Or they might not have received our email.

All of these are possible reasons why the test might yield a false negative.

But for us, it was enough to ditch the audio newsletter idea and move on to another one. Perhaps we were wrong to do so. But at least we had some evidence to back up our decision.

Conclusion

Things we tried:

👍 Science

👍 Karl Popper

👍Albert Einstein

👍 Assumption Analysis

👎 Audio Newsletters TD

🤷‍♂ LSD (we didn’t actually try this)

Takeaway

Using science to test your startup ideas is hard because science is hard. Despite taking the time to develop contingency theories and design severe tests, you still always get inconclusive results.

But, as Zach and I discovered with Owl & Scroll, the alternative is worse. Spending hours and hours on an idea without any evidence requires that you luck your way into a startup. Science is far from perfect, but it is surely better than that.