Bullet Holes, WWII planes and Startups

Most people have heard of the Manhatten Project and its contributions to science, fewer people are aware of the work done by a collection of mathematics based out of Columbia University. Wars are won by 5% improvement. 5% faster troops, 5% more efficient planes 5% faster tanks, 5% more accurate bullets 5% more efficient supply routes and so on. The Statistical Research Group at Columbia was tasked with helping the war effort via math.

One of the projects was determining where to armor planes. Too much armor made the planes too heavy to fly but not enough meant plans were getting shot down. Most people’s reaction to the problem was to look at the planes after they came back from battle and see where they had been shot and put more protection on where the most bullet holes are. Seems pretty intuitive and reasonable, if you want fewer planes to be shot down protect them where they are most frequently hit.

However, it turns out that is precisely the wrong strategy it took a statistician, Dr. Abraham Wald, to see the flaw in this logic. Most people were only looking at the data from the planes that made it back. The planes that were shot down were unaccounted for in the data about the distribution of bullets. So it stood to reason any place we had data about bullet holes in planes wasn’t that bad a place for a plane to be shot, the places where there were zero bullet holes are the places to protect because it is those places where a bullet strike will take down the plane.

While this is a cool fun bit of math nerd history, you are likely wondering what does this have to do with startups?

As an executive when you are looking at your metrics your job is to ask where the planes that didn’t make it back are and what happened to them?

The way in which we measure performance often doesn’t encourage or force out of the box thinking. Nothing in the creation of a good OKR prevents you losing track of the missing planes. With the current obsession of metrics and growth hacking, it can be all too easy to fall into traps of only looking at what can be easily measured.

We are obsessed with adding new customers not making the existing ones more profitable. Markets and VC’s reward growth above all else, as result we worship CAC/LTV, a ratio that most founders live and die by. Any executive who doesn’t know or understand their CAC/LTV regardless of what you manage should be a major red flag, but while it is a super useful shorthand it doesn’t always give the fullest picture. For example, customer retention costs are not factored in, fewer executives know their increase in LTV per dollar of Customer Retention spend. Knowing what the right mix of dollars to spend on increasing LTV vs acquiring new customers empowers far better capital management. As does understanding your working capital. CLTV/CAC (CLTV is lifetime value minus COG) is critical because that ratio is the amount of working capital a company gets per user/sale. While lately, it may feel like growth is king, the reality is cash is always king and high growth companies are dependant on the ability of additional capital which means the future of a high growth company is in the hands of those with the cash. Controlling and understanding your own cash flow is equally if not more important than growth.

Looking at a more customer-focused example of missing planes. Uber in 2016 famously had a terrible app store rating because they had broken with prevailing wisdom and didn’t offer in-app prompts to rate the app. Not shockingly their survivorship bias skewed terribly because the only people who are likely to open up the app store find -> find an app -> leave a review are the pissed off ones. Customers tend to only offer feedback when things go exceptionally well or exceptionally poorly. Had Uber taken that limited information set as gospel they might have made drastic changes to the product when the reality was they just needed to have better insight into their planes (reviews) that were not making home.

Another consumer focused example is when companies assume competition is a zero-sum game (winner take all). It would be easy for either Hulu or Netflix to assume that any user or traction the other gains is a loss for them, however, it turns out that the majority of users subscribe to at least 2 on-demand video services. Not only that once a consumer adopts a behavior shift away from traditional programtic and into their first OTT media consumption their CAC for other OTT providers drops. If you subscribe to Netflix you already understand the value proposition for Hulu which means Hulu only has to prove additive value on content not on value proposition education.

Dating apps integrate with Instagram because they figured out how many people were closing their app to go to Instagram to look up a potential prospect. By allowing you to check the most recent photos on someones Insta both apps saw better engagement. Now it is true some games are zero-sum, a person who buys a BMW is unlikely to buy an Audi within the same time frame. It just requires you as an executive to understand your market and if you are playing a zero-sum or a collaborative game.

Amazon learned that people have an irrational hated of shipping fees, Amazon figured out how many orders were being lost because people are irrational and bad at valuing their time. The introduction of prime, which subsidizes some of the shipping costs have made Amazon what it is. Another brilliant move is the introduction of allowing people to opt into slower shipping, allows the customer to communicate what is important and reduce cost in one clean swoop. The real magic though is the use of a 5 dollar coupon as a reward for choosing slower shipping that allows Amazon in the worst case to drive existing customers to try out new offerings like PrimeNow and in the best case, a customer feels as though they have been rewarded and never actually uses the rebate. If you are like me I have fallen into this trap more times than I would like to admit, I have hundreds of dollars in shipping rewards I will never use. Either way, Amazon wins.

While each of these examples may seem far apart all of them come back to understanding what assumptions have been made and taking a step back to look a bigger picture. Just like war is won by 5% so is business, think about where your missing planes are.