F4P Cards Aren’t Samples (continued)

In the previous story, I considered an example of a F4P (fit-for-purpose) Card set and explained how not to make the mistake of treating it as a sample of a larger group of customers.

I find interesting similarities with the justification of the research method given by the business author Jim Collins in his famous book Good to Great: Why Some Companies Make the Leap… And Others Don’t.

Collins and his research team selected 11 successful “good-to-great” companies out of the initial set of over 1000 companies that appeared on the Fortune 500 list during a period of 30 years. They also studied 17 companies, very similar to the 11 at some point (similar industry, size, revenues, etc.), but taking very different paths, ending in failure. By studying the two sets of companies, Collins’ team identified several key traits of good-to-great companies, in leadership, management, culture, and strategy, that accounted for the difference.

Collins appears very thorough in his research. His book’s lengthy epilogue, and that’s the final 20% of the book’s size, explains his research method in great detail. One of the frequently-asked questions is, what about statistical significance of the study given that it included only 11 examples of good-to-great and only 28 companies overall?

Two leading mathematicians helped Collins answer this question. Here’s what they said.

The concept of statistical significance applies only when sampling data. If you don’t sample data, it doesn’t apply, and you don’t have a statistics problem. Collins did almost the opposite of sampling: purposeful selection.

In different words: if Collins selected 28 companies randomly from the Fortune 500, studied them, and tried to extrapolate the study conclusions on the entire set of 500, only then he would have had a statistics problem and had to measure the statistical significance of his conclusions. To increase the statistical significance, to narrow the confidence intervals, he would have perhaps had to enlarge the sample set of 28. Collins did the opposite: he selected 11 companies not randomly, but purposefully, with four cuts and about twenty selection criteria. These 11 “good-to-great” were completely unrepresentative of the Fortune 500: both in how their CEOs led and managed the companies and in the fact that their stocks outperformed investments in the 500 many times over.

Similarly, we don’t sample customers with F4P Cards. We choose our customers purposefully and study the data and narratives they give us. We don’t have a statistics problem with that.