Numbers Could Be Lies

I read “Hooked: How to Build Habit-Forming Products” by Nir Eyal and Ryan Hoover recently. It’s an excellent book that explains how and why people use some products frequently (multiple times a day) and forget about some other products.

The case studies and research findings mentioned in the book are all great inspirations for building products. However, I don’t agree with one of the experiment approaches, which is an example of getting wrong signals from the wrong audience.

There is a section in the book — “The Framing Effect” — about a social experiment that draws the following conclusion:

The mind takes shortcuts informed by our surroundings to make quick and sometimes erroneous judgements.

The experiment is asking a world-class violinist to play a free concert in a subway station. People paid hundreds of dollars to listen to his concert in the Kennedy Center and Carnegie Hall, but rarely stopped to listen to his violin in the subway station.

The conclusion makes sense that the surroundings affect people’s perceptions. But an assumption in the experiment doesn’t look right: people buying tickets to the concert are the same as the people passing by the violinist in the subway station.

Product/service value is always a relative concept depending on your preference. The violinist’s performance might be worth hundreds of dollars to some, while meaning nothing to others.

I am not a violin lover (I know this is sad, but I just couldn’t find any pleasure listening to any violin performance). So the value of performance to me is $0. That means, even if I received a free ticket to the concert, I wouldn’t go because it still costs me time sitting there. If I had ever come across the violinist playing in a subway station, I wouldn’t stop by either.

I am not his target audience no matter where he is playing his violin. I bet most of the people who passed by the violinist were not a violin lover either; at least not passionate enough for them to stop.

The conclusion was not fairly made. It is no different from conducting an experiment for 2 shavers by showing A to 100 men and B to 100 women, then making a conclusion that shaver A is better than B with more positive feedback from 100 men.

The experiment would be more convincing if it was observing whether people stopped by when the violinist was playing outside a violin shop (or outside the hall of an upcoming violin concert). As the people passing by are more likely to be some violin lovers.

When everyone’s talking about being data-driven in the startup community, it is common to see the world in a flattened way by assuming everyone’s the same and generalizing them as a number.

There are a lot of A/B tests and experiments. But the results could be lies if you don’t understand the traits of your users and don’t segment them accordingly.