The Data Sniff Test

I was watching CNBC yesterday morning, who was reporting a New York Post story based on a third party data report that Instagram lost 25% of its user base in a matter of days as a result of changes to its privacy policy.

This didn’t pass my sniff test because it just seemed extraordinary unlikely that Instagram’s terms of service changes outraged such a massive percentage of users in such a short period of time. So much so that these users left the service en mass.

If anything, the chart looks more like a data error. As if somehow AppData become unable to capture a portion of Instagram usage starting on the 24th.

Yet this story went on for hours. Sending FB stock down 3% on the open:

Zach Seward at The Atlantic has a great writeup explaining the whole thing, and Facebook commented to Techcrunch:

“This data is inaccurate. We continue to see strong and steady growth in both registered and active users of Instagram.”

It took hours for this all to get straightened out, but even within the first minutes of so many outlets running the story, it didn’t smell right and ran against gut instinct. Is the average person’s affection for Instagram trumped by terms of service changes? Most of the people I know using Instagram don’t even know about the kerfluffle.

False precision and the reliance on data in the absence of back of the envelope math or observational logic almost always results in mistakes. If the numbers defy logic, just because they are numbers, does not allow them to trump that logic. And often, back of the envelope calculations, accompanied by logic are far more accurate and actionable.