This is why you’re failing at Big Data
By now you’ve heard about the three Vs of big data. Whether your concern is millions of research panel records, billions of transactional records, or trillions of web tracking records, we all have the same problem. The volume of data increases exponentially, the variety of data keeps increasing, and the speed, well, let’s think lightspeed. These issues alone make big data a worthy opponent.
Big data is also rife with missing data. It’s incomplete data, and it’s complicated data. It needs specialized analytical tools and specialized analysts. But those problems are also not the reason we’re failing.
Why are we failing at big data? Well, let’s take a step back to the survey and focus group world that market researchers love so much. When I think back to the last survey I wrote, it too was quite the beast. For just twelve minutes of respondent time, I spent many hours dreaming of, writing, tweaking, rewriting, and retweaking every single question and answer. I pondered every the, or, if, they, you, and why. I argued with myself about the possible ramifications that every single word might have on my results. In every case, I settled on the best solution, not the right solution. In the end, I had a survey that would carefully address every single hypothesis and research objective on my list. This survey was a beauty and the analysis was quick and easy.
Let’s move forward to our big data project. You know, the one where someone dumped a giant SQL database with thousands of variables and billions of records on your plate and said, “Make our program better.” You weren’t really sure what the program was, you didn’t know what was currently good or bad about it, and none of the database variables matched up with any project plans or research objectives. Actually, there were no research objectives. Except for “make better.” I can assure that is NOT a solid research objective.
Imagine if someone collected together a hundred surveys from a hundred projects and told you to “make better.” I can guarantee you would fail at that survey analysis regardless of how many years of survey analysis you had behind you.
The simple reason we continue to fail at big data is that we fail to create concrete and specific research plans and objectives as we do for every other research project. We know very well that a survey project will fail without carefully operationalized objectives but when we work with big data, we ignore this essential step. We don’t plan ahead with specific variables, we don’t list out potential hypotheses, we don’t have a game plan. “Find something cool” isn’t a game plan. Nor is “how can we improve?” Big data needs big brains to plan and organize and be specific.
Do you want to succeed at big data? Then stop treating it like a magical panacea and do the work. Do the hard work.
