Big Data: Now
“A new flu virus was discovered. Within weeks, public health agencies around the world feared a terrible pandemic was under way. Some commentators warned of an outbreak on the scale of the 1918 Spanish flu that had infected half a billion people and killed tens of millions. Worse, no vaccine against the new virus was readily available.”
This is the 2009 H1N1 Pandemic.
Today, you in your pajamas are in quarantine. It doesn’t matter how prestigious your ex-school is, because now you attend Zoom High School/College. It doesn’t matter how famous your company is, because now you work at Zoom, Inc.. It doesn’t even matter what your hobbies were, because now you spend the whole day sleeping and complaining.
Little do you know, we are actually in a much better situation than those who suffered through 2009. For them, information wasn’t even real-time; add two weeks to the event and that’s when the Centers for Disease Control and Prevention update everyone else.
It’s not like CDC didn’t want to get information real-time, but because they requested doctors to file reports, and most people don’t look for doctors until they realize something is going wrong. A two-week lag during a pandemic is like looking for an enemy in the dark without a flashlight.
Then, Google came along with the flashlight — big data.
“All their system did was look for correlations between the frequency of certain search queries and the spread of the flu over time and space.”
At the time, this was like stumbling upon a gold mine.
Today, huge amounts of data can be found just by doing a simple search. John Hopkins University has a real-time global map, National Center for Advancing Translational Science has a National COVID Cohort Collaborative database with 780.7M rows of data, and using big data, CommonSpirit can even predict COVID-19 surges.
You might be thinking: sure, big data has applications during pandemics, but in everyday life, big data doesn’t make much of a difference. Right?
The simple answer is: of course it makes a difference.
If you get recommended videos on Youtube, recommended items on Amazon, or just ads related to stuff you searched up a while ago, you witnessed big data. If you have a weather forecast app, get notifications about real-time traffic, or notice KFC or McDonalds change their menu, you witnessed big data.
Big data is everywhere. But why?
“Big data’s ascendancy represents three shifts in the way we analyze information that transform how we understand and organize society.”
1 there is more data than ever before. Data has shifted from just single-use numbers; it is the basis upon which we make our decisions, a critical raw material of business, a box of secrets waiting to be revealed. More importantly, the number of data is rising exponentially, thanks to new processing and collecting technology. With these technologies, everyone can analyze and extract new insights from open data.
2 we accept the messiness that comes with more data. Exactitude is crucial for statistics and samples, but not so important for large amounts of data. Sometimes, more trumps better. For big data, we don’t worry so much about the crooks and crannies — we only care about the overall picture.
3 we don’t just look for causality anymore. In 2009, Google realized the usefulness of correlations and used it to get real-time data. The fact that you searched something doesn’t cause you to be infected, but the two are correlated. In fact, a common mistake analysts make is to think correlation equals causation. This mindset can lead to VERY misleading results.
If big data can do so many cool things, why am I still in quarantine?
Well, yes, big data is very powerful, but it is not the key to every door. With the data supporting decisions, we are ones who have to take action. Instead of treating the COVID-19 Pandemic as nonexistent despite the 37,379,320 cases and 1,075,320 deaths, we must face the situation and DO SOMETHING.
Key Takeaways:
- Big data is a powerful tool that helps us extract new insights and make decisions
- We can see its applications in many fields including health, business, media, etc.
- The rise of big data represents three shifts: more data, accepting messier data, and settling for correlation
- Having the data is not enough, we need to take action!
This article is based on the book Big Data: The Essential Guide to Work, Life and Learning in the Age of Insight.
