How Big Data is like Dumplings…and why I joined BigObject
Delicious image courtesy of Din Tai Fung…
They are some of the most delicious dumplings you will ever have in the world! I recently visited Din Tai Fung in the Taipei 101 building on a trip to Taiwan where I immersed myself in the amazing world of big data.
You may be asking “Titus, how in the world are dumplings like big data?” Great question! It dawned on me the moment my dumplings arrived at my table in a 10 dumpling container. I noticed that with the limitation of the size of the container I could only sample 5–10 dumplings at a time…and this is exactly how big data analysis works today!
From my understanding of big data solutions today, you have tons of data…megabytes, gigabytes even petabytes, and you need to take that data and try and mine meaningful insight from it. The general steps to do so are:
- Clean the data from it’s original source like MySQL or Mongo.
- Transfer the data to a big data solution, like Hadoop.
- Analyze the data with languages like R to mine intelligent insights.
The problem with most big data solutions out there is that they are designed with the limitations of 32 bit architecture and a virtual memory layer. This means that the data mining process can take a very, very long time and can even crash your computers!
Today, data sets of gigabytes or even petabytes are common yet in the early 70's when the database was invented, that amount of data was unheard of due to limitations in RAM and storage.
Current big data solutions are designed to work with a foundation of a 70's way of thinking instead of taking a quantum leap using nBit architecture to process data in close to real-time without the limits of memory and the potential of crashing computers.
It’s time for a 21st century paradigm shift and that is why I joined BigObject and how I came to love dumplings!
Join me on my adventure as I continue to learn more about big data and share with you what I learn so you can successfully surf the coming big data tsunami to profitability!
Until nex time…
Titus Blair, Lead Evangelist @BigObject
If you want to find out how BigObject can help you gain intelligence from your data please follow us on twitter @bigobject, facebook and the web or email me directly titus@macrodatalab.com!
