Big Data Analytics — Does it bring possibilities or challenges?
Information Explosion ; Data Tsunami ; Big Data are no more buzzwords and are bringing real value across industries
Living in this digital age where we are surrounded with tremendous amount of data, its essential for us to understand these concepts.
Information explosion was first used in 1940s and Big Data is around since 1990s but how BIG is big data.
Amount of data generated in the world -
2013: 4.4 Zettabytes (ZB)
2015: 18 ZB
2017: 22 ZB
2020: 44 ZB
2025: 175 ZB
In next five years the data generated will grow almost 300% and with a CAGR of 41%.
This is no surprise as smartphones, IoT sensors and other connected devices, are generating zettabytes of data everyday.
As you can imagine, enterprises use this data coming from new data sources (e.g. consumers like us) to extract meaningful & valuable information.
The tools and technologies too have evolved with the time to analyze any amount of data irrespective of type, source and needs.
Most of the organizations knows the benefits of Big Data and how they can get significant value out of it by applying ANALYTICS.
Concept of Big Data Analytics
The concept of Big Data Analytics wherein companies store, process & analyze data is not new but the way it is done now versus in the past has definitely evolved.
The traditional analytics (e.g. basic spreadsheets) have changed into advanced analytics tools to uncover insights so that companies can make faster and informed decisions.
Examples — Hadoop, Predictive Analytics Systems, Data Mining Tools, In-memory analytics etc.
The 4 steps which organizations take to make use of one or more of these analytics tools are -
1. Data Collection
A mix of structured, semi-structured and unstructured data is collected from variety of sources like social media, mobile apps etc.
2. Data Processing
Data is organized, configured and partitioned for analytical purpose by advanced tools and techniques.
3. Data Cleaning
The processed data is checked by removing any errors, duplication, inconsistencies etc. for quality.
4. Analytics
Advanced analytics software with technologies like AI/ML, data mining, predictive analytics etc. are used to enable best outcomes.
Nowadays, most companies are embracing Big Data Analytics as part of their Digital Transformation strategy.
4Vs of Big Data Analytics
Enterprises often struggle with the processing of Big Data so much so that most of the Big Data Analytics projects fail.
Some of the significant characteristics which makes Big Data processing different from other data were defined by Gartner’s Doug Laney in 2001 as 3Vs of Big Data Analytics
- Volume
- Velocity
- Variety
Later, some organizations added more characteristics expanding to some other Vs such as — Veracity, Value and Variability.
The most important addition though was Veracity to the orginial 3Vs, making them 4Vs of Big Data Analytics.
Volume
The sheer volume of data coming from traditional systems as well as external data sources, leads to challenge in processing all the data.
Velocity
Frequent influx of data from multiple sources at a very high speed poses a challenge for real-time insights and analytics.
Variety
Data varies based on sources, type and format, resulting in problems when processing Big Data for quality analytics.
Veracity
Veracity on accuracy of different datasets showing error, incomplete data or showing different records/timestamp for same data etc.
Big Data Processing Categories
Processing of Big Data involves certain steps with commonalities in strategy & software though implementation approach might differ.
The widely used general categories of activities used in Big Data processing are:
- Ingestion
- Storage
- Analysis
- Visualization
Ingestion
The process of adding raw data into the big data system with the help of some dedicated ingestion tools such as Apache Sqoop
Storage
From the ingestion process the data is handed off to components that manage storage. The storage systems are more complex as it sounds because of sheer volume of data.
Analysis
The computation of the available data for analysis to get the desired insights is the most diverse part of any big data system. The data analysis requirements and approach would differ based on the types of insights.
Visualization
Perhaps the most useful layer of any big data system is visualizing data to keep track of trends and making sense of large datasets.
All the different above mentioned categories are essential for organizations to gain value from Big Data.
How Businesses are making most out of Big Data Analytics
There is hype or real-world use cases — you would definitely like to know how Big Data Analytics is changing the world around us.
There is all sorts of data generated from us (social media etc.) and companies leverage this data to improve customer experience by using Big Data Analytics
Some of the industries which are being reshaped by Big Data Analytics are:
Healthcare
Public health and other healthcare institutions are working with Big Data Analytics to make proper use of data collected for example to stop fast spread of Covid-19, currently.
Telecom
Telcos are utilising Big Data Anayltics for improving customer experience for instance with faster response to customer issues and reducing customer churn
BFSI (Banking, Financial Services & Insurance)
Banks are using Big Data Analytics to understand customers better and predict markets e.g. identify and eliminate frauds across BFSI.
Energy
Energy companies across the board are making use of Big Data Analytics for changing consumption patterns, maintainence and outage reduction e.g. smart meters help manage energy usage.
Future of Big Data Analytics
Big Data Analytics have already crossed the hype cycle of towering promises and making strides into boardrooms of organizations.
The technology is now well-established and companies are seeking real benefits from Big Data Analytics if implemented properly.
As companies are powered up to harness the potential of Big Data Analytics, the future looks promising:
1. High demand for data scientists and CDOs (Chief Data Officers)
2. Data privacy and security will be top priorty for instance GDRP (General Data Protection Regulation)
3. Cloud will further break down barriers and we’ll see more migration to cloud as data volumes keep increasing
4. Artificial Intelligence (AI)/ Machine Learning (ML) continue to change the landscape by transforming Big Data into workable stack
5. Proliferation of Edge Analytics would bring real-time actionable Data to the forefront
