Government, stop wasting data
Reflections about big data and information
Big data. Those are the words of the moment. In 2013, big data is predicted to drive $34 billion of IT spending. Big data is being used by companies to give them competitive advantage, whether they sell products or services. It’s a business priority, and not just to the tech industry. But what is big data anyway, and where does it come from?
Big Data definition
The definition of big data is still confusing to many. I’ll break it up for you as simply as possible: basically, big data is data (wow!) and it’s big (wow!). Not helpful at all, is it? Big data is way more than just a typical database. Big data is characterized by three main dimensions, called by IBM “ The Three Vs”: big in Volume, big in Variety, big in Velocity. Experts add a fourth dimension to this definition: big in Veracity.
Volume
The amount of data.
Transactions, records, files, tables. Terabytes of data. Some say that 90% of data available today was produced in the past two to three years. All this data comes from different industries and locations. Many companies still don’t know the amount of data they own.
Variety
Different types of data from different sources.
Structured, semi-structured or unstructured. Processed or raw. Text or multimedia. With the explosion of sensors, smartphones and social networks, data is being generated in countless ways and collected every day.
Velocity
Data in motion.
For many companies, real-time data has an impact on business products and decision making. It’s impossible to process all data in real-time, and some data is more important than other data (for example, finance, GPS, geographic, sports, etc.).
Veracity (The missing link)
Because some data is not accurate, experts believe that verifying the data reliability is important to the quality of the processed data. Certainly, at least once in your life, you must have come across a weather channel (or app) that wasn't reporting the right weather.
Big Data Activities Implementation
A 2012 Oxford/IBM study concluded that, from almost 1100 organizations that were asked about their activities related to big data, 24% had not begun any activities, and 47% were still planning activities. Only about 28% were implementing or had pilots already. Why? About half (49%) of the organizations replied: “Because of the customers, of course!”. What are the outcomes? Organizations actually use big data to make decisions. Smarter decisions. Faster decisions. Decisions that make a difference.
Well, as a customer, I appreciate that. I like the fact that organizations are thinking about me, about my needs, about my desires, whether they’re selling a service, a product or just high-quality processed information. I’ll respect those organizations a bit more because they are putting the user first. So, what’s all this ranting about?
Data that can make a difference
Let us think for a bit. Who’s got access to huge amounts of reliable data (mostly) and from different sources? Governments, of course. They know (almost) everything about you. They have access to tons of registries. They also have access to data generated by various departments, public administration and government-owned corporations. And that data can make a difference.
It was big data analysis that contributed to Obama’s victory over Romney in the 2012 elections. About two months later, the Obama administration announced the Big Data Research and Development Initiative, a project comprising 84 different big data programs spread across six departments. This initiative is intended to explore how big data can be used to address important problems facing the Government.
IMHO, all governments should start a similar initiative. If they don’t, they are wasting data. Important data. As a citizen, I don’t want data to be left aside, unanalyzed, leading to money, time and effort wasting by the public administration. My money, that is. Big data allows a more complete picture of customers’ preferences and demands. And by customers, I mean citizens. Government and government-owned organizations should use those resources to be one step ahead of the competition, but also to offer their citizens better services, products and overall lifestyle. How so?
Think about it. For example, making relevant data more readily accessible across otherwise separated departments can sharply cut search and processing time, therefore creating transparency. Combining data from R&D, engineering and manufacturing units to create concurrent engineering can reduce time to market, and improve overall quality. Analyzing organizations’ variability in performance enables leaders to manage that performance and use it as leverage. By default, in the public sector, citizens are all treated the same way. Big data allows governments to create segments and micro-segments, leading to services and goods customization, targeting specific users or needs (young people, old people, students, and so on). The question is: if they can use big data as an advantage, why don’t they do it?
Well, sometimes there are natural monopolies, like transportation services, mail services, energy services, etc. No competition, no need for a change. Other organizations already have their own strategies, and changing those will take time and money. And some departments just can’t hire data analysts. I can understand that. Public administration is a complex monster, nothing is easy. But wasting data is a costly risk. Wasting data means falling behind. Big Data is a Big Deal.
I strongly recommend reading the Oxford/IBM “2012 Analytics: The real-world use of big data“ study, if you haven’t yet. Gorgeous reading. It explains it all about big data and analytics. You can download it at the IBM website [link], after filling the required information.
A special thanks to my friend Ricardo Carvalho (@rmcarvalho), for fuelling up the idea, and Aarron Walter (@aarron) for sharing his big data analysis strategy at MailChimp.
