Big data checklist: all you need to make better data-driven decisions
This is a checklist for people who need a practical guide for how to use big data
Introduction
When properly taken advantage of Big Data has the power to transform businesses. For example, Harvard Business Review’s study of 330 US companies found that those in the top third of their industry for data-driven decision making were 6% more profitable than their competitors.
But, in their quest to implement Big Data, companies risk asking the wrong questions, collecting the wrong information and failing to analyse their information usefully. Even when Big Data is collected and analysed correctly, enterprise organisations may require cultural change to prevent HIPPO (the highest-paid person’s opinion) based decisions from rendering the exercise useless. By using our Big Data checklist, which you can find at the bottom of this article, you can avoid these issues so you can make better, data-driven decisions.
What is Big Data?
Depending on your source, Big data can be explained by between four and seven ‘Vs’. For our purposes I’m going to outline four: Volume, Velocity, Variety and Veracity.
Volume
Given that today, the amount of information that moves across the internet every second is greater than the whole internet was up until the year 2000, the volume of data available to businesses in the era of Big Data is certainly a defining feature.
Due to its slight misnomer Big Data’s volume component tends to be over emphasised. According to the Academy of management journal ‘Among practitioners, there is an emergent discussion that “big” is no longer the defining parameter, but, rather, how “smart” it is — that is , the insights that the volume of data can reasonably provide’
Velocity
To take advantage of Big Data, real-time analytics are required. Reacting to real-time data allows you to take advantage of market movement before your competitors. For example, Lynn Wu and Erik Brynjolfsson used real-time search engine data to make forecasts about the housing market which proved to be more accurate than those produced by the National Association of Realtors who used a more complicated data model underpinned with historical data.
Variety
The variety of data available in a Big Data system means that its data can come in structured or unstructured forms and be produced by a human or a computer. The power of this is that you can take advantage of lots of different types of information to create more well-rounded analyses. The challenge is that you need to have the right tools and experience to know how to compare across categories
Veracity
Veracity is a newer identifier of Big Data, and it roughly equates to how easy the data is to apply to the question that you want to be answered. Whether the inputs are biased or there are abnormalities in your data, both impact your data’s veracity.
What Big Data looks like when it’s done right
Ultimately, whether your data can be considered ‘big’ depends on whether it can be collected, processed and analysed in a short enough time to meet your business needs.
Big data is empowering for management as it lets you break down your traditional business success criteria. Showing you what success looks like in every detail of your processes.
For example, Big Data is used to track every aspect of F1 racing, from the technical performance of components to driver reactions and pit stop times. As the Academy of Management Journal notes, ‘The emphasis thus moves away from outcomes (win/lose race), to instead focus on each proximal, contributory element for success or failure mapped for every second during the race.’
Companies that focus on specific outcomes and collect data from multiple sources centered around their prospective outcomes are far more likely to achieve them. For example the The United Overseas Bank of Singapore filtered around 2 petabytes (PB) of data to detect financial crime. They improved their anti-money laundering detection capabilities through analysing the hidden relationships of shell companies and high-risk individuals. Their adoption of Big Data meant that they shortened the time it took them to find these suspicious connections from several months to a few weeks.
Smart businesses will adapt the outcomes that they focus on based on their Big Data findings. It’s important to take an agile approach. Prioritise one outcome and deliver it. Then repeat this step with your next outcome. You’re likely to take a different and more efficient path than if you set out all your goals at the start of your cycle.
Big Data checklist
- Ask the right questions
Unfortunately, Big Data is not a silver bullet for a successful business. While predictive analytics tools can help with decision making, there still needs to be a practitioner at the head of your transformation team asking the right questions.
This means identifying your KPIs and deciding which KPI linked outcome you want to focus on. Then you need to think about which analysis results you would deem a success. The more specific you are about what you are looking for, the more scientific the outcome and more useful it will be for your business. If we think back to the Formula 1 example the team was not asking questions like ‘How can we win the race?’ but ‘Which pit stop stage is taking the most time?’. That way they will get a useful outcome ‘Wheel change is slowest…we need to focus on speeding up the wheel change over’ rather than a vague and less useful response ‘You need to finish the course faster.’
2. Treat your instincts with suspicion
Traditionally business transformation decisions have been made based on the experience and intuitions of the people at the top of the company. They’d rely on research findings given to them by their juniors (which were sometimes heavily sugar-coated).
This system clearly lacks all of the benefits that Big Data offers, but it is difficult to override hundreds of years of company culture. What we see in businesses which try to adopt Big Data but fail to address the HIPPO based decision culture is that they use their data only when it underpins their existing opinions. Cultural change needs to come from the top. The Harvard Business Review agrees, ‘few things are more powerful for changing a decision-making culture than seeing a senior executive concede when data have disproved a hunch.’
3. Prioritse security
Increased security comes with compromises in terms of speed and performance. So, security often doesn’t receive the attention that it deserves. It’s important to find the right security protocol for your business.
While data can be protected from outsiders using parameter-based security, sensitive data also needs to be anonymised and encrypted to avoid insiders accessing and leaking valuable information. This is a serious concern for companies using Big Data, for example: failing to anonymise data meant that in 2015 Vodafone lost the personal details of 2 million customers following an attack from inside the company. Verizon’s 2013 Data Breach Incident Report indicates that 14% of data breaches came from inside the attacked company.
Customised security is also an option for companies using granular access control i.e: letting different users access different parts of your data. This increases the volume of data in your store, harming performance speed, so should only be implemented where necessary.
It’s useful to regularly perform data security audits where you identify your potential security risks and set plans for mitigating them.
4. Avoid ‘drowning in data’
Viewing your data on too granular a level can lead to analysis paralysis where you are unable to identify trends and make data-based decisions. To avoid this, start by deciding on the problem that you want to solve, make sure it is linked to your business goals and filter out information that doesn’t relate to the question that you are asking.
One of the main obstacles to performing this sort of analysis on Big Data is that it comes from many sources and in many formats. That’s where a system for flattening, amalgamating, overviewing and filtering data is useful.
Kiplot facilitates these actions through its dataset builder and visualisation-filled live data reports. Find out more www.kiplot.com
