What is big data?
Data is all around us. It includes millions of observations covering areas such as consumer opinions, economic commentary, future expectations, expert opinions, and transaction data such as POS observations. Big data refers to data sets so large and complex that they become difficult to process using conventional database management tools or stock standard data processing applications such as Excel. Big data is not an end in itself — data is important only to the extent it is efficiently and effectively incorporated into decisions.
The concept of big data varies depending on the size and capabilities of the organisation managing the data, and on the capabilities of the applications that it uses to process and analyse the data. For example, organisations suddenly facing hundreds of gigabytes of data for the first time may need to reconsider their data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration. As the amount of data created and stored continues to grow (doubling every 40 months), more organisations will have to confront this issue.
Why is it relevant to Australian businesses?
The capacity to store and retrieve large quantities of information has created an opportunity for all organisations to leverage big data for their own business intelligence (BI) purposes.
Over the past two decades Australian organisations have allocated substantial financial and physical resources to the development of mass databases and to the collection and maintenance of transaction and customer specific data. Companies like Oracle, SAP and Microsoft have provided platforms for large organisations to modernise their data capture capabilities and centralise their reporting functions, dealing not only with the volume and the variety of data (e.g. quantitative and qualitative data) but also the speed with which it is captured. While this has been an important first step, data capture and centralisation is a far cry from data analysis or what we term predictive analytics.
What is predictive analytics?
Information about inter-relationships, trends, events and patterns is embedded within big data. Predictive analytics involves extracting information from data and using it to predict future trends, unknown events and behavioural patterns.
Common pitfalls
Databases developed by organisations are often augmented with BI frontends that present high level business metrics. Sometimes one-size-fits-all predictive modules are added to the applications with the ostensible role of using the information stored in big databases to forecast sales, financial variables and business drivers. This sounds good in theory. In practice, however, there are real risks associated with applying stock-standard forecasting software and algorithms to different types of data. Often the data that firms use to make predictions cannot be modelled with stock-standard forecasting approaches found in standardised BI software.
In our experience, many firms overlook the importance of undertaking detailed analysis of the data available to them and focus on IT specifications. Too much attention is paid to integrating models into existing software platforms such as SAP Business Objects, with insufficient emphasis on the objective itself, being the provision of useful predictions. This results in the data analysis taking a back seat to visual functionality or technical matters such as systems integration. In the end, organisations spend millions only to find that they’ve acquired a tool that is only superficially customised, and does little to assist in decision making and planning.
Additionally, conventional, pre-existing methods are often employed to develop models. Mathematical and statistical innovations typically take years before they filter through to the commercial world in the form of practical easy-to-use software packages. In our view, today’s commercial leaders should seek out and take advantage of these innovations early on — before the rest of the competition catches up.
Concluding thoughts
Predictive analytics is about the combination of statistical and quantitative expertise with information technology and market knowledge to significantly assist decision-makers. It is not about crystal-ball gazing, nor is it a precise science. No-one can predict the future without error, however using predictive analytics to make your decisions 1%, 5% or 10% more accurate is an enormous competitive advantage.
Ensuring that your data undergoes detailed and thorough analysis by experts cannot be replaced by software — you must invest the time and money to fully understand your data. Companies that invest in data analysis will benefit from the full potential of predictive analytics.
