Big data — common pitfalls

Recently, Volvo created an application called roam delivery service which allows deliveries to be made to your car while the car sits unattended in the car park. Couriers access your car to deliver the goods using a single use digital key, which is disabled once the delivery is complete. What this demonstrates is something we already know: the appetite for advanced information services is huge and expanding exponentially. These services are only possible because we’re able to leverage insights derived from massive volumes of data, generated both by people and ‘things’.

Did you know that we, the human race, now create a quintillion bytes worth of data every day? It’s impossible to even fathom what a quintillion of anything looks like; nothing in our everyday experience could possibly put these numbers into perspective. The name we use for all this information is ‘Big Data’ and according to the websites of many major advisory firms, it’s the most important new frontier for business.

There’s no question that organisations are now capturing unprecedented amounts of data. For this we can thank more effective data management systems; increased connectivity through smart phones and other devices; advances in satellite information and imagery; the ever increasing adoption of social networking, and; new technologies designed specifically to track customer’s personal information and preferences.

The usefulness of data is, however, not measured by its volume. Large, unfocused data collection exercises often yield relatively little new information. Recently we completed a major piece of work developing a new business model for a key Victorian State Government department. Before developing our model we tested the significance of over 250 variables which were believed to be important drivers of demand for the authority’s services. What we found was that only a handful provided any real value or significance from a forecasting perspective.

The current hype surrounding Big Data is almost entirely focused on techniques for collecting and storing data. Questions raised usually take the form: “Do I continue expanding the size of my local servers or do I leverage the cloud?” and “If I use the cloud, should I use the public cloud, or invest in a private cloud infrastructure or some hybrid?” This focus on technologies for collecting and storing large volumes of information is misplaced.

Collecting and storing information is important, but only to the extent that it is efficiently and effectively incorporated into everyday decision-making. Therein lies the problem for most organisations struggling to understand what to do about Big Data. Unless the right questions are asked, critical investments in data acquisition and management systems may be wasted.

The fundamental question we must ask ourselves is: “How do I use this data to make better decisions which will increase customer loyalty, improve productivity and save money?”

Basically the value of Big Data is determined by data analysis. Data analysis is what we do in order to provide value-added services, for example fridges and coffee machines that can keep inventory and reorder stock when running low. Predictive analytics take this one step further, using insights from Big Data to provide expectations of future outcomes based on any number of possible scenarios, such as the effect of a new competitor in the market or a shortfall in supply or demand for a product or service.

Despite the complex mechanics that sometimes underpin the analysis, all businesses really need to know is that, if used in the right way, Big Data (coupled with advanced data analysis) can result in increased profits and significant financial savings.

For organisations who want to capitalise on these insights (and any leader who doesn’t is going to be left behind) you need to grasp a few realities first:

• Most of the information being stored has negligible value
• The only way to determine what is meaningful is through sophisticated data analysis
• Off the shelf software packages and generic front-end Business Intelligence (BI) interfaces, which focus largely on data visualization, will not provide you with a competitive advantage (your competitors are using exactly the same tools)
• Only predictive analytics (complex, adaptive statistical models) specifically tailored for your organization and operating environment will give you the full range of insights you need to prepare for every eventuality

Far too often organisations, interested in harnessing the potential that predictive analytics can provide, focus on IT specifications. They spend millions on tools that are only superficially customised and do little to assist in decision making and planning. Insufficient emphasis is placed on the objective itself, the provision of useful predictions. Instead, these insights are often lost in the focus on visual functionality and systems integration. In addition, stock-standard statistical models are applied, which cannot adapt to valuable new sources of information derived from the gigantic Big Data reservoir, without significant further investment and adaptation.

Any analyst who assumes that insights into the year ahead can simply be derived from last year’s data will be disappointed. Millions of observations, including consumer sentiment, economic commentary, futures trading and speculation, as well as transaction data (such as ‘point of sales’ info) must be taken into account in order to develop truly useful forecasting models and simulators.

Unfortunately, the latest mathematical and statistical innovations can take years before they filter through to the commercial world and into easy-to-use off-the-shelf software packages. This isn’t good enough. Today’s commercial leaders need to seek out and take advantage of the latest statistical innovations as quickly as they can to stay ahead.

This is where Predictive Analytics Group can help. Our comprehensive analysis of what data is actually important, coupled with the most advanced adaptive statistical models on the market, provides us with a distinct advantage over those cookie-cutter data management and visualization tools.

The first thing leaders and managers need to ask themselves, in order to really harness the power of Big Data, is “How will these tools help me make better decisions?”