Big Data Analytics: The best disruptive thing you can do to your organization

Why everything you think you know about getting value from data is wrong and how to avoid being crushed by your competition.

By Devin Bost

The reality of big data analytics

From a survey of nearly 3,000 executives, managers and analysts across more than 30 industries and 100 countries, LaValle, Lesser, Shockley, Hopkins, and Kruschwitz (2011) reported:

“Top-performing organizations use analytics five times more than lower performers. . . Senior executives now want businesses run on data-driven decisions. They want scenarios and simulations that provide immediate guidance on the best actions to take when disruptions occur . . . Executives want to understand optimal solutions based on complex business parameters or new information, and they want to take action quickly. . . These expectations can be met — but with a caveat. For analytics-driven insights to be consumed — that is, to trigger new actions across the organization — they must be closely linked to business strategy, easy for end-users to understand and embedded into organizational processes so that action can be taken at the right time. . . We found that organizations that strongly agreed that the use of business information and analytics differentiates them within their industry were twice as likely to be top performers as lower performers. . . They were twice as likely to use analytics to guide future strategies, and twice as likely to use insights to guide day-to-day operations. . . They make decisions based on rigorous analysis at more than double the rate of lower performers. The correlation between performance and analytics driven management has important implications to organizations, whether they are seeking growth, efficiency or competitive differentiation. . . The leading obstacle to widespread analytics adoption is lack of understanding of how to use analytics to improve the business, according to almost four of 10 respondents. More than one in three cite lack of management bandwidth due to competing priorities.”[1]

However, Grover, Chiang, Liang & Zhang (2018) reported that 60 percent of big data projects through 2017 would fail. Why are so many of these big data projects failing if top performing companies make big data analytics such a high priority? What are the key differences between companies whose big data projects fail and companies whose big data projects succeed? We set out to figure out why some companies failed to make use of effective big data analytics and how top performing companies operated differently to obtain significant value from their big data efforts. We discovered a number of key distinctions.

You are in for a major paradigm shift.

Start by thinking very carefully of everything you know about data. Think about how you’ve stored it, when you’ve gathered it, why you’ve collected it, what you’ve done with it, how you’ve utilized it, how you’ve archived it, how you’ve backed it up, and how you’ve used it for reporting and analytics. Think about how you’ve used it in web applications and desktop applications. Think about how you’ve preserved it and protected it and kept it safe. Now take all of those ideas and put them away. All of them. We’re going to start with a clean slate. And if you’re really on the cutting edge, then your current understanding will be strengthened. But if not, then you are in for a major paradigm shift because the technology is changing the way that we think about data, and if you’re not on top of it, you will be left in the dust.

Let’s start with some of the reasons why. According to Alharthi, Krotov, and Bowman (2017), “Currently, about 90% of the digital data available was created in the last 2 years.”[2] That’s the rate that the world is collecting data. But is it true for your organization? In the wake of so many innovative technology trends, with so much data becoming available, what is the most impactful thing that companies can do to take advantage of it? Grover, Chiang, Liang & Zhang (2018) answered, “[N]o single business trend in the past decade has had as much potential impact on incumbent IT investments as [Big Data Analytics].”[3] So, some companies may be thinking, “We just plug in some ‘Big Data Analytics,’ and our problems will be solved, right?” Unfortunately, this type of thinking tends to get companies into a lot of trouble, and we can review what happened during the big ERP boom over a decade ago to see examples of this. Companies that expect to invest in big data analytics and have their problems magically solved will be greatly disappointed unless they adopt the necessary organizational changes that are required to obtain the massive benefits of big data analytics. To this point, Grover, Chiang, Liang & Zhang (2018) warned:

“Gartner predicted that 60 percent of big data projects through 2017 would fail to go beyond piloting and experimentation and would be abandoned. . . If big data are viewed as a valuable asset, it is imperative that businesses determine their actual accounting, economic, financial, or strategic value. . . Data not only exhibit increasing returns but could also yield more value when they are integrated with data from other sources. Furthermore, data are only the input to generating knowledge and insights valuable for decision making. The value of data is revealed through the combination of insight generation and its actual use.”[4]

But wait a moment… 60 percent failure? Does this mean that big data analytics is a high-risk investment that may not be useful to businesses on its own? Let’s look a little deeper. Zakir, Seymour, and Berg (2015) reported:

“Forrester Research estimates that organizations effectively utilize less than 5 percent of their available data. . . In 2012, McKinsey & Company conducted a survey of 1,469 executives across various regions, industries and company sizes, in which 49 percent of respondents said that their companies are focusing big data efforts on customer insights, segmentation and targeting to improve overall performance [10] An even higher number of respondents 60 percent said their companies should focus efforts on using data and analytics to generate these insights. Yet, just one-fifth said that their organizations have fully deployed data and analytics to generate insights in one business unit or function, and only 13 percent use data to generate insights across the company. As these survey results show, the question is no longer whether big data can help business, but how can business derive maximum results from big data . . . [C]ompanies who orientated themselves around fact based management approach and compete on their analytical abilities considerably out-performed their peers in the marketplace. The reality is that it takes continuous improvement to become an analytics-driven organization”[5]

So, we can see that it’s not the technology alone that determines whether a company will obtain value from an investment in big data analytics. Rather, obtaining value from big data analytics largely depends upon how effectively a company can:

1. embrace a fact-based management approach,
2. compete on their analytical abilities, and
3. continually improve to become an analytics-driven organization.

How much can companies that properly leverage Big Data Analytics improve their operating margins?

According to Wamba, Gunasekaran, Akter, Ren, Dubey, and Ghilde (2017):

“[Big Data Analytics (BDA)] is now considered as ‘a major differentiator between high performing and low-performing organizations,’ as it allows firms [to] become proactive and forward looking, decreases customer acquisition costs by about 47% and enhances firm revenue by about 8% (Liu, 2014). . . Indeed, almost 35% of purchases made on Amazon.com are generated from personalized purchase recommendations to customers based on [Big Data Analytics] (Wills, 2014)”[6]

Back in 2011, McKinsey & Company had already discovered the potential value of big data analytics and reported, “In the private sector, we estimate, for example, that a retailer using big data to the full has the potential to increase its operating margin by more than 60 percent.” [7] But they sternly warned:

“The United States alone faces a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts to analyze big data and make decisions based on their findings . . . who can ask the right questions and consume the results of the analysis of big data effectively.”[8]

They also warned, “In a big data world, a competitor that fails to sufficiently develop its capabilities will be left behind.”[9] Are we starting to understand now why Gartner predicted that over 60% of big data projects would be abandoned through 2017? Clearly, the problem is related more to organizational culture and management capability than it is about whether the technology has the ability to deliver the promised value.

So, what management changes do organizations need to make to obtain value from Big Data Analytics?

Management culture needs to embrace the process of making data driven decisions. Gandomi and Maider (2015) noted:

“Big data are worthless in a vacuum. Its potential value is unlocked only when leveraged to drive decision making. To enable such evidence-based decision making, organizations need efficient processes to turn high volumes of fast-moving and diverse data into meaningful insights.”[10]

They noticed several key processes for organizations to make use of big data:

From a management perspective, making use of big data begins when someone (e.g. a decision maker) raises an important question. Once the question is utilized to begin the big data management processes (as displayed in the figure above), human resources need to be quickly coordinated to ensure that the required data is identified, collected, integrated, cleaned, analyzed, and then reported back to management so that a decision can be made. This process can feel very uncomfortable for management who are inexperienced with making data driven decisions. Some managers impatiently prefer to make an immediate decision before obtaining the results of the data analysis, and this is especially common in companies that have not acquired the talent or established the processes that will enable rapid iteration of the big data processes that enable managerial questions to quickly be answered with data so that a correct decision can be made. When the process indicates that additional data needs to be collected, it is advisable that additional related questions can be considered as well so that additional data requirements can be identified as early as possible so that additional data can be collected and integrated to anticipate future questions that may benefit from the data. Without full managerial and team-member support of this data-driven decision process, benefits will be lost, insights gained will remain uselessly in a vacuum, and value gained will likely not exceed the costs.

Some may ask, “Are you asking us to completely change the way we make decisions as a company? Isn’t it unreasonable for us to be expected to make such drastic organizational changes so quickly, especially due to some technology fad that will likely die anyway?” Surely there must be some skeptical managers who are wary to change all of their company priorities in favor of what may seem like a technology fad that they think may be associated with a high risk of failure and a high cost to the company’s future if it fails. However, history has a tendency of repeating itself when people have not learned what they needed to, so perhaps unsurprisingly, the 60% failure rate (of projects related to big data) is actually not a new statistic.

Let’s review what history has taught us — Examining the ERP trend

Do you remember a number of years ago when the new technology trend was for companies to acquire Enterprise Resource Planning (ERP) systems? Many people in the industry were raving that ERP systems would amplify company profits, save companies tons of money, and if companies were not taking advantage of these systems, they were certainly doomed to failure. Many companies bought into the hype with a hope that it would make them more profitable. However, many of these companies became traumatized when they realized that their enormous required investments had completely failed because they did not get the value they were expecting from these ERP systems. Over a decade ago, Loh and Koh (2004) reported:

“Despite the extensive research on improving the success rate of ERP implementation, Buckhout et al. (1999) found that 70% of ERP implementation projects fail to achieve their corporate goals. The high failure rate of ERP implementation calls for a better understanding of its critical elements (CEs) that constitute a successful ERP implementation. Before we dive into the technology and examine ways to decrease the cost of implementing such technologies or strategies, let’s look at the underlying business principles that can make or break any related efforts.” [11]

This was not an isolated finding. In 2009, these findings were confirmed, as reported by Chen, Law, and Yang (2009), who stated:

“Despite the popularity of ERP, the failure rate of ERP implementation remains high. According to a survey of 117 organizations conducted by the Conference Board, 40% of ERP projects failed to meet the business case [12]. This result is corroborated by another study done by information technology (IT) management consultancy Robbins-Gioia LLC, which found that 51% of companies across a wide range of industries stated that their ERP implementations were unsuccessful [50]. Thus, it is critical for executives and managers to fully understand and manage project management issues so that effective approaches can be devised to address project management problems, mitigate interruptions to daily operations . . . and realize the benefits of enormous investments made [41].”[12]

Thankfully, after noting a statistic about the scope of companies adopting ERP systems, a key broad-sweeping insight was reported by Ehie and Madsen (2005), who noticed:

“It is projected that over 70% of Fortune 1000 companies have or will soon install ERP systems and that ERP systems are penetrating the small-to medium size companies with gross revenue less than US$ 250 million [1]. The inability of these companies to realize competitive advantage from ERP implementation is attributable to failure of proper usage of technology to address changes in the design and structure of an organization. Organizations that realize full benefits of a technology are those that make necessary changes in their organizational structure, strategies, and processes [8].”[13]

So, what were reasons for success and failure of ERP projects, and are they the same reasons that big data projects often fail today?

In 2011, Hwang and Grant noticed that the most influential factors of whether ERP systems were successful related to two primary subjects:

1. the specification of the software itself (i.e. ensuring that the software requirements matched what was actually delivered/implemented), and

2. integration between people and the software.

Then they clarified what they meant by “integration” and delivered a striking blow to many companies who expect new technologies to solve organizational problems without requiring total transformation of the company’s processes and operations:

“Integration is the linking of related components to form a unified whole. It provides the foundation for coordination, collaboration, and synergy, and it emphasizes a holistic approach to decision-making, management, and control. We define integration as the collection of related entities, such as computer information systems, manufacturing systems, engineering systems, production systems, management systems, distribution systems, financial systems, accounting systems, and users, to form a unified whole [18]. These entities, when optimally combined, should perform in concert to support and achieve an organization’s goals and objectives. Entire organizations, not just the manufacturing function, should be well integrated if they are to successfully compete in the global economy. The timely information required for collaboration, coordination, synergy, control, decision-making, and management of organizations will not be realized if companies avoid taking a holistic approach to integration [18] . . . The problem resides in an older industrial mindset that still dominates many managers, namely, the ‘technology imperative,’ which views technology as an exogenous driving force that determines or constrains the behavior of individuals and organizations. Unfortunately, this technology-dictates-itself mindset no longer works in a highly uncertain and competitive post-industrial environment. When identical technology is available and easily duplicated, sustained technological advantage is not the result of having it but using it effectively. ERP is not a panacea for all performance problems, but an enabler of business process integration.”[14]

Notice that they mentioned that organizational integration is the key to successful implementation of a large technology like an ERP system. This brings us to a pivotal point: The technology is the enabler, but it alone is not the solution. With the new wave that represents the up and coming “Big Data” trend, we are hearing essentially the same message. The only difference is that the context is Big Data, rather than ERP.

But were there any other predictors of success or failure?

Before we dive back into reporting on exactly what changes companies need to make to successfully implement big data analytics, we will briefly review one additional finding and consider it as an analogy. Hwang and Grant (2011) also noticed a distinction between usability integration (meaning integration between people and the software) and integration of the software and systems. I will comment on the usability aspect briefly because it ties back into another subject where value can be derived from effective big data analytics and machine learning that relates to creating personalized experiences. In that study, Hwang and Grant mentioned:

“There are various ways to investigate the role of users in ERP integration. One way is through the graphical user interface (GUI), which serves as the primary vehicle for interacting with systems. It serves several purposes including task enrichment, productivity enhancement, and navigation; GUI represents the embodiment of the system for users and acts like a lens through which they view the system. The level of satisfaction or dissatisfaction derived from its use is an important measure of ERP success and performance. Studies have found the level of customization of an application and GUI is a significant variable of ERP success [9, 22, 25, 32].”[15]

Isn’t this true with any software? If the users feel like the software makes them less productive, makes their job harder, or causes confusion or creates new problems that do not outweigh the benefits, it does not take an expert to recognize that the users will hate it; and if enough of the users are dissatisfied, they will reject the software, and the entire implementation will fail. So, from this analogy, we can infer an additional key aspect to obtaining a successful implementation of big data analytics:

Improving the user experience.

But some readers may think, “Wait a minute… I thought big data analytics were for decision-makers, not for end-users.” Let’s keep in mind that there are a few primary groups of users who may benefit the most from the results of big data analytics:

· customers

· internal decision makers

· stakeholders

So, if “users” (at least in our case) are defined as managers, team members, customers, stakeholders, or decision makers who need to make decisions that could (or should) be influenced by the results of data, then the analogy should be clear.[16] Their level of satisfaction is a powerful indicator of whether the organization is doing everything it can to make the Big Data Analytics implementation successful, and dissatisfaction may serve as an important indicator that the organization needs to make additional changes (such as to processes or talent) to ensure that the implementation is a success.

So, how do companies adopt the organizational changes required to obtain value from big data analytics?

Chen, Law, and Yang (2009) already hinted that “project management” may have a key role for companies to get value from big data analytics. Let’s first consider the different types of big data analytics applications so we can further evaluate how they may be impacted by project management.

Event-streaming as Big Data

While looking through the lens of streaming data, Davenport, Barth, and Bean (2012) reported:

“There are several types of big data applications. The first type supports customer-facing processes to do things like identify fraud in real time or score medical patients for health risk. A second type involves continuous process monitoring to detect such things as changes in consumer sentiment or the need for service on a jet engine. Yet another type uses big data to explore network [and data] relationships like suggested friends on LinkedIn and Facebook. In all these applications, the data is not the ‘stock’ in a data warehouse but a continuous flow. This represents a substantial change from the past, when data analysts performed multiple analyses to find meaning in a fixed supply of data. Today, rather than looking at data to assess what occurred in the past, organizations need to think in terms of continuous flows and processes. ‘Streaming analytics allows you to process data during an event to improve the outcome,’ notes Tom Deutsch, program director for big data technologies and applied analytics at IBM. This capability is becoming increasingly important in fields such as health care. At Toronto’s Hospital for Sick Children, for example, machine learning algorithms are able to discover patterns that anticipate infections in premature babies before they occur. The increased volume and velocity of data in production settings means that organizations will need to develop continuous processes for gathering, analyzing and interpreting data. The insights from these efforts can be linked with production applications and processes to enable continuous processing. Although small ‘stocks’ of data located in warehouses or data marts may continue to be useful for developing and refining the analytical models used on big data, once the models have been developed, they need to process continuing data streams quickly and accurately.”[17]

Davenport, Barth, and Bean also noted that there are some key personnel differences required for successful big data implementations:

“Although there has always been a need for analytical professionals to support the organization’s analytical capabilities, the requirements for support personnel are different with big data. Because interacting with the data itself — obtaining, extracting, manipulating and structuring it — is critical to any analysis, the people who work with big data need substantial and creative IT skills. They also need to be close to products and processes within organizations, which means they need to be organized differently than analytical staff were in the past. . . Early users of big data are also rethinking their organizational structures for data scientists. Traditionally, analytical professionals were often part of internal consulting organizations advising managers or executives on internal decisions. However, in some industries . . . data scientists are part of the product development organization, developing new products and product features.”[18]

Does your company have a data scientist on every customer-facing product development team?

How about on customer interaction teams? Or marketing teams? Or finance teams? The changes that big data analytics requires to significantly accelerate revenues can be quite disruptive to a company’s organizational structure, existing processes, talent structure, and even goals. Organizations that can effectively leverage fact-based approaches to management and can rapidly iterate on new findings will be significantly more able to adapt to changes in the market and meet customer needs, but becoming fact-based requires a significant shift in mentality where processes are designed to provide insight rather than simply meet company goals. Davenport, Barth, and Bean continued:

“A key tenet of big data is that the world and the data that describe it are constantly changing, and organizations that can recognize the changes and react quickly and intelligently will have the upper hand. Whereas the most vaunted business and IT capabilities used to be stability and scale, the new advantages are based on discovery and agility — the ability to mine existing and new data sources continuously for patterns, events and opportunities. . . A further way that big data disrupts the traditional roles of business and IT is that it presents discovery and analysis as the first order of business. Next-generation IT processes and systems need to be designed for insight, not just automation. Traditional IT architecture is accustomed to having applications (or services) as “black boxes” that perform tasks without exposing internal data and procedures. But big data environments must make sense of new data, and summary reporting is not enough. This means that IT applications need to measure and report transparently on a wide variety of dimensions, including customer interactions, product usage, service actions and other dynamic measures. As big data evolves, the architecture will develop into an information ecosystem: a network of internal and external services continuously sharing information, optimizing decisions, communicating results and generating new insights for businesses.”[19]

Needing to break bad habits

Companies who wish to successfully utilize big data analytics need to make discovery and analytics their top priorities. As was discussed further by Hwang and Grant, users wanted their software interfaces to be customized exactly to their needs, not oversimplified or too complex, but customized to exactly what they needed. The same applies towards our intelligence needs. If you ask a question but are given an answer that is vague, irrelevant, or too specific, the answer was not correctly customized to your needs. If you have trouble obtaining the answer you need, if time is critical, you may feel pressured to make a decision based on your best guess so that progress can continue. If you frequently were placed in situations where you had no way to obtain answers from your data, you could develop a habit of making decisions based on your hunches and guesses. But imagine if it turned out that you were failing to meet 90% of the needs of your customers and employees and nobody even realized it? As stated by McAfee, Brynjolfsson, Davenport, Patil, and Barton (2012):

“The first question a data-driven organization asks itself is not ‘What do we think?’ but ‘What do we know?’ This requires a move away from acting solely on hunches and instinct. It also requires breaking a bad habit we’ve noticed in many organizations: pretending to be more data-driven than they actually are.”[20]

Companies who wish to obtain the significant benefits of leveraging big data analytics need to be prepared to make the company-wide changes necessary to how they make decisions, and in this big data age, companies who refuse to make those changes will inevitably be surpassed by companies who are willing to take that step.

[1] LaValle, S., Lesser, E., Shockley, R., Hopkins, M. S., & Kruschwitz, N. (2011). Big data, analytics and the path from insights to value. MIT sloan management review, 52(2), 21. Available at: http://tarjomefa.com/wp-content/uploads/2017/08/7446-English-TarjomeFa.pdf

[2] Alharthi, A., Krotov, V., & Bowman, M. (2017). Addressing barriers to big data. Business Horizons, 60(3), 285–292. Available at: https://www.sciencedirect.com/science/article/pii/S0007681317300022

[3] Grover, V., Chiang, R. H., Liang, T. P., & Zhang, D. (2018). Creating Strategic Business Value from Big Data Analytics: A Research Framework. Journal of Management Information Systems, 35(2), 388–423. Bold added. Available at: https://www.tandfonline.com/doi/abs/10.1080/07421222.2018.1451951.

[4] Ibid.

[5] Zakir, J., Seymour, T., & Berg, K. (2015). BIG DATA ANALYTICS. Issues in Information Systems, 16(2). Bold and emphasis added. Available at: http://www.iacis.org/iis/2015/2_iis_2015_81-90.pdf

[6] Wamba, S. F., Gunasekaran, A., Akter, S., Ren, S. J. F., Dubey, R., & Childe, S. J. (2017). Big data analytics and firm performance: Effects of dynamic capabilities. Journal of Business Research, 70, 356–365.

[7] Manyika, J., Chui, M., Bughin, J., Brown, B., Dobbs, R., Roxburgh, C., & Byers, A. H. (2017). Big data: the next frontier for innovation, competition, and productivity. McKinsey Global Institute report (2011), pp. 2. Bold and emphasis added. Available at: https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Digital/Our%20Insights/Big%20data%20The%20next%20frontier%20for%20innovation/MGI_big_data_exec_summary.ashx

[8] Ibid. Emphasis added.

[9] Ibid. Emphasis added.

[10] Gandomi, A., & Haider, M. (2015). Beyond the hype: Big data concepts, methods, and analytics. International Journal of Information Management, 35(2), 137–144. Bold and emphasis added. Available at: https://ac.els-cdn.com/S0268401214001066/1-s2.0-S0268401214001066-main.pdf?_tid=f570e357-2edb-4520-be12-3107c0ffb942&acdnat=1531016436_5e1890e74c2d6fc082568596bc35ba6c

[11] Loh, T. C., & Koh*, S. C. L. (2004). Critical elements for a successful enterprise resource planning implementation in small-and medium-sized enterprises. International journal of production research, 42(17), 3433–3455. Emphasis added. Available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.455.3383&rep=rep1&type=pdf

[12] Chen, C. C., Law, C. C., & Yang, S. C. (2009). Managing ERP implementation failure: a project management perspective. IEEE transactions on engineering management, 56(1), 157–170. Emphasis added. Available at: https://www.researchgate.net/profile/Charlie_Chen2/publication/224373950_Managing_ERP_Implementation_Failure_A_Project_Management_Perspective/links/564e2ed908aefe619b0faf4e/Managing-ERP-Implementation-Failure-A-Project-Management-Perspective.pdf

[13] Ehie, I. C., & Madsen, M. (2005). Identifying critical issues in enterprise resource planning (ERP) implementation. Computers in industry, 56(6), 545–557. Bold and emphasis added. Available at: http://www3.cis.gsu.edu/dtruex/courses/CIS8670/Articles/CriticalIssuesinERPImplementation-2005-CompsinIndustry.pdf

[14] Hwang, Y., & Grant, D. (2011). Understanding the influence of integration on ERP performance. Information Technology and Management, 12(3), 229–240. Bold and emphasis added. Available at: https://www.researchgate.net/profile/Yujong_Hwang/publication/226990228_Understanding_the_influence_of_integration_on_ERP_performance/links/54c5d2340cf2911c7a5661c9.pdf

[15] Ibid. Emphasis added.

[16] If this idea is still too abstract, consider a case where a manager wants to determine which feature to enhance in a new product. So, let’s say they have their team’s data scientist query their data for usage, and let’s say that the query result shows that one feature is used by only 5% of their customers and that only 15% of those customers used the feature a second time whereas a second feature is used by 40% of their customers and that 90% of those customers used the feature a second time. These findings may help the manager decide to focus resources on enhancing the second feature.

[17] Davenport, T. H., Barth, P., & Bean, R. (2012). How’big data’is different. MIT Sloan Management Review. Bold and emphasis added. Available at: https://pdfs.semanticscholar.org/eb3d/ece257cca2e8ce6eaf73fd98c1fdcbdc5522.pdf

[18] Ibid. Emphasis added.

[19] Ibid. Bold and emphasis added.

[20] McAfee, A., Brynjolfsson, E., Davenport, T. H., Patil, D. J., & Barton, D. (2012). Big data: the management revolution. Harvard business review, 90(10), 60–68. Bold added. Available at: http://tarjomefa.com/wp-content/uploads/2017/04/6539-English-TarjomeFa-1.pdf