Big Data is not a matter of IT

Why do Big Data Analytics projects fail? Is managing Big Data projects the same as managing IT projects? Which department is the most suitable to implement Big Data projects? Can you be in charge of Big Data projects without mastering the technical skills involved?

The title of this article may scare some, but its only objective is to rethink the focus of Big Data Analytics projects, which by default tend to fall into the hands of the IT department due to its technological component. Big Data is not a matter of IT, at least not exclusively.

We must acknowledge that most of us are still in diapers when it comes to the subject of Big Data Analytics, although data analysis has always been there and there’s a lot already done. The appearance of companies such as Google and Amazon has provided us with the example, the bases and the necessary tools to be able to analyze and extract conclusions from more significant amounts of data in a more efficient way. Basically, by revisiting science and statistics with an added component of computing capabilities and low costs.

Almost 20 years after their first steps, the question is: what are we doing wrong in the adoption of Big Data Analytics in our organizations? CIO’s article on PwC’s and Iron Mountain report “How organizations can unlock value and insight from the information they hold”, reveals that most companies are failing in their Big Data projects.

The data is frightening. Only 4% of companies are implementing Big Data Analytics projects with real success, while 43% say they get small benefits from it and 23% do not get a counterpart. Almost more surprising is that the report shows that three-quarters of the organizations in the study lacked the skills and necessary technology for its implementation, three out of four haven’t yet hired a data analyst and, from the ones that have, only a quarter are using it competently.

So, why do Big Data projects fail? The expert in Big Data, Bernard Marr, explains in his article “Where big data projects fail?” that more than 50% of projects in Big Data Analytics will obtain results below their expectations due to the lack of adequate planning. After his extensive experience in the implementation of this type of projects, he comments the most common mistakes:

1. Starting with unclear business objectives. We must start asking the “why” and then move on to the “how”.
2. Not having made a good business case. Before thinking about technology, we must be clear on which are our business needs.
3. Failures in the project management. Lack of experience, not knowing the technological implications or poor management of change can stifle any good project.
4. Poor communication of the project and the results. We must always think about who the information will adress to and how we can make sure that the message arrives to the user.
5. Not having the skills needed for the project. Currently, it is difficult to hire good data analysts, since they are expensive and scarce.

The history repeats itself. The old problem of IT vs Business. Since the beginning of computing, the adoption of new technologies by companies has given much to talk about. So much so that even today impressive figures are being considered, as only 29% of IT projects are considered successful according to the Chaos Report 2015 of the Standish Group. On the other hand, a report of McKinsey in collaboration with the University of Oxford “Delivering large scale projects on time, on budget and on value” ensures that large IT projects deviate 45% of the time on average in budget and 7% of the cases in time, while in 56% of the cases they provide a lower value than initially estimated.

These figures suggest that there is a fundamental problem with the implementation of the projects: we want to tackle very complex projects without understanding its needs, resources and implications. We don’t have the necessary maturity to address the analysis of data from a strategic point of view since we pose them as technical problems and they actually have more science to them for their iterative discovery process.

It’s a fact that companies are investing, and lots, in capturing data, but they surrender quickly when it comes to using it and extracting value from it. In many occasions we start from two wrong views:

· The data is assumed as a by-product of the activity, instead of being valued as a strategic asset to the company.

· The responsibility rests with IT and data architects, rather than being a cross-cutting resource of the company.

We are facing a fundamental paradigm shift: data is not a problem of IT, but a precious asset of the company beyond the technological aspect.

The reference in studies and technology trends, Gartner, in his publication “How to prevent big data analytics failures”, tells us the need to develop viable strategies to add value to the business from Big Data projects. Gartner’s approach starts with talent and resources through mapping and the acquisition or development of the necessary skills and specialization. Once the strategy and priorities regarding skills are defined, then we can go with guarantees to the Big Data analysis. A critical success factor for its implementation is the ability of the organization to build, grow and sustain a multidisciplinary team to address the identified business problems.

At this point, management has a a critical role. You must stop thinking like an internal client and enable action by leading data analysis projects in their respective areas. Leaders need to focus their efforts on how to analyze and digest these data for internal use through a strategy, identifying data sources, understanding the importance of analyzing information for the rest of departments and creating a structured plan. The problem of management taking control is that a large percentage of executives don’t know all of the information available or how data flows through the company and how it could help their teams. The worst thing is that when someone else takes over control and the moment of truth comes, they do not trust this person entirely.

This is definitely not about blaming someone guilty and starting a war between departments to see who gets the glory, we have to think who are the most appropriate people to carry out big data analytics projects and start laying the foundations of a solid strategy for our data. At the same time, it’s not about hiring talent and waiting for them to solve the problems of our business. Our first step in a Big Data Analytics project is not to hire a data scientist or create an analytics department. Each company is a world of its own. We start from very different experiences, resources and situations. It’s a reality in constant change, that’s why we must stop to analyze what strategy is the most appropriate for each situation and precise moment.

Big Data Analytics has many successful cases in technological startup contexts, for many reasons, but if I had to highlight two, it would be: data-driven talent and culture. This, in part, coincides with one of the conclusions of the CIO article, which says that the common traits among the companies that make up 4% of successes are a consistent model of government, progressive complexity and culture based on data.

In my opinion, there are three options: endow IT with a business vision, provide a technological view to the business perspective, or the most coherent and logic one, to bypass the usual labels and create dynamic mixed teams that are capable of facing major challenges at a global level. In whatever format. All this will depend on the organization, its resources and needs. What seems clear, is that Big Data Analytics projects have more to them as startups- at least in the first instance- as that of technology management by itself. Tasks such as the creation of multidisciplinary teams, a clear strategy definition, change management, project management and knowledge management are the ‘bread and butter’ of a new company with high growth potential. If we stop to think, data grows exponentially, and in many cases, its analysis and use constitute a competitive advantage, especially when they are made in verticals with particular purposes. Is it time we start creating real startups within our companies? Are we prepared to invest in projects of high potential and risk in our businesses?

We live in a generation that has managed to build bridges between management and technology through the science of data analysis and its tools; now it’s up to us to be brave and cross over to obtain results.

If you feel identified with any of the approaches above, do not hesitate to leave a comment. How would you manage a Big Data Analytics project? What do you think would be best for your organization? How do you see your future in the Big Data ecosystem?

This story is published in The Startup, Medium’s largest entrepreneurship publication followed by 286,184+ people.

Subscribe to receive our top stories here.