Challenges Organizations Face in the Journey of Becoming Data-Driven
Artificial Intelligence, Data Science and Machine Learning are the buzzwords of the current industry. Many enterprises have become data-driven by taking appropriate strategies and using continuous effort. Companies are spending millions of dollars in this pursuit. According to Big Data and AI Executive Survey 2020 conducted by NewVantage Partners, over two-thirds of the organizations that were taken part in the survey are Investing $50MM or more in the journey of becoming a data-driven organization.
Organizations are setting up data science division, bringing required technologies, onboarding data scientists, data engineers, visualization experts, and data managers to support the data-driven initiative. Many seminars, conferences, webinars, hackathons, and boot camps are organized over the year across the globe to support and encourage the data science community.
But there is another side of the story. Some aspects of data-driven initiatives are not coming under the light of the discussion. The analytic industry is largely driven by hypes and sensations. The ground reality has not been delved into by most stakeholders. About three-quarters of the participants of the survey conducted by NewVantage Partners reported that implementing big data and AI in business is a challenge.
Only 15% of the respondents have deployed AI into production. Only 51.9% of the participant organizations are speeding up their rate of investment in a data-driven journey in 2020 in contrast to 91.6% of the participants in 2019. So, it is prime time to put the question in front of decision-makers and domain experts. Why the journey of becoming data-driven faces challenges? In this post, I shall point out some important aspects of the fundamental approach of implementing data-driven culture in organizations.
- Traditional and large organizations are facing a hard time incorporating data-driven culture
Startups and new companies are adapting to data-driven business decisions making process because of their dynamism. But an enormous part of the industry is not sure about how to adopt a data-driven approach in their existing organizational culture. This is more evident for large and relatively old organizations. Many productions and manufacturing companies are familiar with traditional business and operational environment. Even if they are setting up a data science team and infrastructure with a significant amount of investment, they are unsure about the way of the utilization of their resources. For some organizations, even if the business teams are aware of the problems but they are not confident about the capability of the data-driven approach and the data science team in solving the problem. They feel doubtful about the end-result of the initiative because of the business acumen they have and the business knowledge data science team lacks. They are unaware of the value a data-driven approach can bring to the business.
2. lack the leadership for a data-driven approach
To make the processes of an organization data-driven, it needs a prominent leader who can change the organizational culture. The only initiation of the process is not enough. Somebody should lead the way and keep spreading a data-driven approach to every corner of the organization. He needs to push data science solutions to every business operation. He should keep motivating the business owners about the bright side of the data-driven decision making and its far-reaching effect. But the current industry is suffering from a lack of proper leadership in data science initiatives. It has a long term negative effect on organizational growth and on the future of the analytics industry.
3. Lack of leadership in the data science divisions
Data science is a new domain. Most data science professionals have few years of experience. An enormous part of the population in this domain is coming from Information Technology and software development background. A small group of professionals is coming from an analytical background. There is a scarcity of professional who has adequate expertise and experience of guiding and leading data science teams. A data science lead has to understand the business problem, convert a business problem into a data science use case, analyze the feasibility of the use case, guide the team towards the solution, draw actionable insight from the solution and if required he will have to build a product out of the solution and deploy it for further use. So often data science teams of organizations suffer from a lack of proper guidance and leadership. It leads to failure of the organization to become data-driven.
4. The gap of communication between the business executives and data scientists
Data scientists are inclined towards technology, mathematics, statistics, and algorithms. What they lack is business acumen because of their limited experience in the domain. The business executives are more interested in their business goals and less aware of the quantitative and technological aspects. This creates a barrier of communication between data scientists and business personnel. Business executives face a hard time making the data scientists understand their concerns about business problems. Data scientists face a challenge to communicate the technological complexity and methodological novelty of their approaches. As a result, the business teams become dissatisfied with the outcome of the data science projects and become less interested to consume the result.
5. The inability of data scientists to draw meaningful and actionable insights from data science projects
It takes veteran and trained eyes to identify and infer wonderful insights from a data science exploration. A wonderful insight is something that is actionable and which brings value to the business. There is a scarcity of professionals with such trained eyes in the analytic industry. Despite putting a lot of effort into solving a business problem through an analytical approach, data scientists sometimes cannot draw insightful conclusions that business can consume. It hurts the success of the whole data science initiative of an organization.
6. The inability of business executives to infer the insights and apply in business
Even if the data science team provides some meaningful actionable insights, the business executives sometimes cannot consume it. It affects the alignment of the data-driven insights with the business process. They cannot understand the real meaning of the insights and their business usage. Executives become unsure about the action to be taken that can benefit the business. So, the insights become useless to them.
7. Inability to identify problems that a data-driven approach can solve
Being a part of the business, executives sometimes cannot see the system from a bird’s-eye view. They think they have used all sorts of effort and all kinds of remedial measures to achieve the business target. The problems in the business process that are needed to be solved by using data science remain unidentified. Lack of efficiency of the process becomes visible only in terms of cost or sale or any other metric.
8. Organizations lack the infrastructure to support a data-driven culture
In earlier days most organizations used to store very little information generated from the business, for example, financial records, legal documents, etc. A hand full of companies was interested in storing customer-related information or customer feedback and other business processes related information. Even if organizations have started storing such data, but those are largely decentralized and incomplete in most cases. There are several reasons for being this data incomplete. As individuals involved in business processes are more interested in achieving their targets in terms of sales or leads or profits than collecting and storing the data generated in the process. As a result, the data collection at the source is largely ignored. It is a major blockage for data science initiatives as data is the fuel of the data-driven approach.
9. A part of the business teams feel threatened of losing importance
Every organization has its existing teams for taking marketing, sales, operations, procurement-related decisions. They decide based on their knowledge, intuition, and hunch. When an organization declares to incorporate a data-driven approach in business decision making, the existing team feel threatened of losing their importance. They feel threatened because if the data science team gathers the required business acumen and draws some meaningful insight using a data-driven approach, then the existing team will lose its position. They cannot realize the fact that the purpose of data science is to complement the existing business teams with a data-driven approach.
10. It takes time to implement a data-driven culture in an organization
It takes time to bring the cultural changes in an organization and nourish a data-driven approach. Because people take time to change their mindset. The data science approach requires time for exploration. It takes months for a project to become useful for business. But business executives expect an immediate outcome. As a result, they become uninterested in using the project outcome.
Data-driven decision making is a fresh approach. It requires planning to adopt such an advanced approach in an organization. To spread the essence of a data-driven approach in decision making; organizations have to accept data science as a religion. The idea should be flown from the top to the lowest level of management. Each individual of the system should understand the need for a data-driven approach to some extent. This initiative will not be successful if it is integrated as a separate and optional service. To leverage the full potential of data science, organizations have to take a holistic approach. The organizations should continue the journey of becoming a data-driven ahead. The organizations have to focus more on human than technology to make this effort a grand success. Because a change in human behavior is much more difficult than a technological change.
