The Road to a Data-Driven Organization: Navigating Data Dilemmas

Many organizations today aspire to become data-driven, but only a few manage to achieve this goal within a reasonable budget and timeframe. The journey towards a data-driven organization is often a vital component of a broader digital transformation, involving a gradual shift in technology and processes to optimize operations. However, change is rarely straightforward, resulting in slower and costlier progress than expected.

The journey towards data-driven organizational growth is marked by numerous data dilemmas. Navigating these challenges effectively requires a well-crafted navigation strategy to reach your destination within a reasonable timeframe and budget [Image: Author].

In this article, I will explore the data-driven paradox: while data holds the potential for growth, it also presents many data dilemmas in achieving that growth. Drawing on existing literature, we will delve into the gains and pains of embracing a data-driven approach, emphasizing the importance of a good data strategy to communicate and resolve the various data dilemmas along the way. But first, let’s clarify what we mean by a ‘data-driven organization.’

The Data Driven Organization

It’s mentioned in almost any (consulting) data white paper, presentation or article: organizations want to become data driven. But what do we mean with a ‘data driven organization’?

For me, a data-driven organization operates on two levels:

1. Data-Driven Decision Making: Here, organizations leverage facts, metrics, and data to guide strategic business decisions aligned with their goals and objectives. This involves using data artifacts such as Key Performance Indicators (KPIs), reports, and dashboards to enhance decision-making processes.
2. Data-Driven Process Optimization: In this stage, businesses use data to improve their existing processes. This could range from simple data-driven dashboards for continuous monitoring and improvement to more advanced applications like employing Artificial Intelligence (AI) to automate key processes, such as customer service through AI chatbots.

Both these levels require a combination of manual and automated actions. While manual interventions are necessary at first, mature data-driven organizations rely more on automated technologies to streamline operations and maximize efficiency. For example, in a typical reporting context, technology provides an actualized dashboard which is then used by people to make the right decision. In a more mature data driven organization, an AI tool could automatically classify potential churners and send them a marketing email without any human intervention.

Data driven organizations have to find the balance between human actions and automated actions [Image: Author].

The data driven paradox states that data leads to growth but that it’s challenging to grow with data. In this next two sections, I give a literature overview of the pains and gains of working with data.

Data will Lead to Growth

Over the past decade, numerous companies have invested in data projects, embarking on their journey towards becoming data-driven organizations. Along this path, they have witnessed the advantages and returns on investment stemming from data-driven decision-making and process optimization. Even amidst the current challenging economic climate, organizations persist in their investments to modernize data platforms, underscoring their belief in data as a vital business asset.

Recent research and publications show that data and AI have the potential to fuel organizational growth in the next years [Image: Author].

With AI dominating the news today, many companies are eager to gain competitive advantages by investing in this transformative technology. Recent research and publications further reinforce the notion that the synergy between data and AI holds the potential to drive additional organizational growth in the upcoming years:

1. PwC found that retailers investing in customer data and implementing data use cases can expect a 3%-5% increase in contribution margin after accounting for initial investments and acquisition costs. Data-driven initiatives like customer engagement, marketing excellence, and operational improvements contribute to this growth.
2. McKinsey predicts that by 2025, smart workflows and seamless human-machine interactions will be as common as a corporate balance sheet. Data utilization will become standard practice, optimizing various aspects of work and leading to increased productivity.
3. AI technologies, including generative AI, are expected to provide substantial returns. McKinsey estimates a 0.2–3.3% increase in productivity from 2023 to 2040 due to automation enabled by AI technologies.
4. Experimental evidence suggests that generative AI, like ChatGPT, can significantly enhance productivity for college-educated professionals performing mid-level professional writing tasks. This AI tool improves output quality and reduces task completion time across all ability levels, leading to increased overall productivity.
5. According to IBM’s Generative AI — State of the Market report, executives expect returns from generative AI to exceed 10% by 2025. This number is derived from the baseline AI capabilities they’ve developed over the past several years.

Data Dilemmas: Balancing Growth and Challenges

Data-driven growth does not come without its share of challenges and dilemmas. Organizations aspiring to be data-driven need to address several hurdles to harness the full potential of their data. These data dilemmas, based on research reports and articles published by reputable sources (McKinsey, The Wall Street Journal, MIT Sloan, BARC, MonteCarlo Data, PRNewsWire, PWC, and UpTempo), highlight the key areas that organizations need to tackle:

1. Sustainable Growth Requires Robust Data Management: Building a solid data management practice is crucial, even with the advent of generative AI. Companies must ensure proper data structuring, governance, and cleansing to harness data effectively. For instance, the Dutch Bank Rabobank halted all AI initiatives until their data management practice will be appropriately organized.
2. Overcoming Cultural Barriers: Many organizations rely on a mix of data and gut feeling for decision-making, making it challenging to create a true data-driven culture. Obstacles like a lack of resources, knowledge, clear roles, responsibilities, and communication need to be addressed for successful transformation.
3. Data Quality and Ownership: Data ownership often lies with data engineers, who may lack domain-specific knowledge, hindering data quality improvements. High-quality data is essential for meaningful growth with data.
4. Cultural Shifts: Cultural issues, such as resistance to change and business transformation, can slow down the organizational transformation necessary for becoming data-driven.
5. Cost and ROI Concerns: Investing in new technologies like AI can be challenging, as it’s often unclear when the investments will pay off. Companies struggle to quantify ROI from AI investments, making it difficult to focus on the right business problems that generate returns.
6. Lack of Confidence in Data: Inability to calculate ROI of marketing campaigns often stems from a lack of confidence in underlying marketing data and inefficient processes.

Organizations aspiring to be data-driven need to address several hurdles to harness the full potential of their data [Image: Author].

Besides these non-technical challenges, companies also need to tackle various technology-related hurdles. The following non-exhaustive list highlights some technological challenges based on publications from Gartner, Algorithmia and Flexera:

1. Lack of In-House Expertise: A significant number of finance AI projects are expected to be delayed or cancelled by 2024 due to the lack of in-house technical knowledge and the substantial upfront cost of building custom data platforms and infrastructure.
2. Complexity in Deploying AI Models to Production: Deploying machine learning models to production is increasingly time-consuming, with a majority of organizations taking a month or longer. Data scientists often spend more than 50% of their time on deployment, and this challenge becomes more pronounced with scale.
3. Managing Cloud Spend: The shift towards cloud technology and services in modern data platforms has brought about a new challenge—managing cloud expenses effectively while handling large volumes of data.

Enterprise Data Strategy

Embracing a data-driven approach enables companies to navigate through data dilemmas and unleash the true potential of their data assets. An effective enterprise data strategy plays a crucial role in overcoming these dilemmas by outlining guiding principles and goals to guide decision-making.

Some key principles that can drive an enterprise data strategy are as follows:

1. Buy Before Build: Opt for existing tools or services instead of building everything in-house, leveraging established solutions to expedite progress.
2. Citizen Data Engineering: Empower every business unit to create its own data pipelines, enabling the transformation of data into valuable insights.
3. One Version of the Truth: Ensure consistency by having a single implementation of management Key Performance Indicators (KPIs) across the underlying data platforms.
4. Centralized Data Platform Management: Streamline changes and maintenance of the organization’s data platform by entrusting responsibility to a single dedicated team.
5. Privacy Before Efficiency: Prioritize privacy and security when considering external services like ChatGPT to handle sensitive internal documents.
6. Usable Data for Management Decisions: Data should be readily available in a usable format to support a wide range of management decisions.

However, a data strategy alone might not prevent potential disconnects between strategic goals and operational choices. For example, if a data engineering team decides to use code-driven data pipelines (e.g. Scala), it may conflict with the organization’s ambition to promote citizen data engineering. Ensuring that such disconnects are recognized and addressed is crucial to aligning data initiatives with the overall strategic direction.

Even with a well-defined data strategy in place, there is no assurance that the final data landscape will align with the outlined principles of the strategy [Image: Author].

This disconnect between strategic and operational data views is evident in the 2023 BARC Data Culture Survey. Executives, operational staff, and data analytics leaders often have contrasting perspectives on the progress of key initiatives, particularly in areas like data literacy and data communication. To implement a data strategy effectively, it becomes imperative to establish clear and structured communication channels for data projects, programs, and architecture.

Foster Communication

By bridging the gap between vision and execution, organizations can cultivate a culture where data is harnessed effectively, leading to enhanced decision-making and sustained growth. A cohesive data strategy, coupled with effective communication, becomes the pathway to transforming into a truly data-driven organization, equipped to thrive in an increasingly data-rich landscape.

Principles

To foster clear communication in digital data transformation programs, it is crucial to make implicit choices explicit. This can be achieved by developing a concise one-pager for each key decision, considering the following points:

1. Clearly State the Decision: Articulate the decision that needs to be made in a straightforward manner, ensuring everyone involved understands its significance.
2. Build a Decision Framework: Provide a decision framework that outlines the advantages and disadvantages of each potential solution, helping stakeholders grasp the implications of their choices.
3. Emphasize Visual Communication: Utilize visual aids, such as sketches or diagrams, to illustrate the decision-making process, making complex information more accessible and engaging.
4. Engage a Broad Audience: Involve a diverse set of stakeholders, including executives, operational staff, and data analytics leaders, to ensure comprehensive perspectives are considered during decision-making.
5. Opt for Vendor Neutral Solutions: Prioritize vendor-neutral solution types that possess longevity in the ever-evolving technological data landscape, enabling adaptability to future changes.

By employing a concise one-pager for each key decision, organizations can foster meaningful discussions and enable informed decision-making throughout their data transformation journey [Image: Author].

By adopting these practices, organizations can facilitate meaningful discussions and facilitate informed decision-making during their data transformation journey. Transparent and well-communicated choices contribute to building a cohesive data-driven culture.

The next section applies these principles to facilitate the selection of the right data pipeline technology.

Example: Selecting the Right Data Pipeline Technology

Decision to be made
Data pipelines serve as essential components for moving towards a data-driven organization, extracting, loading, and transforming data across systems to provide the right data for any type of data product at the right moment. Which type of data pipelining tool need to be selected to address your needs?

Framework
Four criteria can help in selecting the right type of data pipeline technology:

1. Learning Curve: How difficult is the pipelining technology to learn?
2. People: What skill level is required to work with the pipelining technology? Can a business analyst with limited training manage it, or is it better suited for highly experienced data engineers?
3. Power: What are the capabilities of the pipelining technology? Can it handle Extract, Transform, and/or Load operations?
4. Scalability: Can the pipeline easily scale from one to multiple pipelines? Is it capable of processing high volumes of data?

A visual sketch makes the decision process accessible for multiple stakeholders [Image: Author].

The accompanying sketch illustrates how four common types of data pipelines can be applied to these four criteria:

1. No-code: User-friendly tools that facilitate extract-load data movement and basic transformations. Examples include Fivetran, Airbyte, and CDC-tools like Qlik Replicate. These tools are often set up by data engineers but are usable for a wide range of data profiles.
2. Low-code: Visual ETL tools that allow the creation of pipelines through a click-and-drag interface. Examples include Matillion and Azure Data Factory, offering more sophisticated features than no-code solutions.
3. SQL: With the rise of DBT Labs’s DBT tool, SQL has become a powerful language for data pipelining. It is easy to learn, yet potent in exploiting the full capabilities of the underlying computing platform. Setup and maintenance are typically done by data engineers, while actual SQLs can be written by a broader range of data and business profiles.
4. Code: Code-driven pipelines, often using Python or Scala on Apache Spark, are powerful and capable of handling large data volumes. However, maintaining these pipelines requires highly skilled data engineers.

Recommendation
A combination of no-code and SQL, as it allows quick and easy integration of new data alongside the powerful yet relatively easy-to-learn SQL language. This approach ensures the option to allow citizen data engineering in the (near) future.

Conclusion

In conclusion, data’s potential to drive growth is undeniable, yet transforming into a data-driven organization comes with its fair share of dilemmas. To succeed in this endeavor, organizations must tackle both non-technical and technical hurdles, harnessing the true potential of their data assets. An indispensable element of a successful data strategy is establishing a structured approach to democratize decision-making throughout the data-driven transformation program.

By adopting the right practices, organizations can foster a culture of open and meaningful discussions, empowering stakeholders to make informed decisions during their data transformation journey. Transparent and well-communicated choices serve as the building blocks for cultivating a cohesive data-driven culture, where data is embraced as a strategic asset and leveraged to drive innovation and growth.

Questions? Feedback? Connect with me on LinkedIn or contact me directly at Jan@Sievax.be!

This article is proudly brought to you by Sievax, the consulting firm dedicated to guiding you towards data excellence. Interested in learning more? Visit our website! We offer a Data Strategy Masterclass that provides a deeper understanding of the world of data strategy.