Transformation towards a Data-driven Company

This article is a translation of my original story published in the Polish issue of the MyCompany magazine.

We live in the world, in which we describe everything using enormous volumes of data. It’s generated on an increasing amount of levels and ability to analyse it on the fly is crucial. All this so that companies and organisations can make better decisions based on real information.

As business owners or decision-makers, our news feeds bombard us with articles or presentations showing us the power of data analysis and the use of Artificial Intelligence in our operations and business processes. As you look closely, it might appear that these solutions, in the end, are exclusive to technology giants. However, the truth is that every organisation can benefit from them. The question remains: where to start?

Data Silos

New technologies often automate data processing and decisions based on that data. It requires good quality input to make sure the output will be of equally high quality (so-called “Garbage In, garbage out”). So, to begin the adventure of using data-driven technologies, let’s try to answer the question of what “good data” is. Many people confuse it with just a significant amount of information. Organisations decide to implement very large data warehouses, spending millions of dollars on systems that collect and process inputs from even hundreds of data sources. However, these tools often do not provide one fundamental thing — easy and shared access to collected information. Shared, that is, providing an equally interpretable understanding of the given situation.

Your operations are often described by hundreds of metrics

Every organisation that begins the adventure with data analysis faces the problem of data silos. Often, managers believe that their teams make decisions based on facts, like business events from the past recorded in data warehouses. Moreover, there are whole lots of kinds of business facts happening in our systems — describing the productions of goods related to the business, logistics, financial data, HR, online business activities and many more.

Think how quickly you can receive information about revenues generated by a particular segment of your clients in a given, non-standard period?

Many times, we may face a situation where teams which have data, will feel masters and rulers of their collected information. They will protect access to their resources. For example, I’m sure you faced situations where you had to ask for permission to access specific reports (and unfortunately not for security reasons!). To give you another perspective, think how quickly you can receive information about revenues generated by a particular segment of your clients in a given, non-standard period (i.e., in the vicinity of a marketing campaign)? In most companies, you will not be able to get it immediately. Furthermore, it can be particularly dangerous in situations in which the organisation’s remuneration model supports unhealthy competition between teams. Then the disaster of an illusion of ability to make data-driven decisions is at your fingertips.

Our activities are often described by hundreds, if not thousands of business indicators. In the illustrated situation we give the data that creates these metrics into hands of closed-communication teams (or at least separated ones). We end up creating a culture in which everyone can see only their part of the business. Moreover, that means that one can make decisions based on a partial understanding of product knowledge or business processes.

Transformation

To be able to successfully take actions based on the analysis of other activities and using Artificial Intelligence, the organisation should undergo a transformation towards the so-called Data-driven Organisation. What is essential at the early stage is creating a strong vision of the product, appropriate organisational culture and, if it was not done so far, to build the so-called domain knowledge. It will also be essential to involve the proper experts to ensure a smooth process.

Satya Nadella, src: Wikicommons

However, I can’t emphasise enough, that the critical element is the culture. Microsoft’s CEO, Satya Nadella, said: “A data culture isn’t just about deploying technology alone, it’s about changing culture so that every organisation, every team and every individual is empowered to do great things because of the data at their fingertips” (Official Microsoft Blog, “A data culture for everyone”).

The key to success is the ability to respond to changes in the world around us quickly. Innovation is a vital aspect of competitiveness. Thanks to the analysis of data and use of machine learning, we can analyse existing situations, describe the complex business processes in full, and finally draw conclusions for the future. We open the possibility of creating a whole new spectrum of products and services, which are more extensive but yet more straightforward to use. We can experiment ahead of our competition. From our client’s point of view, we can create a solution that they will feel great with, and our support team will know about their needs before contacting us.

Understanding the business

Often the IT department raises terrible associations. Think about it. You may see dark rooms filled with eccentric people communicating using some cipher. As IT, we usually work hard for such a picture. For us what matters is new shiny technologies and tools supported by use of a steadily growing number of more and more complex programming patterns.

Coming back to the roots of our profession is necessary. We need to support each company member at every step.

In a Data-driven Organisation, the technology team can’t be a source of problems but must help the business solve those, sharing their expertise. Coming back to the roots of our profession is necessary. Listen to the problems of other teams so they can be understood in full and solved using the correct approach, not to deepen or ignore them. As IT department, we need to support each company member at every step.

Hence, the technical team must be open for open communication with the business and understanding its needs. The willingness to learn and strive to achieve an ambitious vision of the product is also essential. However, from a technical point of view, we must find a solution that will allow us to build even the most complex products in a light and agile way.

Our team during an Event Storming session

Tools

Many tools can help us here. An example that we use in practice is the migration of the product to an architecture based on business events (the so-called Event-Driven Architecture). It promotes an excellent understanding of processes, like systems or users activities in our software, as well as the creation of events in its relevant parts (modules) and their recording. As a result, they correspond to the actual company image of the product or the process. Furthermore, the cost of maintenance and development significantly decreases.

Event Storming is the right tool to do the job of exploring your business processes in a light and an agile way.

Exploring and describing business processes precisely in and around the product is a very complicated task. At the same time, it requires a light and an agile approach. Event Storming is the right tool to do the job. People from different departments and developers working on any business process meet to model it together. Thanks to this, they can describe existing modules (to find gaps and potential problems) or design new ones. During such sessions, the team of programmers also learns the business domain language, which they will use later in modeling the solution. For implementation, Domain-Driven Design comes in handy. It’s an approach where system components faithfully reflect the reality they represent. It also allows you to involve experts from your company who are not necessarily familiar with the design and implementation of IT systems but have strong business knowledge. As a result, we receive a product that creates real information about itself based on business events from the very beginning.

Mariusz Gil, Source Ministry

“Understanding of business requirements and the space in which the system will operate in the future is the foundation of any IT project. The team needs tools that support communication with the broadly understood “business,” which results in different perspectives on the problem described in the shared business domain language. Event Storming for us is a platform for conversations within the organisation, allowing to identify, model and streamline business processes, as well as to create artefacts needed by the team during the implementation phase. During Event Storming session sharing knowledge takes place on every level, and seemingly irrelevant pieces of information can become a business game-changer, which at the end of the day we can’t be valued enough.” — says Mariusz Gil from Source Ministry.

Such approach provides an excellent mean to describe the product or business processes in the organisation faithfully. Depending on the scale of operations and the number of observed activities, such events can come in massive amounts — we are talking about hundreds of millions a day in our case. On this level technologies for collecting, processing, and analysis of collected information, together with machine learning, will come to help. In our case, especially in connection with legal restrictions, we decided to invest in a data warehouse based on Big Data technology, using a dedicated Hadoop cluster. To perform operations on the data we collect, we use the following tools: Apache Spark, for more extensive batch processes, and Apache Flink, for analysis of the vast number of events in real time.

Cloud technologies can also be used to address these needs. The leading cloud players, like Amazon Web Services, Google Cloud Platform, and Microsoft Azure, offer comprehensive solutions in this area. They allow the creation of appropriate software services, collection and processing (also in real time) of events, their recording and both simple, and extensive tools to implement AI based features.

Correct implementation requires above all a good definition of a real business problem to solve, valuable and understandable data, close cooperation of business and analysts.
Adam Kawa, GetInData

“Although Big Data technology, both open-source and cloud-based, allow today to implement so many business cases, one must be aware that the infrastructure and tools are only a means to an end, not an end in itself. We have seen many companies that have invested millions of dollars in innovative Big Data warehouses, but so far, these have not brought these companies a cent. In their nature, Big Data projects are more complicated, more time-consuming and more costly than “regular” IT projects. That’s why their correct implementation is so essential, preferably following agile methodologies. Also, this requires above all a good definition of a real business problem to solve, valuable and understandable data, close cooperation of business, analysts and administrators (preferably within one team). So only at the end, the choose technologies we find comfortable and reliable in our case.” — says Adam Kawa, CEO, and co-founder of GetInData.

Let your data work for you

Many companies are at the beginning of the journey in which they will benefit from the data they collect. How to start? From the very beginning, the best is to focus our work on the culture of the organisation. That’s a critical moment — understand where you are and how you operate as the business. Furthermore, work with your teams on answers to such questions as: What drives my teams? How do they make their decisions? What do they need to make the right decisions based on the data?

In a Data-driven Organisation, everyone becomes an independent analyst and can draw information about the activities and processes within your business.

The significant strength of the Data-driven Organisation is the change in who and how can access the data. Everyone becomes an independent analyst and can draw information about the activities and processes within the organisation. Having it in mind, think if you have a team that is open and ready for it? Will they be willing and able to make the right decisions? These are probably even more important and crucial questions.

The second step is to develop a common language. Making sure that the same things in the entire organisation mean precisely the same. Let us consider how many inaccuracies or conflicts could arise when we used many different ways of describing the same fact or situation, and thus various people thought about something in a completely different way.

If you’re interested in the transformation towards a Data-driven Organisation and use of AI in your business, I definitely recommend listening to GCP podcast with Cassie Kozyrkov, Chief Decision Scientist at Google.

It is also worth remembering that it is impossible to buy a quick recipe for transforming your organisation. There is no such thing as an all-around silver bullet, if only because of multitude and diversity of solutions, tools, and processes. In preparing the plan appropriate for your business, it will be helpful to carry out audits and engage professionals to help.

It is also worth remembering that it is impossible to buy a quick recipe for transforming your organisation.

Your company can also change thanks to improving the culture of the organisation and the new possibilities of IT technologies. If you care about the actual understanding of business processes, cost reduction or becoming a pioneer in your field, use your data to help your organisation to achieve it. Make sure that it will be used in the right way by every person in the organisation. Possession and processing the right kinds of data will make the use of machine learning possible, and it will help to optimise processes bringing your business to a different level. Welcome to the 21st century, in the era of data, where the new gold is the opportunity to collect and use information.