Building Bridges: Combining Data Engineering and Analytics Translator Skills
TL;DR Business and IT do not always speak the same language, which can lead to missed opportunities. Engineering a bridge by translating the business goals to the organisational data possibilities will maximise the business value of the data. This blog gives you insights in how the combination of the roles Data Engineer and Analytics Translator creates great data opportunities for organisations.
Nowadays, every organisation is collecting data. To actually make these data valuable, it needs to be processed effectively and translated into actionable insights. Therefore, roles like Data Engineers and Analytics Translators have become indispensable.
In this article, I will share how I combine the skills of these two roles within the project I am currently working on at a housing corporation. Even though my job title - Data Engineer - has the word “engineer” in it, my skills are particularly focused on the data part. Which is good, because I was not hired to design or construct a building, but to set up the automation of the data flows. But before I dive into it, let me explain what the skills and responsibilities of a Data Engineer and an Analytics Translator are.
Data Engineer and Analytics Translator
As a data engineer I am responsible for preparing raw data into usable data for interpretation, for example for data scientists or data analysts. My skills in this role are mainly focused on creating and maintaining a data infrastructure for collecting, managing, transforming, and accessing data.
However, those skills are not the only skills necessary in my current role.
Being able to create a bridge between business and IT is highly important if it comes down to maximising business value from data.
This is what makes this role so great for me. As an analytics translator I gain domain knowledge and knowledge of the goals and strategy of the organisation. I do not only want to be focussed on developing the data infrastructure, but I am also curious about what data I am working with and what value it can add to the organisation. I think that combining the insights from both a data engineering perspective, and a business perspective will lead to recognising business opportunities and maximising value from the organisation’s data.
Current Project
I am currently working at the strategy department of an organisation where all data reports are based on raw Excel exports which are manually integrated using specific formulas.
Every time a report needs refreshing, the same methods are applied to gather data manually. Employees spend weeks of their time exporting and organising these Excel sheets, and checking if the data are correct and complete, even though this is not the job they were hired for.
At the same time, a data lake is being developed and maintained in one of the IT departments. Every day new data is loaded, processed and integrated for descriptive and even predictive analytic purposes.
Lack of communication between these two departments leads to an enormous knowledge gap and missed opportunities.
It is my job to create a bridge between these two departments, this benefits creating sustainable data flows and building reports based on these flows to provide insights on a daily basis without much effort.
In the end it will save a lot of time for the employees. Time they can use to do the job they were actually hired for.
Insights in Business Goals
How can I make sure that I will engineer this data bridge between the two departments?
Firstly, for me, it is important to gain insight in the business’ goals and their strategy to achieve these goals.
Secondly, it is important to have an overview of all the data projects within the department and their business value. In this step of the process the main question is, “why?”.
Asking this question repeatedly will give a better understanding of the core (data) problems the department is dealing with.
For me the best way to keep an overview of all the gathered information is by visualising all the different processes within the organisation on a (virtual) whiteboard. This helps me structure my own thoughts, but also verifying the gained insights of the interviews and communicating them to different stakeholders.
Insights in the Current Data Status
Having gained knowledge on the current state of the department. My next step is to have a clear view on what data is available, how these data are processed and what the quality is of the data.
I will make a mapping of the available and usable data on the one side and which data is needed for creating business value on the other side. If the necessary data is not (yet) collected, I will make an estimation on how much effort it will take to extract, load and transform these data to make them of use.
Task Prioritisation
At this point I will have a clear view on the business goals and the current state of the data availability and usability. Next, I will formulate tasks on how to reach the opportunities the data has to offer.
To find out which of these tasks are maximising the benefits the data has to offer, I use the impact versus effort matrix in combination with the how-now-wow method. These are helpful tools for prioritising tasks. In this matrix, impact is plotted on the vertical axis and effort on the horizontal axis. The how-now-wow method is related to this matrix and will provide insights into the innovativeness of the idea’s.
Plotting all tasks on the four different quadrants will help in the decision-making process and will create insight in the priority of the tasks.
Where to start first? The first quadrant to look into in this matrix is the “low-hanging fruit” category, which is the upper right quadrant in the matrix. Tasks in this quadrant, also called the the “Wow!” , are considered as original ideas with a potential of orbit-shifting change and are therefore defined as high impact. Next to that these ideas are easy to implement, and so require low effort.
Next, the lower left quadrant, which is called “quick-wins” also defined as the “Now!” quadrant. The tasks you can find here do not have the highest impact, but also do not require a lot of effort. In other words these are the tasks you can now implement and so a quick-win.
The upper right quadrant shows the tasks that take a lot of effort, but will also make a high impact. These are called “major projects”, they require a greater amount of time and effort to complete, but are considered worthy of the investment. This quadrant can also contain future projects that are not yet implementable and therefore called “How?”.
Last, the “thankless tasks” are shown in the lower right corner of the matrix. The name says it all, tasks shown in the quadrant are, at the moment, not worth spending time on and should not be considered to put into the planning. To add a fourth factor to the how-now-wow we can call this quadrant “Ciao!”. However, it is worth keeping them on your use case backlog. New tools and innovations may make them more feasible in the near future. Consider for instance all the use cases in the Ciao quadrant that moved to Now with the introduction of ChatGPT. Or the Deep Learning revolution that started ten years ago, making image recognition cases feasible.
Implementation
I will not fill out the matrix all by myself. For this I will need the insights of the business as well. As a developer I will have insights on how much effort the defined tasks will take. The business will have a better view on how much value the task can actually add, as they have the domain knowledge, and therefore know what the impact will be.
After defining the priority of the tasks it is time for implementation. For me, working closely together with my stakeholders is important to check if the implementation matches the expectations. This collaboration is also a part of the process I really enjoy. Continuously staying in touch with the stakeholders will help me fully understand the needs of the stakeholders.
Besides, I can also learn from the different views and opinions the stakeholders have on the implementation. Also, it is easier to adjust for changes during the process than reverting implementations that already have been done.
Adoption
As a data engineer, I always look for sustainable (data) solutions. Building data flows that are usable for the long term and not just for the moment. Also it is important that these data flows can be used and maintained even if I do not work there anymore.
Besides, an important part of the data-driven process that must not be forgotten is: the adoption of the implementation. Implementing more data driven solutions in the business processes means that the end users should also be more data minded.
To achieve this they should see the actual value in the implementations, and be convinced that it will help them in their decision-making process. Data driven solutions will work for them, instead of working against them, and therefore will make their work more efficient. End users need to know how, and should be willing to make use of the data driven implementations.
If the end users are not excited about the data innovations, then why even bother making them?
Conclusion
In conclusion, for me, being a good data engineer is not only about having the technical skills. As a data engineer, being able to build a bridge, between business and technology, will add even more value to an organisation. By translating one of my own analytical projects to you, I hope I created more insights in how the combination of two roles can maximise the business value of data and analytics within an organisation.
Aurai provides custom data solutions that help companies gain insights into their data. We engineer your company’s future through simplifying, organizing and automating data. Your time is maximized by receiving the automated knowledge effortlessly and enacting better processes on a foundation of relevant, reliable, and durable information. Interested in what Aurai can mean for your organisation? Don’t hesitate to contact us!
