From Analyst to Engineer
With so many data-related jobs to choose from, a data-analyst-turned-engineer shares insights to help you make informed career decisions.
Navigating the vast landscape of data-related jobs can be a daunting task. With so many options available, it’s easy to become overwhelmed and unsure of which path to take. That’s why it’s essential to have a clear understanding of the different roles and what they entail. As someone who has transitioned from a data analyst to a data engineer, I can shed some light on these two positions and help you understand their similarities and differences. Whether you’re considering a career change or thinking about a career in data, my experience can provide valuable insights that can help you make an informed decision. So join me on this journey as we explore the world of data and discover which role is the best fit for you.
My career so far
College (maths, further maths, physics) → university (BSc in mathematics) → graduate data analyst → data analyst → senior data analyst → senior analytics engineer → senior data engineer consultant.
After three years of studying maths in university, I was quite lost and wasn’t sure what jobs to apply for. LinkedIn was just becoming popular then, so before exam season started I created a profile. A few weeks later I got contacted by a recruiter about a graduate data analyst programme. I was unsure what data analysis was but the requirements and skill set aligned to my interests so I decided to apply and a few interviews later, including a maths and English test and one long relaxing summer, I started my first job as a graduate data analyst. One could say that I inadvertently landed my first job in the field of data, with very little knowledge or direction, and that’s all right: you don’t need to have everything planned out from the start.
When I started my data career, I found the experience exhilarating due to the abundance of new technologies and business concepts. I enjoyed learning, and being a data analyst was particularly fulfilling as I was able to create valuable insights, communicate them effectively, and promote more data-driven decisions.
However, as an analyst, I experienced two big challenges:
- communicating data successfully and
- cleaning data.
These problems come hand in hand as inaccurate insights can result from unclean data. I encountered problems like duplicates, missing data, and complex structures, which meant that I spent hours manipulating data to get the analysis done.
Typically, data engineers are tasked with resolving issues in the data pipeline that causes these problems. However, waiting for errors to be fixed isn’t my strength. And I really enjoy troubleshooting. That alone should have hinted that I might like data engineering, but it took me three years to realise.
After working as a data analyst for a few years, I craved a new challenge. While I enjoyed creating dashboards, I found greater satisfaction in building data pipelines, optimizing queries, and troubleshooting code. However, lack of confidence prevented me from pursuing data engineering roles, as I believed I didn’t have sufficient skills or experience; impostor syndrome is a common obstacle for individuals who doubt their abilities and feel like they don’t belong in a field. Although I can’t provide you a method to avoid it, I can tell you how I eventually felt less of an impostor.
I started noticing my peers asked me more engineering questions and I also got more involved in conversations that had more of an engineering focus than analytics. There is no one single route to be confident, but it often involves stepping out of your comfort zone, acknowledging knowledge gaps, and expressing a willingness to learn and explore new opportunities. As I became more immersed in data engineering work, I eventually mustered the courage to express my interest in the field to my manager.
What pushed the realisation in the end?
During a discussion with my director at a previous job we questioned whether the team had the correct job titles. I had to justify why the team needed a job title change from data analyst to data analytics engineer (which eventually got changed) due to the wide range of engineering skills in the team because we have to deal with large amounts of data. My team was building products using data and optimising queries on a daily basis. After that conversation, I came to the realisation that I am a data engineer and shouldn’t hold myself back due to self-doubt when my abilities prove otherwise.
Why am I telling you this?
After having about five job titles in five years, I’ve come to understand that job titles may not carry as much weight as I believed. Sure, they offer a general sense of the job description, but you can also tailor a job to what you want it to be. You could be an analyst who performs more data engineering work or vice versa. It’s more about making the most of your skills and talents and pursuing your interests.
What was my biggest challenge in moving from data analyst to engineer?
As I mentioned earlier, confidence in myself, specifically the feeling of being an impostor, was one of the most challenging aspects of changing the direction of my career; particularly because without being confident that “I am an engineer,” I couldn’t then find the courage to apply to relevant jobs. On top of that, when I changed job roles, a lot of my previous responsibilities stayed with me, so as an engineer I was doing a lot of analytics. I did enjoy analytics and still do, but it was challenging to change between the two roles on a daily basis.
What do they actually do?
Both data analyst and engineer roles are essential in the field of data, with differing focuses and responsibilities.
Data engineers are responsible for preparing data for analysts, which includes designing, building, and maintaining data pipelines, databases, and warehouses, while analysts consume, interpret, and analyse the data to extract valuable insights and answer business questions. Although their focuses differ, they share very similar skill sets.
Requirements for data analyst and engineer jobs often overlap, which is not surprising. However, it is important to keep in mind that job descriptions are just a starting point, and individuals may perform their roles differently based on their individual strengths and weaknesses.
Conclusion
From my research and experience, analysts and engineers have transferable skills that can be used by either of the roles, and the choice between the two usually comes down to this:
- what outcomes usually motivate you and
- which skill set you prefer to use day-to-day.
If you have a preference for the business-facing aspect of things, analytics may be the path for you. Conversely, if you enjoy delving deep into technical problems, engineering may be more suitable. However, it is not always a choice between one or the other; as I mentioned earlier, you can usually tailor your job how you want it—do as much analytics or engineering as you desire. Additionally, there are emerging tech roles such as analytics engineers, so it’s important not to get too caught up in job titles and instead focus on researching the skills and responsibilities of the position you’re interested in. Lastly, this is only my experience and there are many other paths into data or data engineering (you don’t have to become an analyst first before becoming an engineer), although analytics is one of the most common routes into data.
Phrances is a data professional who specialises in data engineering and analytics. She loves working with different technologies to build data pipelines and products to help businesses understand the value of their data.
Slalom is a global consulting firm that helps people and organizations dream bigger, move faster, and build better tomorrows for all. Learn more and reach out today.
