Data Engineer to Lead Data Engineer:
My Journey in the Health Care Company:
Embarking on my journey with this health care company four years ago as a Data Engineer, I was eager to dive into the world of data engineering and analytics.
The thrill of working with extensive datasets, coupled with the potential to influence critical business decisions, was a constant source of motivation.
Year One and Two: Data Engineer
In the beginning, my experience was both challenging and perplexing. I spent the first two years loading CSV files into Hive tables, using Sqoop to transfer data from Oracle databases, and making GET requests to various APIs. To be frank, it often felt like I was just moving data around without a clear understanding of the bigger picture. I was essentially following the guidance of a senior colleague who helped outline the tasks for each assigned story. Despite the seemingly mundane nature of the work, I struggled to grasp the complexities involved.
The knowledge transfer sessions were overwhelming; they seemed like a blur of technical jargon with little practical meaning. I found myself working 16–18 hours a day during the first few months, relying heavily on Google to navigate through basic Python and SQL tasks. The vast codebases and complex queries, some spanning thousands of lines, were daunting. It took me about six months to become somewhat familiar with basic production-level applications. Writing queries was part of my job, but I had no idea how or where the data was ultimately used. My main concern was ensuring that my ETL pipelines ran on schedule and that the data was processed correctly.
During this period, I picked up knowledge of the Hadoop ecosystem, PySpark, and shell scripting. Most of my collaboration was with that one senior colleague, who gradually helped me gain the confidence to speak up in meetings. Interacting with project managers, business analysts, and other engineers was a significant milestone. Continuous learning was crucial, and I felt fortunate to be paid to learn. Staying updated with the latest tools and technologies helped me streamline our data processes, and my dedication to improvement did not go unnoticed.
Promotion to Senior Data Engineer
At the end of my second year, I was promoted to Senior Data Engineer. This new role brought with it a broader perspective and additional responsibilities. Previously, as a Data Engineer, my focus was primarily on the tasks assigned to me, with ample time for learning. However, in my new role, I had to work more independently and assumed that self-sufficiency was expected. I became adept at solving daily issues and knew whom to contact for specific needs.
Instead of working on isolated tasks, I now handled entire features, which involved adding new functionality or enhancing existing ones. Mentoring junior engineers became part of my responsibilities. I was determined to provide them with the application knowledge I had wished someone had given me. Despite the increased responsibilities, the company’s expectations remained the same. My team lead continued to offer guidance, sharing his technical knowledge generously.
I started discussing technical findings with other senior engineers, which broadened my understanding. I improved my proficiency in Python and SQL, mastering complex queries and navigating codebases with ease. Learning about various data ingestion, reporting, and visualization tools was part of my growth. My role shifted from execution to strategy, focusing on optimizing processes and identifying new opportunities for leveraging data. I began to see the bigger picture — how the company made money, where my role fit into the overall process, and how my work impacted the business.
Promotion to Lead Data Engineer
Two more years later, I achieved another significant milestone: promotion to Lead Data Engineer. This role marked a profound shift in my career, requiring me to balance technical expertise with managerial skills. I now had to collaborate with multiple roles: junior engineers, senior engineers, project managers, data scientists, managers, and cross-functional teams. Coming from a purely technical background, this new role presented unforeseen challenges.
Effective communication became crucial, as I needed to convey technical details to business stakeholders and vice versa. With four years of application and platform knowledge, I became the go-to person for troubleshooting and problem-solving. I knew the data inside out and had been involved in the productionization of new pipelines and the establishment of our data warehouse.
Understanding the business aspects became second nature. I was comfortable discussing requirements with business stakeholders and data scientists. My focus expanded to include long-term solutions, considering scalability, maintainability, robustness, code quality, data security, and data integrity. Designing solutions that spanned multiple code components and applications became part of my responsibilities, often involving frontend development with React and Node.js, which gave me the opportunity to learn JavaScript.
While maintaining technical expertise, I dedicated more time to planning and prioritizing projects. Ensuring our data infrastructure scaled effectively involved overseeing complex data pipelines, implementing robust governance practices, and staying current with emerging technologies.
Leading a team required more than technical knowledge. I connected with individual engineers, understanding their strengths and providing assistance to foster their growth. Different approaches were necessary to align a diverse team of 12 engineers towards a common goal. My interactions with the team evolved, prioritizing an inclusive environment where everyone felt empowered to contribute their best. Regular meetings and one-on-one sessions were pivotal for understanding individual strengths and nurturing career aspirations.
The transition from Senior to Lead Data Engineer demanded a deep understanding of both technical and managerial aspects. I focused on fostering a collaborative environment and promoting a culture of continuous learning.
Interested to know what I did in each role? find out here
Reflections on My Journey
Over the last four years, the constant influx of new technologies and tools often felt overwhelming. There were times when experienced engineers sought my advice, a humbling reminder that not everyone knows everything. I embraced opportunities to learn Google Cloud technologies as the company planned to migrate to the cloud. Earning certifications in Google Cloud and AWS, all funded by the company, was a significant part of my growth. Learnt and built applications in micro services arcitecture, intoduced event driven architecutre in the team.
Throughout my journey, I outperformed my peers by embracing continuous learning and discipline. While my seniors and experienced colleagues provided valuable insights, my proactive approach to self-improvement set me apart. Consistently seeking new challenges and opportunities for growth, both technically and professionally, defined my journey.
In conclusion, my progression from Data Engineer to Lead Data Engineer has been marked by continuous growth and evolving responsibilities. By staying disciplined and committed to learning, I have navigated challenges and seized opportunities to make meaningful contributions to the company. My journey underscores the importance of adaptability, proactive learning, and a relentless pursuit of excellence in achieving career growth and success.
If you are aspiring Data Engineer or a Data Engineer trying to add more weight to your skill bag or even if you are interested in topics like this, please do hit the Follow 👉 and Clap 👏 show your support, it might not be much but definitely boosts my confidence to pump more usecase based content on different Data Engineering tools.
Thank You 🖤 for Reading!
