Researching for a Data Engineer role
Demystifying Data Engineer hiring process
This is the third article in the Demystifying Data Science Interview series. The series talks about the journeys one would take in order to land into Data Science roles.
Sam is a Full Stack engineer with several years of experience. More recently he has developed an interest in Data Science, being inspired by the Data Science team at his company. He enrolls in several online courses, learning the trades and building his skills by working on side projects. Through this, he gains an understanding of the end-to-end Data Science process and recognized its importance.
Sam’s company is a large insurance company with multiple business units. One of the teams that Sam works with has a growing Data Science team. As they take on new projects, they are looking for additional Data Engineers to help support the growing demand for data internally.
This presents Sam with an opportunity to break into the Data Science field by leveraging his Software Engineering background. So he applies for the position internally.
Sam’s Profile
- Full Stack Engineer with an interest in Back End
- Interested in Data Science — Engineering
- Taken some online courses, built some projects to solidify his learnings
Company Profile
- Large company
- New project looking for new data engineer > internal switch
Company Maturity
This is quite a mature company with several mature products. There are plenty of opportunities to work on various types of projects across the tech stack, as well as get exposed to different markets. Projects can be much larger in scope, involving many teams across many departments.
A related note is that this is the opposite of being the first Data Science professional at another company. The foundation has been built already and you can continue building on top of it. There is less for you to figure out fully on your own.
In summary, a mature company provides plenty of stability, but it may take longer than smaller companies to make decisions and get things done.
What does this mean for Sam, our aspiring Data Engineer?
Working at a larger company brings with it many benefits that Sam could leverage.
For one, he doesn’t need to look elsewhere to switch to Data Engineering — an opportunity opened up within his company that he went for. This allows Sam to be exposed to various Data Science roles by working closely with others and by being able to switch to one.
Being in a larger team or company can also mean that there are more senior people to learn from. Their combination of more overall experience and time with the project will be invaluable to Sam. He can level up his skills much faster by working with existing systems and the mentorship of the team.
Lastly, there is an opportunity for a higher level of specialization. The larger the company, the more focused teams can be on certain domains. If Sam is passionate about the insurance industry, he can further specialize in it with this role.
Industry
Insurance has historically used a ton of data to help it build a sustainable business and industry. Data is essentially at the heart of the industry.
That said, machine learning and data science are helping the industry evolve or disrupting it completely. Remember, no industry is immune to these changes. Opportunities to improve inefficiencies through automation are all over the place.
What does this mean for Sam?
Sam already got a front-row seat into how important the data is through his role in helping build the companies’ products. With his newly gained skills, he could be aware of better techniques to improve processes or the product. As he ramps up in his new role, he could be in a position to suggest new data product ideas, or how to better serve the internal teams.
All of this will help give Sam a perspective on the industry and observe how data science can disrupt an industry that is already data-driven.
Role & Availability of Data
Insurance companies having always leveraged data to make their business decisions have no lack of data of course. Data helps inform their product offerings — it plays a very important role.
Furthermore, the data is easily available for the companies. They don’t need to scrape it from the web or an external party. People answer lots of very personal questions when they sign up for insurance — e.g. remember when you signed up for your car or home insurance forms. All the information people provide helps the insurance companies decide on the best rates to charge their customers.
What does this mean for Sam?
Our aspiring Data Engineer, Sam will be in a team where data is already available and used widely. Given this, his role may require him to spend less time gathering data and more time in helping build/improve the platform to more easily serve the data that already exists.
Sam’s Role within the Data Stack
Bringing it back to Sam, he is excited to get into Data Science by leveraging his Software Engineering skills. Two paths available to him would be Data Engineer or Machine Learning Engineer.
The Data Engineer role within his company could be a good first role to dip his toes in. He will learn how companies build their data foundations along with the challenges they face. Working to serve other Data teams will give him a wide perspective on how data is leveraged across a company and in an industry as a whole.
The Machine Learning Engineer role is more concerned with bringing models into production. This could also be an exciting career path for Sam to pursue after he builds his data foundations. As the work done by the Data Science team advances, the teams could require an ML Engineer to help them productionalize models and integrate them with the rest of the software.
Sam is in a good position to pursue either role.
The Data Engineer Interview Process
So Sam is interested — what is the journey that awaits him?
Step 1: Apply or Recruiter reaches out
In this case, Sam was able to apply to an internal position which opened up as the team was growing.
Step 2: Initial Interview
This initial interview would be different from the usual chat with HR for a large company. Here the initial interview could be an informal conversation with the hiring manager to discuss the team’s needs and Sam’s interests.
If both the hiring manager and Sam feel there is a fit, Sam can go on to the next step.
Step 3: Technical/Behavioral Interview with the Hiring team
Already being part of the company, Sam can skip the usual technical screen that external candidates have to do. He can jump straight to the interview with the hiring team, where they will assess his technical skills and cultural fit.
This interview could be a combination of System Design, Data Structures, and Algorithms along with behavioral questions. Given that the team works with multiple Data Science teams, they may prioritize the ability to clearly communicate with multiple stakeholders and be a team player who is able to deliver to internal customers.
Thanks for reading! We will continue this series with other scenarios where this same framework can be applied. Until the next episode!
Subscribe to our Acing AI newsletter, if you are interested:
Interested in learning how to crack machine learning interviews?
