Data Engineering: What it entails and How to get started

A Chat with Jennifer Ebe and Emeka Boris Ama

On Saturday we had a Chat with two data engineering experts: Jennifer Ebe of Paystack and Emeka Boris of Mono on what data engineering entails and how to get started with data engineering.

What is Data Engineering?

Data Engineering is the complex task of making raw data usable to data scientists and groups within an organization.

A Data Engineer’s make the job easier for data scientists by getting the data and cleaning it. They transport data from one place to another and develop architectures.

Difference between Data Engineer and Data Scientist:

Data Engineers are focused on building architecture for data generation while data scientists are focused on advanced mathematics and statistical analysis on that generated data.

Common Buzz Words in Data Engineering

In every tech career path, understanding the concept is very important. If you understand the concept, you can utilize and tool. These are words are; ETL Pipeline, ELT, Cloud Computing, Databases, Data Warehouses, Data Lakes, Data Marts, Data Pipelines.

• ETL pipeline is a set of processes used to Extract, Transform and Load it to a data warehouse. The difference between the ELT and ETL is the objective of the task.

What is Data Processing?

Data in its raw form is not useful to any organization, this is where data processing comes in. It is a method of collecting raw data and translating it into useful information. Data processing techniques are; Batch Processing, Online Processing Real-Time Processing and Multiprocessing.

Data Engineers Skills and Toolbox

ETL tools, SQL & No SQL, Python, R, Data APIs, Database Systems, Data Warehousing solutions, Monitoring systems, Cloud Engineering, Kafka, Airflow, Terraflow, Kubernetes.

Getting Started in Data Engineering

To start a career in Data Engineering, SQL should be your foundational programming language because you’ll be working with relational databases; then Python or R to enable you to collaborate with software engineers and data scientists. Since data engineering is an upcoming field, there’re a lot of things that you have to learn like; data structures, algorithms, manipulating data systems, building pipelines and monitoring. The demand for data engineers has risen by 38% this year.

Landing your first Data Engineering job

These are ways that you can land your first job in data engineering;

• Build a solid portfolio!
• Reach out and connect to people in your industry.
• Be vocal about your projects.
• Reach out to companies data engineers within companies and ask to intern
• Join a community, contribute and genuinely offer to help

A DAY IN HER LIFE AS A DATA ENGINEER

Jennifer shared what a typical day in her life as a Data Engineer looks like:

It would start with a short meeting where she would provide status updates to the rest of the team about the work she did the previous day, what she planned for the day ahead and any problems or blockers to completing the work she was doing. The rest of the day is spent working on the tasks that have been assigned, this might mean writing code, developing an integration
process to move and transform data. Talking directly with stakeholders to better understand their requirements and to test and validate work that has been completed, monitoring of data pipelines and systems. She would cry a little sometimes when my code her does not work as expected😆

Challenges you’ll face as a Data Engineer

• There are going to be new technologies and new tools every day. You have to be willing to learn so that you can be on top of your game.
• You need to have a diverse skill set.
• Scalability and performance of pipelines
• There’s no one size fits all ETL.
• No two datasets are the same.
• Broken Pipelines/ Data pipelines maintenance

Here’s a link to the recording of the session.

Do follow us on our social platforms; Facebook, Twitter, Instagram, LinkedIn. You can also join us on our Slack Community for more interesting Slack Chats.