Data Engineering — An Introduction
Understanding Data Engineering at A Glance
In recent times, the concepts of Data Engineering and Data Engineer have gained significant popularity, alongside data analysis and data science. Many in the field of data management are currently engaging in discussions about data engineering, asserting that there is a higher demand for data engineering roles compared to data analysis and data science. It is argued that the data engineering process plays a pivotal role in overall data management.
What is Data Engineering and Who is Data Engineer?
Data Engineering is a process that includes development, research, and construction of data pipleline which transforming raw data into high quality, consistent, and reliable dataset that is suitable for analysis by end users.
Data Engineer, in the other hand is people who execute the data engineering process. Below are common tasks of data engineer:
- Acquire data set and align with the business needs.
- Design and develop data pipelines to do the Extract, Transform, and Load (ETL) process from various data sourced.
- Implement and maintain database to store data effectively.
- Collaborate with management to understand company and business objectives.
- Ensure data quality and and integrity through data cleansing process.
- Monitor and throubleshoot data pipeline issue.
- Collaborate with data analyst and data scientist to understand their neeeeds and requirements for further analysis.
- Implement security measure to ensure protection of sensitive data.
To effectivelly perform the following tasks below skills are need to be acquired by data engineers:
- Programming Skills : Python, Java, SQL.
- Data Warehousing : Amazon Redshift, Google BigQuery.
- Database Management : PostgreSQL, MySQL, MongoDB, Cassandra.
- ETL Tools : Apache Spark, Apache Airflow, Talend.
- Data Modelling : ER Diagram, Database Schema skills.
- Big Data Technologies : Spark, Hadoop.
- Cloud Platforms : AWS, Google Cloud Platform, Azure.
- Version Control : Knowledge in version control system such as Git.
In addition to the mentioned technical skills, data engineers should have several soft skills such as communication skills because they will collaborate with various stakeholders, and eager to learn because technologies in the data fields are always keep changing.
Data Pipeline
Data pipeline is a process to extract data from multiple resources, cleansing the data, then finally store the data into target system, typically database or data warehouse.
Data pipeline process includes following:
- Data Collection : Process of collecting data from various data sources, such as database, API, file system (csv, excel, etc), event. The data could be have different format.
- Data Cleaning : In this stage data is cleaned, the cleaning process involves checking the missing value, correcting any error, transforming data into consistent and reliable format. The goal of the cleaning process is to produce accurate and reliable data.
- Data Integration : Data from different sources are merged into one single dataset.
- Data Storage : The processed that is stored in the target system.
Those following process is commonly known as ETL process, Extract, Transform, and Load process. Extract : data collection, Transform : data cleaning & integration, Load : data storage.
Extract : to extract data from multiple resources, data can be extracted from file, database, API, Web scraping, sensor data, third party.
Transform : change the data into format that is suitable with the target system. Data is transformed so that it has consistent structure and definition. Transformation process typically includes data cleaning, data aggregation, data filtering, joining data, data validation, data mapping, and calculations.
Load : finally data then loaded into target system, such as data warehouse, operational database, and many more.
ETL vs ELT
Besides ETL there is also a popular term called ELT, nowadays ELT also has been widely used by industries. Here is the difference between ETL and ELT.
Extract Transform Load (ETL)
- Extract data from various resources → transform the data → load data to target system.
- Data transformation is conducted in different server.
- Raw data is not sent to data warehouse.
- Ideal for small dataset that does not require complex transformation.
Extract Load Transfrom (ELT)
- Transformation process occurs after the load process.
- Data transformation is processed in the data warehouse.
- Sent raw data into data warehouse.
- Can handle a large amount of dataset.
ETL vs ELT process
Data Warehouse & Data Mart
Either using ETL or ELT process data will store in the data warehouse or also data mart. So what is different between data warehouse and data mart.
Data Warehouse is a centralized storage that is used by end user to peform data analysis, it can contain raw data or processed data. A data warehouse will have some databases.
Data Mart is a part of data warehouse, data mart is intended for specific user, for example data mart for sales & marketing team, human resources team, etc.
Data warehouse and data mart illustration:
Here’s a brief summary of data engineering. I hope this article provides additional insights for readers, particularly those interested in learning about data engineering.
