ETL — Understanding It and Effectively Using It

Using ETL as an Enabler for Data Warehouses, Data Hubs, and Data Lakes

Hashmap
Hashmap
Jan 7, 2019 · 13 min read
Image for post
Image for post

An Introduction to ETL

Why Do We Need ETL?

Traversing the Four Stages of ETL — Pointers to Keep in Mind

Data Extraction and Data Cleaning

Data Source

Usage and Latency

Auditing your Source Data

Analyzing Data Extraction

Steps to Perform Extraction

Challenges Faced During the Extraction Process

Data Cleansing Requirements

Data Cleansing Problems

Potential Problems with Data from a Single Source

Potential Problems with Data from Multiple Sources

Data Cleansing Approach

Data Analysis

Data Profiling

Data Mining

Defining Transformations and Mapping Rules

Verification

Transformation

Backflow and Cleaned Data

Data Transformation

Basic Transformation

Advanced Transformation

Data Loading

Aggregation

An Effective Loading Process

Referential Integrity Constraints

Other Considerations for Effective Loading

Challenges with Incremental Loads

Final Thoughts


HashmapInc

Innovative technologists and domain experts helping…

Hashmap

Written by

Hashmap

Innovative technologists and domain experts accelerating the value of Data, Cloud, IIoT/IoT, and AI/ML for the community and our customers http://hashmapinc.com

HashmapInc

Innovative technologists and domain experts helping accelerate the value of Data, Cloud, IIoT/IoT, and AI/ML for the community and our clients by creating smart, flexible and high-value solutions and service offerings that work across industries. http://hashmapinc.com

Hashmap

Written by

Hashmap

Innovative technologists and domain experts accelerating the value of Data, Cloud, IIoT/IoT, and AI/ML for the community and our customers http://hashmapinc.com

HashmapInc

Innovative technologists and domain experts helping accelerate the value of Data, Cloud, IIoT/IoT, and AI/ML for the community and our clients by creating smart, flexible and high-value solutions and service offerings that work across industries. http://hashmapinc.com

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store