DATA DATA ENGINEERING 101

Data Engineering Beyond Hype

Let’s try to dig into Data Engineering core concepts beyond the hype.

Let’s face it, Data Engineers are no longer boring plumbers of data moving it from one place to another. Sure, Data Scientist was the sexiest job post in the last decade, and prompt engineers, are massively hyped right now.

But Data Engineers too are getting a lot of media attention lately. Good or bad, you can’t ignore the recent hype in Data Engineering, fuelled by massive growth in cloud data platforms.

At last, after years of neglect, when we search on Google, 100s of articles and roadmaps are thrown at you to become a Data Engineer. But sadly, Data Engineering is a difficult area to break into. There are hundreds of tools and technologies to master, a lot of concepts to practice, and also there are skills that come only with experience.

State-of-Data-Engineering-Report

This might seem scary, but there is a silver lining. You don’t have to learn everything at once. Unlike other fields, in Data Engineering there are some core concepts that are fundamental to it. Once you master these core concepts, you can skill up in any tools in a couple of weeks.

Hence, I would like to explore some of those core concepts in a series of articles in a tools agnostic manner. This is a real primer, a Data Engineering 101 that looks Beyond The Hype.

Here is the list of blogs in the series. I will keep adding to the list as I write on more topics. If you are starting your Data Engineering journey, or have a couple of years of experience in Data Engineering and would want to level up, keep following the series. Hope this helps in the learning journey. Do share your feedback and thoughts in the comments or drop a mail.

1. 60+ Years of history in Data Processing
2. The Real Value of Data For Business
3. Data-Engineering is a Great career move in 2022–23
4. How to Become a Data Engineer: Complete Roadmap
5. Best Practices for Data Ingestion
6. Data Pipeline Architecture: ways to build Data Pipeline
7. First DE Project: Incremental Roadmap

SQL 201:

1. deleting-duplicate-data-with-no-primary-key-a-data-engineers-worst-nightmare
2. when-do-you-self-join-a-handy-trick