Open in app
Sign inGet started
Trending
Parquet is Good for OLAP but Not for OLTP Use Cases. But Why?

Parquet is Good for OLAP but Not for OLTP Use Cases. But Why?

Many engineers and data scientists praise Parquet for its efficient compression and fast query performance. While it’s a highly valued…
Go to the profile of Ritam Mukherjee
Ritam Mukherjee
Inside a Netflix Data Engineering Interview.

Inside a Netflix Data Engineering Interview.

A Real time Media — OTT Domain Use Case Question and How to Solve It Together.
Go to the profile of Brahma, The Data Engineer.
Brahma, The Data Engineer.
A Python Library every Data Engineer should know

A Python Library every Data Engineer should know

As a data engineer in a large company, ensuring data quality is a key responsibility. Even if you perform your tasks diligently and rarely…
Go to the profile of Robin von Malottki
Robin von Malottki
Latest
Python Automation for Data Engineers: How to Save Time and Streamline Your Workflow

Python Automation for Data Engineers: How to Save Time and Streamline Your Workflow

Learn How to Eliminate Repetitive Data Tasks with Simple Python Techniques
Go to the profile of Satyam Sahu
Satyam Sahu
Cracking PySpark JSON Handling: from_json, to_json, and Interview-Ready Insights

Cracking PySpark JSON Handling: from_json, to_json, and Interview-Ready Insights

Master PySpark’s from_json(), to_json(), json_tuple(), get_json_object(), and schema_of_json() with practical tips for interview success
Go to the profile of Pritam Deb
Pritam Deb
Fundamentals of Data Engineering Chapter - 01 Summary

Fundamentals of Data Engineering Chapter - 01 Summary

Join me in learning about data engineering. Getting the book Fundamentals of Data engineering by Joe Reis and Matt Housley is the first…
Go to the profile of Adarsh Menon
Adarsh Menon
A Look Inside a Dell Data Engineering Interview

A Look Inside a Dell Data Engineering Interview

Let us Solve It Together and learn Data Engineering Better.
Go to the profile of Brahma, The Data Engineer.
Brahma, The Data Engineer.
The Rise of Data Lakehouses: Is Your Data Warehouse Becoming Obsolete?

The Rise of Data Lakehouses: Is Your Data Warehouse Becoming Obsolete?

Discover the Benefits, Challenges, and Migration Tips for Adopting a Data Lakehouse
Go to the profile of Satyam Sahu
Satyam Sahu
Why Engaging with Upstream Stakeholders Matters in Data Engineering

Why Engaging with Upstream Stakeholders Matters in Data Engineering

How often have unexpected job failures or data quality issues caught you off guard because of unplanned changes from upstream sources? Such…
Go to the profile of Samhitha Poreddy
Samhitha Poreddy
Next Stop After Data Analyst (Data Science or Data Engineering)

Next Stop After Data Analyst (Data Science or Data Engineering)

Graduating from business school and making a career transition into the data world, I hopped on the “Data Science” hype train dreaming of…
Go to the profile of Harris Wan
Harris Wan
Data Skew in Spark : Using Salting while avoiding common mistakes

Data Skew in Spark : Using Salting while avoiding common mistakes

Data skew occurs when the data distribution across partitions is uneven. Imagine you’re working with user transaction data, and two users…
Go to the profile of Ritam Mukherjee
Ritam Mukherjee
Scaling Apache Spark: Understanding Cluster Utilisation with a 50-Node Setup

Scaling Apache Spark: Understanding Cluster Utilisation with a 50-Node Setup

In this article, we will explore how resource management impacts performance in Apache Spark. We will use a 50-node Spark cluster setup to…
Go to the profile of Ritam Mukherjee
Ritam Mukherjee
A Beginner’s Guide to Apache Airflow with GCP Composer.

A Beginner’s Guide to Apache Airflow with GCP Composer.

An effective approach to orchestrating data workflows.
Go to the profile of Adediwura Boluro-Ajayi
Adediwura Boluro-Ajayi
About Towards Data EngineeringLatest StoriesArchiveAbout MediumTermsPrivacyTeams