How to become a Data Architect

Modern SQL — Not so commonly used, yet powerful features

Most data platforms have evolved and so has the way we write SQLs. Although the basic fundamentals remain the same. I thought it would be useful to write about not so talked about SQL features.

Get best performance for PySpark jobs using Parallelize

I have seen sometimes even more that 25x speed when operations are using parallize. This does depend on the other workloads on the cluster. Still the difference is significant

Troubleshoot Spark/Pyspark performance issues

Steps to help troubleshoot common performance issues in Spark/Pyspark jobs taking EMR/Databricks as example. Of-coarse all these after reviewing there is no change in the data trend or volume.

TL/DR

Rethinking Surrogate Keys: Efficient Alternatives for Modern Data Models

Generative AI — Threat/Opportunity — Data Engineering

Generative AI is a rapidly developing field with the potential to revolutionize the way data engineers work. Rechnology can be used to automate most tasks that are currently performed manually, such as data cleaning, feature engineering…

Data Engineering

Study plan

Foundations of Data Engineering (5 days):

Data Integration

Data integration is the process of combining data from multiple sources into a single, unified view. ETL and ELT are two common techniques used for data integration. Both are used to move and consolidate data from various sources into a target system, such as a data warehouse or a data lake. In…

Data Modelling in Columnar Data Stores ?

Introduction:

Data Modelling in Columnar Data Store?

This article provides an introduction to columnar data stores, outlines advantages and popular examples. Additionally, the article outlines key…

Glossary of terms — Data Architecture

Data Modelling:

• Entity Relationship Diagram (ERD) — Shows how entities are related.
• Normalization — Organizes data into separate tables to minimize data redundancy.