Get best performance for PySpark jobs using Parallelize

I have seen sometimes even more that 25x speed when operations are using parallize. This does depend on the other workloads on the cluster. Still the difference is significant

How to become a Data Architect — Data Modelling

Best book ever if you already know some data modelling…

Data Modelling in Columnar Data Stores ?

Introduction:

Data Modelling in Columnar Data Store?

This article provides an introduction to columnar data stores, outlines advantages and popular examples. Additionally, the article outlines key…

Rethinking Surrogate Keys: Efficient Alternatives for Modern Data Models

Troubleshoot Spark/Pyspark performance issues

Steps to help troubleshoot common performance issues in Spark/Pyspark jobs taking EMR/Databricks as example. Of-coarse all these after reviewing there is no change in the data trend or volume.

TL/DR

Glossary of terms — Data Architecture

Data Modelling:

• Entity Relationship Diagram (ERD) — Shows how entities are related.
• Normalization — Organizes data into separate tables to minimize data redundancy.

How to transition to a Data Architect

Here is a list of topics that will help in the transition from…

Generative AI — Threat/Opportunity — Data Engineering

Generative AI is a rapidly developing field with the potential to revolutionize the way data engineers work. Rechnology can be used to automate most tasks that are currently performed manually, such as data cleaning, feature engineering…

Data Integration

Data integration is the process of combining data from multiple sources into a single, unified view. ETL and ELT are two common techniques used for data integration. Both are used to move and consolidate data from various sources into a target system, such as a data warehouse or a data lake. In…

Modern SQL — Not so commonly used, yet powerful features

Most data platforms have evolved and so has the way we write SQLs. Although the basic fundamentals remain the same. I thought it would be useful to write about not so talked about SQL features.