Data Architecture : A Brief Overview

Data architecture is a critical aspect of successful data engineering. It lays the foundation…

Understanding the Medallion Architecture in Data Engineering

Databricks Asset Bundles (DABs): Deploying with GitHub Actions for CI/CD

Automated Deployment of DABs…

Building Scalable Data Pipelines: A Beginner's Guide for Data Engineers

If you're just starting out in data engineering, you might feel overwhelmed by all the different tools and concepts. One key skill you'll need to master is building data pipelines. A data pipeline is simply…

Building a custom Apache Spark Docker image with AWS Glue Data Catalog support as metastore

Partitioning vs. Bucketing in Big Data: A Beginner's Guide

When dealing with massive datasets…

Data Structures and Algorithms Interview Questions for Data Engineer

Frequently Asked DSA Questions…

Updates around Dataform assertions

New features and improvements to enhance the productivity of implementing tests

Testing Apache Airflow DAGs locally with Testcontainers and LocalStack