Open in app
Sign inGet started
Trending
Fail Fast or Quarantine? Two Data Quality Patterns Every Spark Engineer Should Know

Fail Fast or Quarantine? Two Data Quality Patterns Every Spark Engineer Should Know

Learn when to fail fast or quarantine bad data in Spark pipelines.
Go to the profile of Marcel Kennert
Marcel Kennert
No SQL? No Problem: Ask Your Database Questions in Plain English

No SQL? No Problem: Ask Your Database Questions in Plain English

Non-members can access the full article through this Link.
Go to the profile of Ritam Mukherjee
Ritam Mukherjee
Databricks Gen AI Engineer Associate Day 1: Introduction

Databricks Gen AI Engineer Associate Day 1: Introduction

Your First Step Toward Building Real-World Enterprise-Ready Gen AI Applications on Databricks
Go to the profile of THE BRICK LEARNING
THE BRICK LEARNING
Latest
🚨 Warning: This Data Engineering + ETL Secret Could Get You Fired!

🚨 Warning: This Data Engineering + ETL Secret Could Get You Fired!

The ETL Conspiracy
Go to the profile of Mayur Koshti
Mayur Koshti
15 Common Spark Errors in the Big Data Industry — Causes, Detection & Detailed Fixes

15 Common Spark Errors in the Big Data Industry — Causes, Detection & Detailed Fixes

Apache Spark is widely used for building distributed data processing pipelines, but it frequently encounters operational and runtime…
Go to the profile of Solon Das
Solon Das
Databricks Gen AI Engineer Day 9: Evaluating RAG Solutions for Relevance and Accuracy

Databricks Gen AI Engineer Day 9: Evaluating RAG Solutions for Relevance and Accuracy

Techniques, Metrics, and Human-in-the-Loop Evaluation for High-Quality GenAI Systems
Go to the profile of THE BRICK LEARNING
THE BRICK LEARNING
Databricks Gen AI Engineer Day 8: Assembling a RAG Application on Databricks

Databricks Gen AI Engineer Day 8: Assembling a RAG Application on Databricks

LangChain + ChatDatabricks + Mosaic Vector Search = A Production-Ready RAG Pipeline
Go to the profile of THE BRICK LEARNING
THE BRICK LEARNING
ML Capstone Project Day 6: Time Series Analysis for E-Commerce User Behavior

ML Capstone Project Day 6: Time Series Analysis for E-Commerce User Behavior

Predicting, Diagnosing, and Optimizing Funnel Performance Over Time
Go to the profile of THE BRICK LEARNING
THE BRICK LEARNING
Apache Kafka TableFlow with Shift Left strategy in Real Time Streaming of Data

Apache Kafka TableFlow with Shift Left strategy in Real Time Streaming of Data

Most of you who are beginners in data engineering might not have heard of the term shift left, what exactly is it responsible for? The…
Go to the profile of Devanshu Dandekar
Devanshu Dandekar
Data Engineering with Databricks Day 78: Building Bronze-Silver-Gold Layers with SAP Data

Data Engineering with Databricks Day 78: Building Bronze-Silver-Gold Layers with SAP Data

Now that we’ve mapped SAP tables to retail analytics use cases, it’s time to build the foundational Lakehouse architecture on Databricks…
Go to the profile of THE BRICK LEARNING
THE BRICK LEARNING
Parameterization in Azure Data Factory: Make Your Pipelines Smarter and Reusable

Parameterization in Azure Data Factory: Make Your Pipelines Smarter and Reusable

As data engineers, we often build pipelines that follow a similar structure but operate on different datasets, environments, or…
Go to the profile of Vishal Singh
Vishal Singh
Using DBT with linked servers in SQL Server.

Using DBT with linked servers in SQL Server.

Quick tutorial on using DBT with linked servers in SQL SERVER
Go to the profile of Adediwura Boluro-Ajayi
Adediwura Boluro-Ajayi
Corporate Finance on Databricks Day 1 : Financial Narrative Generation Using Agentic RAG + ML…

Corporate Finance on Databricks Day 1 : Financial Narrative Generation Using Agentic RAG + ML…

Use Case Overview: Automating Financial Narrative Generation
Go to the profile of THE BRICK LEARNING
THE BRICK LEARNING
About Towards Data EngineeringLatest StoriesArchiveAbout MediumTermsPrivacyTeams