Hive Query Failure Saved By Config Change — Tez! Creating Data pipelines to ensure information can be extracted from data in a consistent way is an interesting and challenging area in Software Engineering. A quick background and then the simple but effective solution. Built a data pipeline for one of…

A couple of posts back, I gave a brief introduction to Kedro as a Data Science-MLOps tool. Today, we will consider MetaFlow. We all know that data science projects can quickly become complex endeavours, involving multiple data sources, preprocessing steps, model development, and more. Managing the workflow and ensuring reproducibility…

In today’s data-driven world, organizations are generating and collecting vast amounts of data at an unprecedented pace. To derive valuable insights and make informed decisions, businesses require a robust infrastructure that can efficiently handle, process, and analyze this data. Enter the new buzzword in the world of data: Data Mesh…

Data engineering is the backbone of any successful data-driven organization. Efficiently processing, transforming, and managing data is crucial for generating actionable insights and driving business decisions. As data engineering projects become more complex and dynamic, the need for reliable tools to streamline these processes becomes paramount. One such tool that…

Delta Live tables and Data Lakes are both data storage and processing solutions, but they serve different purposes and have distinct characteristics. Here’s an overview of the differences between the two: 1. Purpose: - Delta Live tables: Delta Live tables, also known as Delta tables, are designed for real-time data…

1. Understand the Basics Familiarize yourself with the concepts of AWS Step Functions and Apache Airflow. Understand how they manage workflows and data pipelines, and the differences between them. 2. Identify Existing Pipelines Take an inventory of your existing data pipelines running on AWS Step Functions. …

In the ever-evolving landscape of data-driven technologies, two crucial fields have gained significant prominence: MLOps (Machine Learning Operations) and Data Engineering. While they may seem closely related at first glance, MLOps and Data Engineering serve distinct purposes in the lifecycle of data-driven projects. …

Machine Learning Operations (MLOps) is an essential discipline that combines software engineering and data science practices to streamline the deployment and management of machine learning models. In this blog post, we will explore how to set up MLOps using Kedro, an open-source Python framework that provides a robust development environment…

As a Data Engineer, you get to work with a lot of data in various formats on various platforms. This is just a quick look at having a quick win with data ingestion. In Apache Impala, you can create a table using a CSV file stored in Hadoop Distributed File…