using Airflow, Python, SMTP, Execution Data orchestration is the process of coordinating the execution and monitoring of the data pipelines. Apache Airflow is a well-known open-source workflow management system that provides data engineers with an intuitive platform for designing, scheduling, monitoring, and maintaining their complex data pipelines. Today we will…

Using Python, Pandas & Great Expectations With Great Expectations you can expect more from your data. Great Expectations is one of the leading tools for validating, documenting, and profiling your data to maintain quality. Data Quality is key to making informed decisions. Inaccurate data erodes users’ trust and adaptability of…

Using Apache Kafka, Debezium and Postgres Today we will continue with the Stream Processing and stream Postgres database changes to Apache Kafka. Previously we set up the environment and set the stage for real-time data processing from Postgres. In the previous session we installed Apache Kafka, Debezium and the rest…

Using Jupyter Lab, Jupyter Scheduler and Jupyter Notebook As we create and automate processes, we are likely to run into scheduling needs as we want our processes to run on regular intervals and produce a desired output in an automated fashion. Recently we needed to schedule our Jupyter notebooks on…

using Apache Kafka, Debezium and Postgres Stream processing is a method of continuously collecting and processing real-time or near real-time data. We have covered Stream Processing in the previous article. Today we are going to kick off the Data Streaming series with PySpark using Apache Kafka. Apache Kafka is a…

Using Python, XlsxWriter, Excel Today we will cover how to produce data analysis reports in Excel using Python. Previously we created well formatted excel report. We filtered out the required data format and prepared the output for consumption. We will focus on the following topics. How to create a Pivot…

Stream vs Batch Processing Today we are going to cover stream processing. What it is and how it differs from Batch Processing. So far, we have been processing data with a Batch processing approach. Soon we are covering Stream Processing with PySpark and Apache Kafka. Prior to starting If you…

using Python, Pandas, SQL Server, Postgres and Pytest In the previous article we covered Pytest and developed data quality tests on an Excel dataset. Today we will integrate these data equality tests into the Python ETL pipeline. This ETL pipeline extracts data from a SQL Server database and loads it…

using Python, Pandas and Pytest Testing is an important aspect of ETL pipelines. It ensures we are delivering accurate information to our stakeholders. We want to make sure our data is current, consistent and accurate. If we perform no testing, we build a number of tables, never knowing if something’s…