Published inData Engineer ThingsI completed a Senior Data Engineer Code Challenge for fun, and this is how it went. PART IIQuestion: Using MySQL’s public employee sample database, create a DAG to move data from the employee’s table to BigQuery.Apr 27, 20245Apr 27, 20245
On Learning... Continuous LearningHello friends, I am not sure I have said Happy New Year, so Happy New Year, Friends 🥳! I hope the year is going according to plan and you…Mar 9, 20243Mar 9, 20243
Published inData Engineer ThingsI completed a Senior Data Engineer Code Challenge for fun, and this is how it went.Question: Using MySQL's public employee sample database, create a DAG to move data from the employee's table to BigQuery. There should also…Mar 2, 202411Mar 2, 202411
Published inArt of Data EngineeringWriting PySpark Integration Tests with ChispaThis past week, I was tasked with writing an integration test for several PySpark jobs. Before now, I had only created unit tests, so this…Nov 4, 2023Nov 4, 2023
Published inData Engineer ThingsMigrating Data From MongoDB to Azure SQL Server with Azure Data FactoryWhy do we need to move data?Aug 5, 20232Aug 5, 20232
Published inTowards Data EngineeringData Warehouse vs Data LakeI have been a data engineer for over four years and have mostly worked with data warehouses. Recently I started working with a data lake…May 6, 2023May 6, 2023
Published inTowards Data EngineeringSelf-Healing Data Pipelines- Part 1Hello! Missed me? First, some personal news! I finally finished my Datacamp data engineering track!May 1, 20231May 1, 20231