Run Spark batch workloads without having to bother with the provisioning and management of clusters!. If you are interested in running a simple pyspark pipeline in Serverless mode on the Google Cloud Platform then read on.. The ability to run spark jobs in serverless mode is a great idea, however…

Datastream introduced the ability to seamlessly replicate data from Cloud SQL to BigQuery late in 2022. Given that (near) real time replication of operational databases is one of the most common uses cases we observe as part of our client requirement, we decided to review this service and put it…

In my last blog we looked at how near real time BigQuery data sync across two regions could not be achieved using the dataset copy service provided by Google and how we had to abandon the approach. …

2022 was the fifth warmest year on record. The July heatwave pushing temperatures to 40°C in parts of the UK was so brutal, that it was a rare occasion that my camera was switched on during office meetings. There were times, I felt that if I stayed out too long…

In Part 1 of this blog we setup the infrastructure and other building blocks of our CICD pipeline — in this part we will show how we promote code within our managed environments via our CICD pipeline. A managed environment is any Cloud Composer environment managed by a CICD service…

In this 2-part Blog, we shall look at how we implemented CICD for Directed Acyclic Graphs (DAGs) on Google Cloud Composer using Google Cloud Build as our CICD platform. In Part 1 (this part), we set the basic building blocks in place to allow us to do a multi project…

Dataflow Flex Templates are a great way to package and distribute your DataFlow Pipeline. That being said, many engineers trying to implement flex templates for the first time, face quite a few challenges. …

Hive to BigQuery is a common data migration path and there are plenty of ways to get there. This article describes the approach we took with one of our clients. …