PinnedKeven PintoinQodea Google Cloud Tech BlogDataproc Serverless & Airflow 2 Powered Event Driven PipelinesIn my previous post, I demonstrated how one can get a Dataproc Serverless pipeline up and running from the CLI. In this post, we’ll look at…Apr 12, 2022Apr 12, 2022
PinnedKeven PintoinQodea Google Cloud Tech BlogRunning pyspark jobs on Google Cloud using Serverless DataprocRun Spark batch workloads without having to bother with the provisioning and management of clusters!.Mar 24, 20221Mar 24, 20221
Keven PintoinQodea Google Cloud Tech BlogA review of the Google Datastream Service - Cloud SQL (PostgreSQL) to BigQueryDatastream introduced the ability to seamlessly replicate data from Cloud SQL to BigQuery late in 2022. Given that (near) real time…Mar 23, 2023Mar 23, 2023
Keven PintoinQodea Google Cloud Tech BlogUsing Datastream as a Data Sync and DR toolIn my last blog we looked at how near real time BigQuery data sync across two regions could not be achieved using the dataset copy service…Mar 14, 2023Mar 14, 2023
Keven PintoinQodea Google Cloud Tech BlogBigQuery Cross-Region Replication Gotchas2022 was the fifth warmest year on record. The July heatwave pushing temperatures to 40°C in parts of the UK was so brutal, that it was a…Jan 26, 2023Jan 26, 2023
Keven PintoinQodea Google Cloud Tech BlogA Centralised Approach to CICD of DAGs on Google Cloud Composer with Google Cloud Build — Part 2In Part 1 of this blog we setup the infrastructure and other building blocks of our CICD pipeline, in this part we will show how we…Oct 27, 2022Oct 27, 2022
Keven PintoinQodea Google Cloud Tech BlogA Centralised Approach to CICD of DAGs on Google Cloud Composer with Google Cloud Build — Part 1In this 2 part Blog, we shall look at how we implemented CICD of DAGs on Google Cloud Composer using Google Cloud Build as our CICD…Oct 20, 2022Oct 20, 2022
Keven PintoinQodea Google Cloud Tech BlogBuild, Test and Deploy Dataflow Flex Templates in Google CloudDataflow Flex Templates are a great way to package and distribute your DataFlow Pipeline. That being said, many engineers trying to…Aug 11, 20224Aug 11, 20224
Keven PintoinQodea Google Cloud Tech BlogHive to BigQuery: Orchestrating the House Move of the WarehouseHive to BigQuery is a common data migration path and there are plenty of ways to get there. This article describes the approach we took…Jun 19, 2022Jun 19, 2022