PinnedKeven PintoinAppsbroker CTS Google Cloud Tech BlogDataproc Serverless & Airflow 2 Powered Event Driven PipelinesIn my previous post, I demonstrated how one can get a Dataproc Serverless pipeline up and running from the CLI. In this post, we’ll look at…7 min read·Apr 12, 2022----
PinnedKeven PintoinAppsbroker CTS Google Cloud Tech BlogRunning pyspark jobs on Google Cloud using Serverless DataprocRun Spark batch workloads without having to bother with the provisioning and management of clusters!.6 min read·Mar 24, 2022--1--1
Keven PintoinAppsbroker CTS Google Cloud Tech BlogA review of the Google Datastream Service - Cloud SQL (PostgreSQL) to BigQueryDatastream introduced the ability to seamlessly replicate data from Cloud SQL to BigQuery late in 2022. Given that (near) real time…8 min read·Mar 23, 2023----
Keven PintoinAppsbroker CTS Google Cloud Tech BlogUsing Datastream as a Data Sync and DR toolIn my last blog we looked at how near real time BigQuery data sync across two regions could not be achieved using the dataset copy service…5 min read·Mar 14, 2023----
Keven PintoinAppsbroker CTS Google Cloud Tech BlogBigQuery Cross-Region Replication Gotchas2022 was the fifth warmest year on record. The July heatwave pushing temperatures to 40°C in parts of the UK was so brutal, that it was a…7 min read·Jan 26, 2023----
Keven PintoinAppsbroker CTS Google Cloud Tech BlogA Centralised Approach to CICD of DAGs on Google Cloud Composer with Google Cloud Build — Part 2In Part 1 of this blog we setup the infrastructure and other building blocks of our CICD pipeline, in this part we will show how we…5 min read·Oct 27, 2022----
Keven PintoinAppsbroker CTS Google Cloud Tech BlogA Centralised Approach to CICD of DAGs on Google Cloud Composer with Google Cloud Build — Part 1In this 2 part Blog, we shall look at how we implemented CICD of DAGs on Google Cloud Composer using Google Cloud Build as our CICD…15 min read·Oct 20, 2022----
Keven PintoinAppsbroker CTS Google Cloud Tech BlogBuild, Test and Deploy Dataflow Flex Templates in Google CloudDataflow Flex Templates are a great way to package and distribute your DataFlow Pipeline. That being said, many engineers trying to…7 min read·Aug 11, 2022--4--4
Keven PintoinAppsbroker CTS Google Cloud Tech BlogHive to BigQuery: Orchestrating the House Move of the WarehouseHive to BigQuery is a common data migration path and there are plenty of ways to get there. This article describes the approach we took…6 min read·Jun 19, 2022----
