Feb 6, 2020Schedule Your Dataflow Batch Jobs With Cloud SchedulerI am glad that the tutorial has received a lot of attractions from readers, so I have published it on the GCP community tutorial site. Cloud Dataflow is a managed service for handling both streaming and batch jobs. For your streaming jobs, you just need to launch them once without…Dataflow3 min readDataflow3 min read
Jan 7, 2020Dataflow CI/CD with CloudBuildDataflow is a managed GCP service for both batch and streaming processing. The service is based on the Apache Beam project. CloudBuild is a managed GCP CI/CD service. In the post, I will show how to set up CI/CD for Dataflow using Cloudbuild. The code can be found here. As…Cloud Build3 min readCloud Build3 min read
Dec 23, 2019Kedro in Jupyter Notebooks On Google GCP DataprocGoogle Dataproc provides managed Spark environments for customers to execute large scale data analysis jobs. See the official documentation for detailed information. Kedro is a data analytics tool developed by QuantumBlack team for data scientists and data engineers to manage data pipelines. Read the blog post for more information. In…Data Science2 min readData Science2 min read
May 29, 2018Yet another post of what I have learned in the past X years of working as a Y, with X=5 and Y=software engineerI have learned so much from reading blog posts written by people sharing their experience of learning A, using B or doing C, that I want to follow in their footsteps and share my own story of working as a software developer for the the past five years. It would…Startup5 min readStartup5 min read
Apr 9, 2018I Do Not Trust This ComputerI just watched the fantastic documentary “Do You Trust This Computer” by Chris Paine. I highly recommend it to anyone who wants to understand the potential impact of artificial intelligence can have on the future of human race. “You are my creator, but I am your master. ” — Mary…Artificial Intelligence2 min readArtificial Intelligence2 min read
