Building an end to end Machine Learning Pipeline in Bigquery

Google BigQuery is one of the more advanced data warehouses in the market, and has out of the box support for building and training ML models using SQL like statements without requiring any code. This is extremely powerful, however managing end to end ML pipelines in this way can be fragile and requires manual steps to updating training and prediction.

In this article we walk through building a simple end to end BigQuery ML pipeline using the open-source framework Dataform to help us manage the end to end process of data preparation, training and prediction.

Google BigQuery provides some Machine Learning algorithms such as Linear regression, Binary logistic regression etc. To find out more about the models that BigQuery supports, check out the documentation.

A typical workflow for building a machine learning model looks like:

1. Data exploration
2. Data pre-processing (data transformation)
3. Model training
4. Model evaluation on test dataset
5. Prediction/Inference on real dataset

In this article I am going to follow the tutorial from Google Cloud documentation to create a machine learning model with Google BigQuery, please read the official documentation for understanding the technical details.

For managing our end to end pipeline, we are going to use Dataform to help us version control our…