Automating table backups in VertexAI training tasks — through BigQuery
Discover the advantages and the necessity of backing up a table during its usage in a VertexAI training task. We at REEA implemented an automated approach by utilizing Cloud Workflows to retain the tables used in a training task, with the objective of preserving the exact records for future analysis.
Goal: Our machine learning team needs to have the capability to compare data in its original form, as utilized during the training process. Having access to the dataset clone they are able to make decisions on what had changed or measure improvements in the data itself.
This article explains how to automate the process using Cloud Workflow. EventArc triggers the workflow with a VertexAI audit log entry for a new training pipeline. Workflows then extracts the required information, including the BigQuery table to be cloned. Once all necessary data is gathered, the clone statement is executed, creating a new BigQuery table that serves as the retaining dataset for the VertexAI model.
Problem definition
In VertexAI, when the data source is a live BigQuery table and a training job is initiated, it is necessary to duplicate the table to ensure that a copy of the records is retained in their original state for the purposes of model training.
