Data Duets with BigQuery and Gen AI
(views expressed are personal)
With Gen AI a common pattern that is observed, is loading your data to a foundation model and asking for response or generations. This needless to say has raised a lot of Governance and Data privacy related concern. In addition, all developers and practitioners are to learn this additional skill of Gen AI and keep pace at it, so as not to miss out the latest feature.
To alleviate this problem, key products in GCP are getting integrated with Gen AI capabilities natively. This can prevent potential data exports or learning new skills for end users and developers to reap the benefits of Gen AI. In this article, we will explore how the ‘Duet AI’ integration with BigQuery can significantly accelerate the development and analysis effort.
Duet AI, is the AI powered assistance in Google cloud. The feature is in public preview now. Once enabled we get to see these icons in the BigQuery console,
Let us explore the Data Duets with BigQuery and Gen AI,
- As a newbie or even a pro, we look forward for chat like responses on queries such as how do we load data to Bigquery, How do i mask data in Bigquery, How do i apply row level restrictions and so on?. Duet AI, provides a chat like interface where we can ask these questions and get instant responses,
2. For Query development, one can open the query explorer and type their prompts beginning with a ‘#’ sign , such as from the table, find the stores with highest sales for last quarter and then press enter, SQL query is generated and suggested, clicking on the tab icon, prints the query in the console,
Query generation for multiple tables , may require few changes in the column names, as can be seen below. Duet AI, in essence can be used as a peer programmer and not to completely replace the Developer.
3. Let us now take a complex query and try the explanation of it. I have taken a complex query for the TPC-DS dataset used to benchmark performance of Datawarehouses.
After selecting the query, there is a magic wand that appears at the bottom left of the query, on clicking it the Duet Ai chat window opens at the right side with the explanation of the query getting generated
The explanation of the query generated was fairly comprehensive. It will be particularly useful for migration queries written on other platforms and being migrated to BigQuery.
4. Most Data Analysts and Data Engineers will prefer a Python IDE to write queries or build data pipelines, so to accelerate their development, Duet AI has been integrated in IDEs in GCP such as Cloud Workstation and Colab Notebooks in Vertex AI as well. In Colab Notebooks, we get this tooltip to either write a query or generate with AI. After clicking on generate with AI one can start giving the prompts as shown below.
Once Duet Ai is enabled, we can notice, the Duet Ai icon in the bottom right hand corner,
The Duet AI chat appears on the left hand side, where we can enter our prompt to generate the code
Duet AI options are evolving , this will only accelerate the development of Data Analysis and Data Engineering better in the near future. It will also prevent us from scouting for any Gen AI models to be called as an API for AI assistance.
