Snowpark DevOps at Ludicrous Speed: The GPT-4 Advantage

> snowdev ai — streamlit “Want to see a time series plot on gmv”

Published in

Snowflake Builders Blog: Data Engineers, App Developers, AI/ML, & Data Science

4 min readAug 23, 2023

Snowpark provides a powerful way to build applications and work with data on Snowflake’s platform. However, setting up pipelines, deploying code, and managing environments can slow down development velocity. This is where combining Snowpark with DevOps practices and GPT-4’s AI capabilities can help accelerate workflows.

The great news is you don’t need to be a DevOps expert to leverage these benefits. The new Python package snowdev makes Snowpark deployments easy for any developer.

Snowdev handles all the DevOps complexities behind the scenes — allowing you to go from code to production faster.

What is SnowDev?

SnowDev is a command-line interface (CLI) tool designed to streamline common development tasks when building on the Snowflake platform. It handles initializing project directories, testing code locally, deployment, and more.

Some of the key features and benefits of SnowDev include:

Scaffolding projects — snowdev init creates the boilerplate code and config files needed to start developing UDFs, procedures, and Streamlit apps for Snowflake. No need to manually setup directories and config.
Local testing — Test functions and procedures locally before deploying to your Snowflake account. SnowDev uses local session object to test local environment.
Single command deploy — With one snowdev deploy command, SnowDev will handle uploading code, registering UDFs/procedures, swapping between temp and prod objects, and testing the deployed objects.
Built-in AI assistant — SnowDev has conversational AI capabilities powered by models like GPT-4. The AI can provide coding suggestions, generate boilerplate code, optimize code, and more based on natural language prompts.
Simple package management — Easily add Python packages locally and have SnowDev bundle them into a zip file and upload to Snowflake stages for access from UDFs and procedures by talking to the snowflake anaconda package channel.

By handling many of the repetitive aspects of setting up projects and deploying code, SnowDev speeds up the development process and lets you focus on writing code.

Action

1. Setup

Setup Python Environment -
Ensure you have Python version 3.10 and above.

pyenv install 3.10.0
pyenv local 3.10.0

Install Snowdev

pip install snowdev

Initialize Snowdev -
This will setup the folders required for the project (src, pyproject.toml, .env)

snowdev init

2. Create Stored Procedure

Now that you have SnowDev installed and initialized, it’s time to see it in action! Let’s walk through the deployment of various components. Before adding these components make sure to fill in your .env with the snowflake credentials see the example.env.

Add a new stored procedure

snowdev new --sproc test_script

This will initialize a new stored procedure files inside the src directory

# src/sproc/test_script/app.py

import datetime

from snowflake.snowpark import Session
from snowflake.snowpark.functions import col


def handler(session: Session) -> str:

    current_time = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
    print(current_time)
    return current_time


# Local testing
if __name__ == "__main__":
    from snowdev import SnowflakeConnection

    session = SnowflakeConnection().get_session()
    handler(session)

and a app.toml file

# src/sproc/test_script/app.toml

[tool.connection]
database = ""
schema = ""
role = ""

[tool.poetry.dependencies]
python = "3.10.0"
snowflake-snowpark-python = "1.5.1"

The connection can be configured on where to deploy the procedure if not given it will use the default session environment based on the .env

Test the stored procedure locally -
This will initiate a separate virtual environment for the test_script project based on the app.toml

snowdev test --sproc test_script

You should see an output as the current time.

Deploy to snowflake

snowdev deploy --sproc test_script

snowdev-deploy-stored-procedure-snowpark

Congratulations!. You have deployed a stored procedure in snowflake in just few steps.

3. Now use AI to create stored procedure

The AI method uses GPT-4 model from OpenAI, and an in-memory vector database — Chroma DB. Before initializing make sure to add your openai key to the environment.

Run the embeddings -
This will create and store the embeddings into our ChromaDB vector database in memory. It embeds some of the sample scripts and Snowpark methods as a knowledge base.

snowdev ai --embed

Create stored procedure -
Let’s create a stored procedure using AI — GPT4

snowdev ai --sproc "I want to fetch data from order table and predict the bad orders using snowflake ml"

Look what we created

In just few steps we are able to deploy and automate the entire process using snowpark.

Conclusion

Combining Snowpark, DevOps principles, and GPT-4 creates a powerful stack. Workflows that were manual and slow can now be automated and optimized.

Teams spend less time on repetitive tasks and more time building. Snowpark developers gain a force-multiplying AI copilot in GPT-4.

To start experiencing the benefits, evaluate integrating GPT-4 into your Snowpark DevOps processes for frictionless development velocities. The future is bright when AI works alongside human developers to ship quality code quickly.

SnowDev is open source. Contributions and feedback are welcome!