Custom Packages in Streamlit-in-Snowflake — 2 easy ways

Introduction

Streamlit in Snowflake, now in Public Preview in AWS, is a powerful — and convenient way to bring your data to life in data applications. Streamlit in Snowflake makes it simple to create, edit, deploy, and share Streamlit applications. Now you can harness the full power of Python, the Python ecosystem of libraries, and the data processing capabilities of Snowflake.

In this short blog post I’ll share two simple ways we can bring in powerful, custom libraries into Snowflake with Streamlit in Snowflake.

Photo by Joan Gamell on Unsplash

Overview of the Solution

We need to upload the files for Streamlit to use into a location that can be accessed from our Streamlit application. There are 2 ways to do this:

• Upload the various files to a directory structure in the same location as the source code for the Streamlit application in Snowflake
• Upload a ZIP file of the auxiliary code so that the Streamlit application can import it

Identifying the Stage Location for the Streamlit

For both of these solutions, we need to know the “root location” for the Streamlit.

Determining the root location depends on whether you created your Streamlit via SQL using CREATE STREAMLIT or via the Snowsight UI. In either case, we just need the “root_location” attribute returned in a DESCRIBE STREAMLIT <streamlit_name>.

If you created the Streamlit using SQL, and you know the database and schema that it is in, you can simply run

DESCRIBE STREAMLIT dbname.schemaname.streamlitname;

using the actual database name instead of dbname, the actual schema name instead of schemaname, and the actual Streamlit name instead of streamlitname.

If you created the Streamlit in Snowsight, your Streamlit will get a hash-type string of letters and number as the name, and the title that you enter in the UI will be saved in the “title” field for the Streamlit. If you know the database and schema for your Streamlit, execute SHOW STREAMLITS IN SCHEMA dbname.schemaname (using the actual database name instead of dbname and the actual schema name instead of schemaname). If you know the database, but not the schema, you can execute SHOW STREAMLITS IN DATABASE dbname (using the actual database name instead of dbname). If you do not know the database or the schema, you can execute SHOW STREAMLITS IN ACCOUNT.

Once you execute the SHOW STREAMLITS command, you can find the Streamlit that is yours based on the “title”, owner, or other identifying information. When you do, note the “name” field. The name will be a string of letters and numbers that appears to be some type of hash. You can also execute a command like the following immediately after your SHOW STREAMLITS command:

SELECT “name” FROM TABLE(RESULT_SCAN(LAST_QUERY_ID())) WHERE “title” = ‘My Cool Streamlit’;

Here I’m showing how to find the Streamlit with a particular title, using My Cool Streamlit as an illustrative example.

With the name, you can now do a DESCRIBE STREAMLIT <name> using the name field from the previous step and note the root location.

Upload the Files Directly

The simplest way to upload the auxiliary source files is to create a directory structure in the same directory as hosts the Streamlit source code. The structure you define in the directories will determine how your Python code imports the packages. This aligns with how Python manages imports on a normal Linux environment.

To load the files to the Stage, you can use one of the following options (among others):

• Use the Snowsight UI to upload files to the Stage (see here)
• Use a SQL client (e.g., SnowSQL) and the PUT command (see here)

Once the files have been uploaded to the Stage, you can simply access them like you normally would in Python. For example, if you created a directory in the root location named my_package and included a file named my_code.py which includes a function named my_function(), your Python code could look something like this:

import streamlit as st
from my_package import my_code

my_code.my_function()

You can update a single file in the directory but doing individual PUT operations (via SnowSQL, Snowsight, etc).

Uploading a ZIP of Files

If you have many files to upload, then you may want a more bulk operation for putting the set of files into the Stage as a set. To do this, we can create a ZIP file of the files we want to import in our Streamlit application. This ZIP file can be a directory structure of files, as well as a set of files. Then, we can upload this ZIP file to the same Snowflake Stage that holds the Streamlit application code.

At this point we can import the packages (or files) into our Streamlit application by modifying the Python path via the sys package, and using the standard import operations.

Preparing the Files

To create the ZIP file, we can use the Linux zip utility. At this point you have a lot of options for how you can structure that ZIP file. Typically, what we see is that folks put their code in a directory, optionally with additional subdirectories or a whole directory structure. This makes it easy to package the code with ZIP using the -r recursive option. Change directories to the parent directory of the directory that contains your code and run:

zip -r my_package.zip my_package

This will create the file my_package.zip of the directory my_package and all of its files and subdirectories. You can change the name of the ZIP file or the directory to suit your needs.

The zip utility is quite powerful and you can view its documentation to see how to include or exclude files or file types, as well as a variety of other powerful options.

Uploading the ZIP File

Once you have the root location, you can upload the ZIP file to that root location in a number of ways:

• Use the Snowsight UI to upload files to the Stage
• Use a SQL client (e.g., SnowSQL) and the PUT command

Accessing the Packages in the ZIP File

We can add the ZIP file to the Python path and then use normal Python import syntax to load the package(s). The following snippet of code demonstrates loading the my_code file from the my_package.zip file and calling the my_function() function.

import sys
sys.path.append("my_package.zip")
from my_package import my_code

my_code.my_function()

Summary

This post walks through a solution to a simple, but often critical, step to creating rich data applications in Streamlit in Snowflake.

Hopefully, this unlocks a whole bunch of Streamlit applications you have been looking to bring to Snowflake. We can’t wait to see what you build!