Snowpark ML with AWS and Azure for AI and ML

AI and ML

Artificial Intelligence (AI) and Machine Learning (ML) are rapidly evolving technologies shaping our future. Simply put the process of developing ML models is: 1. performing exploratory data analysis and preparing historical data for model training 2. training and evaluating models and 3. deploying the model (or models) for inference. And through these three steps making certain to track all of the artifacts (data, models, deployments, etc.) making certain the lineage between these artifacts is tracked.

Snowflake’s AI Capabilities with Snowpark

Snowflake offers robust AI capabilities with its Snowpark ML. Snowpark ML is the Python library and underlying infrastructure for end-to-end ML workflows in Snowflake, including components for model development and operations. With Snowpark ML, you can use familiar Python frameworks for preprocessing, feature engineering, and training. You can deploy and manage models entirely in Snowflake without any data movement, silos, or governance tradeoffs

Azure’s AI Capabilities with AzureML

AzureML is a service provided by Microsoft Azure designed for developing, training, and deploying machine learning models at scale. It offers a wide range of tools and services for AI, supporting various stages of the ML lifecycle, from data preparation to model deployment. AzureML integrates seamlessly with various data sources and platforms, providing a flexible and powerful environment for AI development.

Within an AzureML workspace users can leverage an IDE to perform their ML tasks as well as leverage a UI to perform certain functionality. AzureML includes a robust set of tools for MLOps and tracking ML work including frameworks like MLFlow.

Amazon’s AI Capabilities with SageMaker

Amazon SageMaker is a fully managed service that enables developers and data scientists to quickly build, train, and deploy machine learning models at scale. SageMaker removes the heavy lifting from each step of the ML process, making it easier to develop high-quality models.

With tools for every step of the machine learning process, SageMaker helps users to get more accurate results faster. The two primary “sub-products” within Sagemaker are Sagemaker Studio for an IDE, code first development interface and Sagemaker Canvas for GUI interface for preparing data, building and deploying models.

Using Snowflake with AzureML

The integration of Snowflake with AzureML is detailed in the below architecture. Most joint customers will leverage an AzureML workbook inside the workspace and utilzie the Snowpark push-down service to prepare the data then once the data is prepared customers will decide whether they want to train model in Snowpark ML or materialized their data into AzureML via the Snowflake connector and train models in AzureML. Last customers will either deploy models to AzureML or push those models into the Snowpark ML registry and leverage the models for inference right with in Snowflake.

Using Snowflake with Sagemaker

The integration of Snowflake with Sagemaker is detailed in the below architecture. Most joint customers will leverage an Sagemaker notebook inside the workspace and utilize the Snowpark push-down service to prepare the data then once the data is prepared customers will decide whether they want to train model in Snowpark ML or materialized their data into Sagemaker via the Snowflake connector and train models in Sagemaker. Last, customers will either deploy models to Sagemaker Studio or push those models into the Snowpark ML registry and leverage the models for inference right with in Snowflake.

Worth noting that with Sagemaker Canvas customers can utilize the Snowflake connector to perform some pushdown functionality for data prep, but with Canvas users would have to train and deploy models in Sagemaker.

Conclusion and important points

For ML users that are using either AzureML or Sagemaker its often an easy experience to use Snowpark ML with these platforms offered by hyperscalers, Please check out this blog for more details on how to deploy models from Sagemaker or AzureML to Snowpark ML.

Additionally, here are important notes and things to consider:

• With Sagemaker customers will likely want to leverage Lifecycle Configuration Scripts to set up their Sagemaker environments for work with Snowpark. Check out this blog here.
• With AzureML users will have to build a customer Snowpark environment (as of publishing this blog) however in the coming months AzureML will offer a curate Snowpark environment to make it easy for customers to work with Snowpark in AzureML.
• When working with Snowflake + Sagemaker orAzureML customers will have to practice discipline via naming conventions to make sure training data can be appropriately mapped to models and then to deployments.
• All three platforms offer a wide array of functionality in terms of MLOps and model tracking please reference documentation or reach out to you Snowflake account team to get assistance in making these architectural decisions…its often straightforward with just a little assistance :-)
• Getting started with Snowflake and Sagemaker.
• Getting started with Snowflake and AzureML.