Announcing General Availability of Snowpark Python Local Testing Framework
We are thrilled to announce the General Availability (GA) release of the Snowpark Python Local Testing Framework! This framework emulates the Snowpark execution environment to allow software developers and data engineers to test Snowpark DataFrames, stored procedures, and UDFs locally to streamline the development and testing process.
Today’s GA release includes support for creating and calling stored procedures and UDFs locally, which was previously not supported. This is also not the end–we are continuing to expand the breadth of the local testing framework and address unsupported features.
“Snowpark’s Local Testing Framework has allowed us to automate the validation of transformation logic using standard DevOps tooling. This provides ongoing regression testing and greatly increases confidence levels within our production delivery pipelines.”
- Kevin Mellott, Principal Data Consultant at Continuus Technologies
Key Benefits
- Local Development: Develop and test your Snowpark Python code without the need for a Snowflake account, reducing the barrier to entry and speeding up iteration cycles.
- Efficient Testing: Utilize familiar testing frameworks such as PyTest to write unit and integration tests, making it easy to integrate into existing development workflows.
- Enhanced Productivity: Quickly iterate on your code with local feedback, allowing for rapid prototyping and troubleshooting before deploying to your Snowflake environment.
Getting Started
The following sections illustrate the basic functionality of the framework. For more details and information, please see our documentation and tutorial.
1. Setting Up the Environment
To begin, set up your Python environment by installing the snowpark and pytest packages.
pip install "snowflake-snowpark-python[localtest]"
pip install pytest
Next, create your local testing session:
from snowflake.snowpark import Session
session = Session.builder.config('local_testing', True).create()
2. Creating Local DataFrames
You can create DataFrames from local data sources, such as CSV files or Pandas DataFrames and operate on them as you normally would. In this case, we are saving the DataFrame as a table in the local, emulated environment.
table = 'example'
session.create_dataframe([[1,2],[3,4]],['a','b']).write.save_as_table(table)
You can also operate on these dataframes using the local session as you normally would:
df = session.create_dataframe([[1,2],[3,4]],['a','b'])
res = df.select(col('a')).where(col('b') > 2).collect()
print(res)
3. Creating UDFs and Stored Procedures
With today’s GA release, you can create and call UDF and stored procedures using all the same Snowpark APIs. In this case, the UDF and stored procedure are being called on the table we created in the previous step.
from snowflake.snowpark.functions import udf, sproc, call_udf, col
from snowflake.snowpark.types import IntegerType, StringType
@udf(name='example_udf', return_type=IntegerType(), input_types=[IntegerType(), IntegerType()])
def example_udf(a, b):
return a + b
@sproc(name='example_proc', return_type=IntegerType(), input_types=[StringType()])
def example_proc(session, table_name):
return session.table(table_name)\
.with_column('c', call_udf('example_udf', col('a'), col('b')))\
.count()
# Call the stored procedure by name
output = session.call('example_proc', table)
Usage with PyTest
The local testing framework can be used with PyTest fixtures so you can switch between local emulation and live connection from the command line. We have detailed a tutorial that explains how to get started, but the following steps summarize the main points:
1. Create a PyTest fixture
In your conftest.py file, create a PyTest fixture for the Session object and a PyTest option for — snowflake-session. The parameter will allow you to switch between using local emulation or connecting to a snowflake account.
# test/conftest.py
import pytest
from snowflake.snowpark.session import Session
def pytest_addoption(parser):
parser.addoption("--snowflake-session", action="store", default="live")
@pytest.fixture(scope='module')
def session(request) -> Session:
if request.config.getoption('--snowflake-session') == 'local':
return Session.builder.configs({'local_testing': True}).create()
else:
snowflake_credentials = {} # Specify Snowflake account credentials here
return Session.builder.configs(snowflake_credentials).create()
2. Use the Fixture in test cases
The PyTest fixture is named “session”, so now you can use the fixture from any test case by simply specifying a parameter in the test case handler called “session”.
from project.sproc import my_stored_proc
def test_create_fact_tables(session):
expected_output = ...
actual_output = my_stored_proc(session)
assert expected_output == actual_output
3. Run your tests
Thanks to the — snowflake-session we added in step 1, you can now run pytest and easily specify if we want to use the local testing framework’s emulated execution environment or connect to your Snowflake account.
To run the test suite locally:
pytest --snowflake-session local
To run the test suite against your Snowflake account:
pytest
See the tutorial for more information.
The Snowpark Python Local Testing Framework significantly reduces the friction in developing and testing data applications. By enabling local testing, you can develop faster, catch bugs earlier, and ensure your Snowflake deployments are robust and reliable.
For detailed documentation and examples, check out the official Snowflake Documentation. To connect with us directly, please check the latest list of existing issues or create a new bug report in our Github repository.
Happy coding!