Announcing the GA release of Snowflake Python Connector with nanoarrow!

Anurag Gupta Avani Chandrasekaran Adam Ling Dewey Dunnington Matt Topol Sri Nadukudy

Today, we’re thrilled to announce the GA release of the nanoarrow-based Snowflake Connector for Python v3.5.0, available to all Snowflake Customers. The Snowflake Connector for Python is one of the most popular Snowflake drivers and a critical part of the Snowflake Python Ecosystem across Snowpark, Snowpark ML, Python APIs, Cortex, Streamlit in Snowflake, and innumerable Snowflake partner tools & integrations. With millions of weekly downloads, it powers critical Snowflake customer integrations and pipelines. This is a huge release for Python Connector, removing two biggest pain points of the connector for Python developers.

• Reduced the wheel size to ~1MB and installation size to ~5MB from 50MB ( 90% reduction)
• Removed a hard dependency on a specific version of pyarrow.

This Python connector release is based on the nanoarrow 0.3 release and uses nanoarrow to convert Snowflake’s Arrow-based wire protocol into user-friendly Python objects. Snowflake customers can now build Snowflake-native applications using the connector in serverless environments like AWS Lambda without any friction.

Earlier in July 2023, we published a preview release of the nanoarrow-based connector version 3.1.0a1. The preview release of nanoarrow-based Snowflake Connector for Python gathered a lot of strong interest from the Snowflake developer community and very positive feedback. There were ~8k downloads of the alpha package per day after the release. Since then, we have updated the connector to:

• version 3.1.0a2 in August 2023, which was built on top of 3.1.0a1 and including all the latest changes in the released GA version 3.1.0
• version 3.3.0b1 in September 2023, which was built on top of v3.1.0.a2 and including all the latest changes in the released GA version 3.2.0
• version 3.3.0 that uses pure nanoarrow by default for result Arrow data conversion along with an option that allows switching between the nanoarrow and the arrow converters

Since we replaced Arrow C++ with nanoarrow, we completed performance benchmarking and stress testing of the connector to validate there were no regressions. During this process, we worked with nanoarrow maintainers to improve error handling and align the error experience of handling corrupted arrow data with the GA versions. Since the GA version 3.3.0 release, we have been monitoring the stability and performance with no issues to report. We have also added general improvements and bug fixes in the GA version 3.4.0, which was released earlier this November.

The nanoarrow 0.3 release focused on rounding out existing features and improving documentation based on the Snowflake developer’s experience implementing the connector. For more information on nanoarrow including a full changelog of recent improvements, see the nanoarrow documentation.

Upcoming plans

We plan to add to our connector performance framework additional unit-level tests for nanoarrow use cases to further track stability and ongoing performance. Our goal is to provide a seamless experience and help achieve the best performance for our users.

nanoarrow is still a relatively new library that is under active development, and the community is working towards the next release. We consider nanoarrow 0.3.0 release as GA. Here is the current roadmap for the project.

Snowflake and Voltron Data teams plan to continue working together on new improvements, help prioritize bug and security fixes, and track API changes in new nanoarrow versions and review for new opportunities where we think nanoarrow can benefit our customers. Voltron Data helps enterprises design and build composable data systems with open standards.

Get started today!

We invite you to try out the GA release of nanoarrow-based Snowflake Connector for Python, and we would love to hear your feedback at developers@snowflake.com

The nanoarrow-based Snowflake Connector for Python can be installed from pypi with the command:

pip install snowflake-connector-python

Also do check out our Quickstart on How to build Custom APIs with Python Connector .

Additional resources

For more information on Snowflake Connector for Python, please visit:

https://docs.snowflake.com/en/developer-guide/python-connector/python-connector

For more information on nanoarrow, please see the documentation: https://apache.github.io/arrow-nanoarrow/dev/index.html or reach out via Github: https://github.com/apache/arrow-nanoarrow/tree/main