Snowflake’s New Python API Empowers Data Engineers to Build Modern Data Pipelines with Ease
In today’s data-driven world, developer productivity is essential for organizations to build effective and reliable products, accelerate time to value, and fuel ongoing innovation. To deliver on these goals, developers must have the ability to manipulate and analyze information efficiently.
Yet while SQL applications have long served as the gateway to access and manage data, Python has become the language of choice for most data teams, creating a disconnect. Recognizing this shift, Snowflake is taking a Python-first approach to bridge the gap and help users leverage the power of both worlds.
Our previous Python connector API, primarily available for those who need to run SQL via a Python script, enabled a connection to Snowflake from Python applications. This traditional SQL-centric approach often challenged data engineers working in a Python environment, requiring context-switching and limiting the full potential of Python’s rich libraries and frameworks. Since the previous Python connector API mostly communicated via SQL, it also hindered the ability to manage Snowflake objects natively in Python, restricting data pipeline efficiency and the ability to complete complex tasks.
Snowflake’s new Python API (in public preview) marks a significant leap forward, offering a more streamlined, powerful solution for using Python within your data pipelines — and furthering our vision to empower all developers, regardless of experience, with a user-friendly and approachable platform.
A New Era: Introducing Snowflake’s Python API
With the new Snowflake Python API, readily available through pip install snowflake, developers no longer need to juggle between languages or grapple with cumbersome syntax. They can effortlessly leverage the power of Python for a seamless, unified experience across all Snowflake workloads encompassing data engineering, Snowpark, machine learning and application development. This API is a testament to Snowflake’s commitment to a Python-first approach, offering a plethora of features designed to streamline workflows and enhance developer productivity.
Key benefits of the new Snowflake Python API include:
- Simplified syntax and intuitive API design: Featuring a Pythonic design, the API is built on the foundation of REST APIs, which are known for their clarity and ease of use. This allows developers to interact with Snowflake objects naturally and efficiently, minimizing the learning curve and reducing development time.
- Rich functionality and support for advanced operations: The API goes beyond basic operations, offering comprehensive functionality for managing various Snowflake resources and performing complex tasks within your Python environment. This empowers developers to maximize the full potential of Snowflake through intuitive REST API calls.
- Enhanced performance and improved scalability: Designed with performance in mind, the API leverages the inherent scalability of REST APIs, ensuring efficient data handling and seamless scaling to meet your growing data needs. This ensures your applications can handle large data sets and complex workflows efficiently.
- Streamlined integration with existing tools and frameworks: The API seamlessly integrates with popular Python data science libraries and frameworks, enabling developers to leverage their existing skill sets and workflows effectively. This integration allows developers to combine the power of Python libraries with the capabilities of Snowflake through familiar REST API structures.
By prioritizing the developer experience and offering a comprehensive, user-friendly solution, Snowflake’s new Python API paves the way for a more efficient, productive and data-driven future.
Snowflake Python API package structure unifying connector, core , Snowpark, and ML sub-packages under Snowflake
How to use Snowflake Python APIs?
Authentication
snowflake.core
represents the entry point to the core Snowflake Python APIs that manage Snowflake objects. To use the Snowflake Python API, you'll follow a common pattern:
- Establish a session using Snowpark or a Python connector connection, representing your connection to Snowflake.
- Import and instantiate the
Root
class fromsnowflake.core
, and pass in the Snowpark session object as an argument. You'll use the resultingRoot
object to use the rest of the methods and types in the Snowflake Python API.
from snowflake.snowpark import Session
from snowflake.core import Root
connection_params = {
"account": "ACCOUNT-IDENTIFIER",
"user": "USERNAME",
"password": "PASSWORD"
}
session = Session.builder.configs(connection_params).create()
root = Root(session)
Create a database, schema, and table
database = root.databases.create(Database(name="PYTHON_API_DB"), mode="orreplace")
schema = database.schemas.create(Schema(name="PYTHON_API_SCHEMA"), mode="orreplace")
table = schema.tables.create(Table(
name="PYTHON_API_TABLE",
columns=[TableColumn("TEMPERATURE", "int", nullable=False), TableColumn("LOCATION", "string")],
),
mode="orreplace")
Manage Tasks
- Use Task APIs
tasks = root.databases["MYDB"].schemas["MYSCHEMA"].tasks
# Create a serverless task that uses a SQL statement.
task1_def = Task(
"MYTASK1",
definition="MERGE INTO target USING source_stream on target.k = source_stream.k WHEN MATCHED THEN UPDATE SET target.v = source_stream.v",
schedule=timedelta(minutes=1)
)
task1 = tasks.create(task1_def)
def dosomething(session: Session) -> str:
df = session.table("target")
df.group_by("a").agg(sum_("b")).save_as_table("agg_table")
return "Success"
# create a user-managed task from a Python function.
task2_def = Task(
"MYTASK2",
definition=StoredProcedureCall(dosomething, stage_location="@mystage", packages=["snowflake-snowpark-python"]),
warehouse="test_warehouse"
)
task2_def.predecessors = ["MYDB.MYSCHEMA.MYTASK1"]
task2 = tasks.create(task2_def)
# resume the tasks
task1.resume()
task2.resume()
# suspend them
task1.suspend()
task2.suspend()
# delete them
task2.delete()
task1.delete()
- Use the DAG APIs
dag = DAG("my_dag", schedule=timedelta(days=1))
with dag:
dag_task1 = DAGTask("dagtask1", "MERGE INTO target USING source_stream WHEN MATCHED THEN UPDATE SET target.v = source_stream.v")
dag_task2 = DAGTask(
StoredProcedureCall(
dosomething, stage_location="@mystage",
packages=["snowflake-snowpark-python"]
),
warehouse="test_warehouse")
)
dag_task2 >> dag_task1 # task1 is a predecessor of task2
schema = root.databases["MYDB"].schemas["MYSCHEMA"]
dag_op = DAGOperation(schema)
dag_op.deploy(dag)
Managing compute pools
new_compute_pool_def = ComputePool(
name="MyComputePool"
instance_family="CPU_X64_XS",
min_nodes=1,
max_nodes=2,
)
new_compute_pool = api_root.compute_pools.create(new_compute_pool_def)
Managing container services
from textwrap import dedent
from io import BytesIO
from snowflake.core.service import Service
session.use_role("my_role") # Perhaps this role has permission to create and run a service
specification = dedent(f"""\
spec:
containers:
- name: web-server
image: {image_repository.fetch().repository_url}/nginx:latest
endpoints:
- name: ui
port: 80
public: true
""")
service_def = Service(
name="MyService",
compute_pool="MyComputePool",
spec=ServiceSpecInlineText(specification),
min_instances=1,
max_instances=1,
)
nginx_service = schema.services.create(service_def)Getting Started with the Snowflake Python API
Getting Started with the Snowflake Python API
Install it today by simply using pip install snowflake
Check out our Python API Quickstart, Developer Guide, and API Docs to get started today.
Our Quickstart guide makes it easy to see how the Snowflake Python API can manage Snowflake objects like Task/DAG , WH , Databases , Table , Schema , Snowpark container etc. The API allows you to create, delete, and modify tables, schemas, warehouses, tasks and much more. In this Quickstart, you’ll learn how to perform key actions — from installing the Snowflake Python API to retrieving object data and managing Snowpark Container Services.
Dive in to experience how the enhanced Python API streamlines your data workflows and unlocks the full potential of Python within Snowflake. To get started, explore the comprehensive API documentation, which will guide you through every step.
We recommend that Python developers prioritize the new API for data engineering tasks since it offers a more intuitive and efficient approach compared to the legacy SQL connector. While the Python API connector remains available for specific SQL use cases, the new API is designed to be your go-to solution. By general availability, we aim to achieve feature parity, empowering you to complete 100% of your data engineering tasks entirely through Python. This means you’ll only need to use SQL commands if you truly prefer them or for rare unsupported functionalities.
The New Wave of Native DevOps on Snowflake
The Snowflake Python API release is among a series of native DevOps tools becoming available on the Snowflake platform — all of which aim to empower developers of every experience level with a user-friendly and approachable platform.
These benefits extend far beyond the developer team. The 2023 Accelerate State of DevOps Report, the annual report from Google Cloud’s DevOps Research and Assessment (DORA) team, reveals that a focus on user-centricity around the developer experience leads to a 40% increase in organizational performance.
With intuitive tools for data engineers, data scientists, and even citizen developers, Snowflake strives to enhance these advantages by fostering collaboration across your data and delivery teams. By offering the flexibility and control needed to build unique applications, Snowflake aims to become your one-stop shop for data — minimizing reliance on third-party tools for core development lifecycle use cases and ultimately reducing your total cost of ownership.
We’re excited to share more innovations soon, making data even more accessible for all. For a deeper dive into Snowflake’s Python API and other native Snowflake DevOps features, register for the Snowflake Data Cloud Summit 2024. Or, experience these features firsthand at our free Dev Day event on June 6th in the Demo Zone.