How to Build Applications with Django and Snowflake
An overview of why Django and Snowflake are important to each other, how to build a simple application with the new open source Django connector using data from Snowflake, and how the connector is keeping up with the latest updates.
Snowflake announced that the Snowflake Connector for Django is now updated and available in Snowflake Labs on GitHub. This is great news for everyone that has already connected Django to Snowflake— and for anyone that is about to discover the possibilities behind this.
In my 2022 talk “Why would anyone use Snowflake as a backend for Django?” I used thousands of data points to show the high demand for having Snowflake and Django together — and since then the ties have gotten stronger and healthier: The django-snowflake connector built by the original Django fellow Tim Graham was initially sponsored by Cedar Cares — and now Snowflake is sponsoring its maintenance and growth.
Let’s explore here why people want Snowflake and Django together, and how to get started.
What’s Django
My 3 sentence summary of what is Django:
“Django is a versatile Python framework widely used to power large websites and countless intranet sites. With its comprehensive tools, scalability, and emphasis on security, Django provides developers with an efficient and developer-friendly environment. Its flexibility and robustness have made it a top choice, enabling the creation of powerful web applications for various purposes.”
If you dig around Internet history you can find that Django has been powering Instagram since its beginnings, and it still is a top choice for creating small and big sites. For example, the Stack Overflow 2022 survey shows it in the top 10, with a 15% share of web frameworks. But Django goes beyond web frameworks — it can also act as a powerful API layer between your data in Snowflake and applications that need a REST API (see django-rest-framework — the top 3rd party Django package on the 2022 Django developer survey). As a REST layer, Django can be a great backend for ReactJS apps.
The Django ORM is especially interesting for web developers mobilizing data in Snowflake — if I had to summarize it (with the help of GPT):
“The Django ORM simplifies database interactions by offering a high-level API, eliminating the need for manual SQL query writing. This is great for web developers who find Python’s syntax and structure more comfortable than SQL. With Django ORM, developers can define database models as Python classes, leading to more intuitive code. The ORM’s query generation system can automatically optimize SQL queries, enhancing performance and minimizing the risk of SQL injection vulnerabilities. Additionally, Django’s MVC architecture improve code organization, maintainability, and the ability to evolve the database schema over time. The integration of Django ORM with the broader Django ecosystem provides Python-centric developers with an array of additional tools, including authentication, form handling, and an admin interface, streamlining the web development process. Ultimately, Django ORM empowers Python-loving web developers by offering a familiar and robust framework for seamless interaction with Snowflake databases.”
What about Snowflake and Streamlit?
While Django and Snowflake can be a great pairing — we shouldn’t forget about all the awesomeness that Snowflake is cooking with Streamlit and the Native Application Framework: Streamlit gives us an incredibly easy way to build interactive data apps — and the announced native integration with Snowflake means we can quickly and securely deploy these apps within the Snowflake security perimeter.
When mobilizing your data you have choices: The incredible agility of Streamlit, the versatile and powerful features of Django and its ecosystem— while sometimes just building an interactive dashboard in a traditional BI tool will do.
What can Snowflake offer to Django developers?
Traditionally Django gets paired with transactional databases like MySql and Postgres. In many use cases with massive amounts of data, developers have chosen to build pipelines that replicate data from Snowflake to one of these databases. This adds complexity, and shouldn’t be necessary if keeping this data in Snowflake can offer better performance, security, governance, timeliness, and simplicity.
As Scott Fought — who developed an internal connector within his company before django-snowflake existed — told me:
“I’m not a Django super fan. I’m obsessed with optimization. I was put on a project that used Django on PostgreSQL and they had jobs that were failing because Postgres wasn’t performant enough. We tried cranking up the instance, but the performance wasn’t there because it didn’t scale queries (single threaded). I wrote the backend as a POC to prove that even though the transaction times were slower, once the DB is doing enough work, the benefit switched to Snowflake. I hardened the backend and several teams still use it for their projects.”
Companies that have chosen Snowflake can now hire Django developers — offering them the ability of Snowflake to process incredible amounts of data; the isolation of compute; the irrelevancy of indexes; the governability of using the main data tables while keeping them protected with row-level, column-level, and data-masking policies; the richness of public and premium data on the Snowflake Marketplace; and so much more.
Also especially attractive for Django developers is the Snowflake ability to understand and run Python — Django’s language of choice. With this they can push logic and code into the database — with UDFs and stored procedures — that can process huge amounts of data without having to copy it between different platforms to get the job done.
In the near future, we can also expect the connector to support Unistore and Hybrid Tables — enabling transactional workloads without leaving the safety of a well governed database.
Enough theory — How to get started?
If all this sounds great to you, let’s get ready to code: Gilberto Hernandez already published a quickstart for Django and Snowflake.
Let’s check the basic steps:
- Install Django and django-snowflake with
pip install django django-snowflake(and let’s celebrate how the connector is now being kept in sync with the latest Django versions). - Create a user and a database in Snowflake that you would like Django to use.
- Run the Django migrations, proving that Django can create its administrative tables within Snowflake.
- Load data into Snowflake, and create Django models using
inspectdb. This is a fast way to connect your existing Snowflake tables to Django — as long as they are tables that adjust to Django’s expectations. - Browse your data with a Django view and its admin interface.
Those are the basics. It’s easy and cool. But we are only getting started.
Next steps for Snowflake Developers
If you are new to Django and excited about connecting it to Snowflake, there are interesting topic we could go deeper into in upcoming posts.
For example:
- Understand the Django basics: Controller, Model, Views.
- Adapt your existing tables to the Django ORM: The ORM has certain restrictions, like requiring all tables to have a column with a unique id. You could create views over your existing tables to do this mapping.
- Mapping the Knoema time-series in the Snowflake Marketplace to the Django ORM: A particular example of the previous point are the time-series shared by Knoema. For example, if you want to map the Knoema timeline with the historical currencies exchange rates — you’ll need to create at least 2 views (one adding a unique id column per row, and another with a list of the different currencies).
- Foreign keys in the ORM: Once you have the above two tables, the ORM can construct SQL queries leveraging the connection between them — by using the foreign keys you describe.
- Optimizing ORM calls: The ORM is powerful, and you can make the most of it by learning Django ORM optimization concepts like projections,
select_related,prefetch_related, and how to push down to Snowflake aggregations and filtering.
- Python UDFs with Django ORM: The ORM leverage your Python UDF in Snowflake by mapping them inside SQL views.
- Authentication with a private/public key: You can use the classical user/password to authenticate — but also more secure ways to connect.
- Dashboards in Django: You can use for favorite JS dashboard framework in Django — for example, Chart.js.
- Deploying Django: When moving your Django server to production, where are you going to deploy it? There are many alternatives, and as an example — with AWS Elastic Beanstalk.
- Experiment with GeoDjango: This is in my TO-DO list, as I’d love to build maps leveraging Snowflake’s and Django’s GIS abilities.
Next steps for Django Developers
If you are new to Snowflake, there’s some basic concepts that we can discuss with more depth in a future post:
- The Snowflake free trial: You don’t need a credit card, just create a new account and try all this out.
Snowflake is different:
- You won’t need table indexes — your queries should run fast without them.
- Unlimited isolated compute — your Django workloads will run in parallel, without affecting other users of the same data.
- The Snowflake Marketplace: Find ready-to-use public and premium datasets, constantly being updated and in-place.
- Role based governance: Learn how to secure your data with roles and policies like row-level, column-level, and data-masking.
- Moving Python code into Snowflake: Snowflake goes beyond SQL — as a Python developer you can bring your code inside with UDFs and Stored Procedures.
Want more?
- Try this out with a Snowflake free trial account — you only need an email address to get started.
- Give us your feedback and requests on the Snowflake Django connector GitHub tracker.
- Stay tuned for future posts on this topic — including more in depth tips with an overview of my talk at Snowflake Summit about the Django connector.
I’m Felipe Hoffa, Data Cloud Advocate for Snowflake. Thanks for joining me on this adventure. You can follow me on Twitter and LinkedIn. And subscribe to reddit.com/r/snowflake for the most interesting Snowflake news.
