Snowflake takes the next step in the fight with Databricks and Fabrics
Snowflake is positioning themselves closer to DBT and their big rivals Databricks and MS Fabrics by releasing three new DevOps capabilities.
Last week Snowflake announced that they have created three new DevOps capabilities in their ecosystem. These three capabilities is GIT integration, File Execution, and Target State Definitions. These three capabilities together create the possibility to work with version control and CI/CD processes for database objects in Snowflake.
The GIT-integration and File Execution is a nice addition that will make creating CI/CD pipelines to run code from GIT in Snowflake easier. This is something that can already be done today by most CI/CD pipeline tools.
What really made me excited when reading the announcement was Target State Definitions and the inclusion of the new keyword CREATE OR ALTER. This together with the above-mentioned capabilities will make developing and deploying database objects in Snowflake a lot easier.
If you want to work with version control and database objects right now there is some different approaches you can take and different tools to use. All of them have different upsides and downsides as all tools do.
I come from a classic SQL Server background, and I am used to working with the SQL Server Database projects. The upside to this approach is the ease with which you tackle the changes while developing. You only ever have to write create scripts and the database project will create the underlying alter scripts to tackle the changes needed to update the database objects without any data loss. The drawback is that handling breaking changes is a bit cumbersome and requires different manners of workarounds to be able to deploy them.
Other common tools for handling version control of database objects are Terraform and Liquibase, these two take different approaches to handling versioning of the database objects. Of these two Liquibase has a different approach compared to SQL Server Database projects and Terraform in that you write your initial create statement and then write alter statements with the changes that should be applied. Liquibase then keeps track of the changes by the versioning number that the developer adds to the file name. I have colleagues that find this approach straight forward and likes it, I on the other hand find it a bit unwieldy and hard to use.
Terraform is a lot like SQL Server Database projects in that you write create scripts. The drawback is that you have to write quite a lot of yaml and it is not really made for keeping track of states, its purpose is really to create and manage infrastructure in the cloud. You also have to be careful with how you implement it because it is quite destructive by default, and you can end up with data loss if you do not handle it with care.
Last year I was involved in a project to implement version control in Snowflake and we ended up choosing Terraform. The major reason being that I found Liquibase and similar tools too clumsy. I never really felt like it was a great fit for what we tried to use it for, but it did its job. I would have loved to have these new DevOps capabilities available though, it would have shortened the time it took for us to deploy the tooling and the solution would most likely have been smarted and easier to use.
The modern and right now most popular tool to use for version control and much more is of course DBT. For our use case we did not see the need to implement it at that point in time and I can see scenarios where you do not want to implement DBT and for those I really think that these new capabilities from Snowflake will make the life of all developers simpler and increase their throughput.
As all of you that have used these different tools surely know it’s never as easy as the backside of the box makes it out to be and it’s going to be interesting to see how these capabilities actually work in Snowflake and what the drawbacks will be. One thing I know for certain is that I am excited for the future of the data platforms and how they can be used to deliver new insights for companies.
Read more here: https://medium.com/snowflake/devops-in-snowflake-how-git-and-database-change-management-enable-a-file-based-object-lifecycle-1f61a0d5257c
