Snowflake DevOps with GitHub & Liquibase
In this blog we are going to discuss how we can utilize Liquibase, a Database Change Management Tool, to deploy database objects & DML scripts in Snowflake using GitHub Action CI/CD pipeline. The scope of this blog is limited to a basic DevOps setup to showcase MVP.
I had read a post from Jeremiah Hansen about the same topic, but with Azure DevOps as an automation tool. I was very impressed by Jeremiah's post and wanted to try with different technology stacks, so I had a slightly different setup but the end goal is the same, i.e. deployment in Snowflake using Database Change Management tool via automation.
What is DevOps?
DevOps is concerned with automating the development, release, and maintenance of software applications. As such, DevOps is very broad and covers the entire Software Development Life Cycle (SDLC). The landscape of software tools used to manage the entire SDLC is complex since there are many required capabilities/tools, including:
- Requirements management
- Project management (Waterfall, Agile/Scrum)
- Source code management (Version Control)
- Build management (CI/CD)
- Test management (CI/CD)
- Release management (CI/CD)
This blog will focus primarily on automated release management for Snowflake by leveraging the GitHub Actions service from GitHub. Additionally, in order to manage the database objects/changes in Snowflake, I will use the liquibase, Database Change Management (DCM) tool.
What we are going to use for our setup?
- Repositories: GitHub
- Pipeline: GitHub Action
- Database Change Management Tool: Liquibase
What is GitHub Action?
“GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want” (from GitHub’s GitHub Actions). GitHub Actions was first announced in October 2018 and has since become a popular CI/CD tool. To learn more about GitHub Actions, including migrating from other popular CI/CD tools to GitHub Actions, check out Learn GitHub Actions.
What is Liquibase ?
Please refer to my recent blogs on liquibase to get detailed insight on liquibase.
What you will need as prerequisites for this demo?
- A Snowflake account, with the user having the privilege to create objects in the schema.
- A GitHub account
- A GitHub Repository.
- Liquibase Docker, the latest setup details.
- Integrated Development Environment (IDE): Gitbash, Visual Studio Code, or any supported IDE of your choice.
How about a quick demo?
How to create a Repository in GitHub?
Go to GitHub home page and Click on Repositories> New
How we can set up a Repository?
Once the blank repository is created. Clone the repo using Gitbash CLI. Use the below command.
git clone https://github.com/<accountname>/<repositoryname>.git
Create a blank folder named dbscripts under the main repository folder. It will look like below:
Where we can save our Liquibase secrets like username, password, etc.?
This is a very important step to secure your static parameter values into GitHub secrets. This vault parameter will be passed in the CLI call. We need to set the below values. Once set, you can only update the value but never see the existing value.
You can set this up under Setting(tab)> Secrets(left side sub tab)
After setting secret, it will look like below:
How to set up CI/CD pipeline in GitHub Action?
Action Workflows represent automated pipelines, which include both build and release pipelines. They are defined as YAML files and stored in your repository in a directory called .github/workflows. In this step, we will create a deployment workflow that will run schema change and deploy changes to our Snowflake database.
Firstly, create a workflow under Action> Set up this workflow and give some names to your yml file.
Sample YML file.
Finally, click on the green Start commit button near the top right of the page and then click on the green Commit new file in the pop-up window (you can leave the default comments and commit settings). You'll now be taken to the workflow folder in your repository.
Click on the Run workflow button and then on the green Run workflow button in the pop-up window (leaving the default branch selected)
Your workflow will look like below:
Now, try to add a new DB change script under dbscripts folder. You will see that automated workflow will be trigger and deployment will happen in Snowflake.
Hope this blog helps you to get insight on Snowflake DevOps using GitHub Action & Liquibase. Feel free to ask a question in the comment section if you have any doubts regarding this. Give a clap if you like the blog. Stay connected to see many more such cool stuff. Thanks for your support.
You Can Find Me:
Follow me on Medium: https://rajivgupta780184.medium.com/
Follow me on Twitter: https://twitter.com/RAJIVGUPTA780
Connect with me in LinkedIn: https://www.linkedin.com/in/rajiv-gupta-618b0228/
Subscribe to my YouTube Channel: https://www.youtube.com/user/rajivgupta780184
#Keep learning #Keep Sharing #EverydayLearning.
