How to Promote Change in your DataOps Project
Working in a data team is hard! Importing business data from multiple systems and transforming that data into valuable analytics-ready models can feel like a never-ending battle. In bigger teams, the problems tend to grow: every change requires more collaboration and coordination, and sources, pipelines, models and datasets can require configuration in multiple places. To manage these changes safely and securely, teams need to develop and test changes in a space separate from production.
At Matatika, we’ve found the most optimal experience for development is to use isolated developer environments — somewhere where we can test and iterate without fear of breaking things. Unfortunately, this can make it difficult reproduce issues seen in production — if you were to move your changes around manually, this alone would pretty much become a full-time job! We need a way to manage all these changes with:
- A workspace separate from the code
- Configuration that’s easy to define for development, testing or production environments
- Full change history with release and rollback
- A production-like “staging” environment that makes it easy to understand and reproduce issues
Prerequisites
This guide assumes you have the following:
Prepare your Meltano project
We are going to setup a development branch that we can promote change to a production branch (main or master) from.
If you don’t already have a development branch (e.g. dev), create a new branch through GitHub, or locally:
git checkout -b dev
git push --set-upstream origin devIf your repository is private, you will also need to install the Matatika Workspaces GitHub app to allow Matatika to access the repository.
Setup your Matatika workspaces
We are going to setup two workspaces to reference the development and production branches of your repository.
Go to the Matatika Cloud and create your production workspace — give it a name like My Project (PROD), and under the “Advanced” expandable section, provide your GitHub repository HTTPS URL and production branch name (defaults to main if not provided). Once complete, click “Continue” and wait for the workspace to complete initial setup.
Next, let’s create the development workspace — follow the same steps as before, but this time give it a name like My Project (DEV) and make sure to provide the development branch name (e.g. dev).
Deploying changes
Any changes to a repository branch can be deployed back to the referencing workspace — either manually from the Matatika Cloud workspace settings via the “Deploy” button, or by setting up a GitHub webhook for auto-deployment on push.
We do not recommend setting up a webhook for your production workspace, as you may end up with unwanted changes deployed in the event that someone pushes to the production branch unexpectedly
For your Meltano project:
- Plugin
.lockfiles under thepluginsdirectory are deployed as installed plugins - Schedules and jobs defined in the
meltano.ymlare deployed as pipelines - Plugin configurations defined in the
meltano.ymlare deployed as project components
Setup your releases
We are going to leverage GitHub pull requests to promote change between development and production workspaces as “releases”. This makes the process of change promotion collaborative, secure and traceable.
You can add as many or as few intermediate steps to be satisfied as you like (e.g. security analysis, tests, lint, reviewer approval) before the pull request can be merged, to ensure the changes pending promotion are stable and up to quality standards. Once everything is ready, merge the pull request and deploy the workspace.
Alternatively, you can skip the pull request process entirely and just merge branches directly with Git (without touching GitHub) — although, we do not recommend this
After deploying, all changes observable in the development workspace will be reflected in production. Congratulations — you’ve successfully promoted changes from development to production! It’s that easy.
