CI pipelines made simple with Rony Framework

Rony is an awesome and rapidly growing framework, developed to provide data engineers with means to easily create complete solutions for complex problems. It helps the engineer create every component of a data engineering project, from CI pipelines to airflow dags, from terraform code to etl jobs. In just a few commands you create your own, complete and fully working solution!

One of the features of this amazing tool is the ability to create deployment pipelines that work on different CI tools. This article covers some design decisions, presents the structure of the generated pipelines, as well as how to use Rony to create these pipelines.

Design Decisions

Prior to implementing the CI feature to Rony, some of common challenges faced on A3Data projects, during CI implementation, were identified:

• The process of making changes to CI can be time consuming if you have to push modifications multiple times to fix bugs;
• Installation of software and tools during CI jobs slows down the pipelines;
• Development and staging environments are not always used during deployments;
• The process of figuring out the exact permissions needed by a deployment can be time consuming;
• CI code sharing between collaborators is not efficient, as different projects might use different CI tools

To address these challenges, the following design decisions were made:

• A custom docker image, to be used on all pipelines, was deployed to DockerHub. Learn more here.
• The docker image created uses iamlive to track and display all the AWS permissions used during the scripts execution.
• All the jobs and steps of the CI pipelines were written to bash files, and theses scripts are called by the CI tools;
• A docker-compose and a Makefile were written, in order to allow local reproduction of the exact same environment the CI scripts are run on cloud CI tools;
• Development, staging and production environments were introduced to the CI pipelines. By choosing which branches to push code to, the user can activate each of the environments. Therefore, the user can select which environments to use, without making any changes to the code.

Pipelines Structure

The pipeline structure is quite simple:

• On a development branch (“dev”), any pushes or pull request will trigger a “Validate Terraform” and a “Test and Lint” job. “Validate Terraform” runs terraform validate , to verify the code contains a valid Terraform configuration. “Test and Lint” is an empty script, that should be populate by the user and other Rony modules with any testing or lint verification that by might be relevant to the project;
• On a staging branch (“staging”), any pushes or pull requests will trigger the same jobs as the development branch. Additionally, on a push or when a pull request is merged, the deployment pipeline is triggered. The deployment pipeline consists of 4 jobs. The first job (“Build and Push”) should be populated with build of docker images, pushes to ECR, creation of zip files for lambda functions, and similar tasks. The second job (“Plan”) runs terraform plan , so the user can check which resources are being created. The third job (“Apply”) runs terraform apply on the staging workspace. The last job (“Destroy”) runs terraform destroy on staging workspace, and must be triggered manually by the user;
• On the production branch (“master” or “main”) the workflow is similar to the “staging” one. The only differences are:terraform plan and terraform apply are run on the “prod” workspace; the “Apply” job must also be triggered manually by the user (so the “Plan” output can be verified).

Project directory structure

Rony’s CI module adds the following folders and files to your project folder:

• .github/workflows: Folder containing yamlfiles with Github workflows configuration;
• .gitlab-ci.yml: File containing Gitlab workflows configuration;
• ./CI/scripts: Folder containing all the shell scripts used on the workflows;
• ./CI/docker-compose.yml: Docker compose file, used to create a container to locally reproduce the environment the CI scripts are run at;
• Makefile: a makefile that helps users run the CI scripts locally, inside the container created by docker-compose, with just a command line;

How to use it?

Install Rony

First step to start making use of Rony CI pipelines, is to install Rony. You can find instructions here. If it is your first time using Rony, this article (TODO) provides a good introduction to its main functionalities.

Creating a new project

When you create a new Rony project, the CI module is automatically added to your project.

Adding pipelines to existing project

If you have an existing project, and wants to include Rony’s CI workflow to it, just run (inside your project directory) the command rony add-module CI_workflows all the files and folders will be added to your project.

Remember that your Terraform files should be located at ./terraform .

Adding custom steps to jobs scripts

If you want to add any steps to a job on the pipeline, you just need to edit its corresponding bash script inside the folder ./CI/scripts . It is that simple!

Running pipelines on Github

(TODO: vídeo mostrando pipelines rodando no Github)

Running pipelines on Gitlab

(TODO: vídeo mostrando pipelines rodando no Gitlab)

Running jobs locally

Sometimes you might need to run a job locally: to test new steps you added to it, to deploy a development environment or any other reason. You can run all of the CI jobs locally with make.

The first step for that is adding your cloud credentials to environment: We recommend doing it by using tools like AWS-vault to securely store the credentials, preferably with multi factor authentication enabled.

After that you can run the following commands:

• make tf-fmt formats Terraform files
• make tf-show shows resources present on Terraform state
• make tf-validate validates current Terraform configuration
• make build-dev Runs “Build and Push” job, in the development workspace
• make plan-dev Runs “Plan” job, in the development workspace
• make apply-dev Runs “Apply” job, in the development workspace
• make destroy-dev Runs “Destroy” job, in the development workspace
• make plan-staging Runs “Plan” job, in the staging workspace
• make apply-staging Runs “Apply” job, in the staging workspace
• make destroy-staging Runs “Destroy” job, in the staging workspace
• make plan-prod Runs “Plan” job, in the production workspace
• make apply-prod Runs “Apply” job, in the production workspace
• make destroy-prod Runs “Destroy” job, in the production workspace

When running this commands in Linux distributions, you will probably need to run them with administrator privileges. In that case, make sure to run sudo with the -E flag, so it runs with your current environment variables ( sudo -E make tf-fmt , for example)

I want to contribute to this awesome project!

If you want to contribute to Rony framework or any of its derived projects, create an issue on the github repository and let us know what you are doing, or any suggestions you might have. We will be glad to help you developing it!

TODO: We also have a Discord channel where we discuss our next steps