Journey to GitOps, Part 2: Infrastructure

By Vadim Gedz
Lead DevOps Engineer, Dyninno Technologies

Welcome to the second part of the “Journey to GitOps” article. Assuming that you are already familiar with the setup we are building, here are two key points to repeat:

1. We are using Kubernetes to host everything.
2. We are utilizing the GitOps approach with ArgoCD to manage everything in Kubernetes.

In this article, I will focus on the provisioning aspect of our journey. While I won’t go into what Infrastructure as Code (IaC) is, since there are already many articles on the topic, I will delve into our specific use case.

What to use?

When it comes to the decision on which IaC solution to use, we’ve opted to use Terraform for all our infrastructure needs rather than CloudFormation or other alternatives, despite most of our infrastructure residing in AWS. The advantage of using Terraform is that we can have a unified solution across all environments.

Terraform lets us focus on mastering one tool and seamlessly switch providers when needed. Furthermore, we can create our own integrations in cases where a particular integration is missing. For instance, we had to nearly rewrite the entire Dkron provider to meet our needs.

The problem

You may wonder about the best approach when you have to manage access for a lot of operations teams with different roles and access rights to the infrastructure. There are a few options available to address this issue. For example, we can consider splitting terraform code into multiple projects, allowing specific team members access only to the relevant part of the infrastructure. Alternatively, we can create per-user or per-group IAM roles to ensure secure access to the infrastructure.

Instead of providing direct or limited access to the cloud for everyone on the team, it might be worthwhile to find out other methods to enforce IaC while ensuring security. We had to explore these options and their potential benefits. Additionally, we should take steps to guarantee that all changes made during local work in the Terraform code are committed back to the git repository.

First steps

After brainstorming ways to improve our Terraform workflow, we decided to run Terraform from the GitLab-ci pipeline. This approach provided an immediate improvement on our previous method of running Terraform locally. For one, we no longer needed to provide everyone with credentials for cloud access. This helped to ensure that the state defined in Git matched the state in the cloud (at least to a certain degree). Of course, manual changes made through click ops are not currently covered by this approach.

We weren’t completely satisfied with this solution, as we felt there was room for improvement. First, we found it inconvenient to review pipeline job logs to determine what changes were being made. This process was time-consuming, particularly when making multiple changes throughout the day.

Second, it requires a pipeline configuration that can’t be called user-friendly. For example, one of template jobs looks like this:

.plan:
  stage: plan
  rules:
    # run on merge requests, when TF folder changes
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes:
        - $TF_ROOT/*
    # run on default branch, when TF folder changes
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
        - $TF_ROOT/*
  script:
    # run terraform
    - cd ${TF_ROOT}
    - terraform init -input=false -reconfigure
    # create plan for apply
    - terraform plan -input=false -out="${PLAN_CACHE}"
    # print plan for gitlab reports
    - terraform show -json "${PLAN_CACHE}" | jq -r "${JQ_PLAN}" > "${PLAN_JSON}"
  artifacts:
    paths:
      - ${TF_ROOT}/${PLAN_CACHE}
    reports:
      terraform: ${TF_ROOT}/${PLAN_JSON}

This configuration is not a rocket science, but still, we prefer to have something simpler.

Third, we needed to pass the Terraform cache between init/plan/apply jobs, which required yet another configuration we wanted to avoid. Additionally, as we are using ephemeral build pods we had to provision external cache storage that required yet another configuration.

As more teams adopted this setup, we received not very positive feedback. Hence, we started looking for a different approach.

Current Setup

Following our latest brainstorming session, we decided to experiment with Atlantis (https://www.runatlantis.io/) as an alternative solution for managing Terraform interactions.

Like the other solution we discussed, Atlantis eliminates the need for per-user credentials when executing Terraform commands remotely. However, Atlantis sets itself apart by posting plan and apply results as comments in the relevant Merge Request. This convenient feature allows team members to easily review and manage Terraform interactions without needing to sift through lengthy logs.

version: 3
projects:
  - dir: terraform/provision
    terraform_version: v1.2.0
    workflow: terraform
    autoplan:
      enabled: true
      when_modified: ["**/*.tf", "**/*.tfvars"]

In short, Atlantis offers an efficient and user-friendly approach to Terraform interactions, enabling team members to easily manage infrastructure by writing comments in the Merge Request. And the only additional configuration required looks like this:

The workflow diagrams

What did we gain from it?

Here are some of the benefits we’ve experienced with the adoption of Infrastructure as Code (IaC) through Terraform and Atlantis:

1. Providing direct access to the cloud to every team member is no longer necessary, thereby enhancing security and reducing complexity.
2. All changes are easily traceable as they’re managed through IaC with Terraform, and the entire history is available in version control, ensuring better governance and compliance.
3. We’ve established a standard approach that can be used across all our projects, considerably reducing the time needed to onboard people who switch projects.

By utilizing Terraform and IaC in general, we’ve streamlined our infrastructure management, improved governance, and reduced the potential for human error.

Vadim Gedz, Lead DevOps Engineer, Dyninno Technologies
https://github.com/shini4i