Designing Future-Proof Infrastructure as Code Repositories

When organizing your company’s repository structure, whether starting from scratch or refactoring existing repos, having a proper folder structure is crucial for maintainability and future-proofing.

In this post, we share our opinionated best practices based on our experience working with customers of different sizes and organizational shapes.

General Terraform and CI/CD Best Practices (For Multiple Repos/Folders)

If the CI/CD tool easily allows path-based pipeline triggers, a single repository can be used. If path-based triggers are difficult to set up, multiple repositories are preferred. Access controls also play a role in the decision. If granular access controls are required, multiple repositories are recommended. If granular access is not necessary, a single repository can be used.

Keep it simple

• Avoid making your Terraform code a software engineering project.
• Avoid creating a single custom module that deploys everything you need for an environment or application.

Sometimes, being verbose and copy-pasting code is better than creating complex modules with multiple options.

Use Official Modules

• Try to use official modules whenever possible and always remember to pin a version (do not use the master or latest since it may will break in the future)
• Avoid using a single pipeline that deploys everything. Hey, pipelines are free! Create separate pipelines for different components to ensure customization doesn’t break the entire deployment process.

Leverage on Identity Federation

• Use Identity Federation (ie: GitHub apps) to “link” your CI/CD pipelines with identities in the cloud provider, instead of service account keys in your CI/CD pipelines.

It is important to remember that the code will grow in time, people will move in and out of your team, and you will need to onboard new engineers. Having a complex Terraform or pipeline structure will delay things, and it is prone to errors.

Application Infra? Platform Infra? We’ve Got You Covered

For application-related infrastructure, a repository per application is suggested. For platform-related infrastructure, the decision is based on infrastructure categories. Separate repositories are recommended for IAM, cloud organizational structure, security, networking, shared compute, and shared data.

It’s essential to separate application-related infrastructure (e.g. messaging topics, databases, storage buckets) and platform-related infrastructure (e.g. networks, DNS, shared-compute (Kubernetes), IAM). This separation allows for better access control and permissions over the repositories.

• Application-related infrastructure:

It includes all infrastructure tightly tied to the lifecycle of an application (or microservice).

• Platform-related infrastructure:

It includes the “more static” foundational infrastructure, which is shared among all your apps in your cloud.

You may want to give the application developers the capability to deploy a new microservice with a new database, without needing to mess around with the repo that handles the networking stuff for the entire org.

Recommendations for Application-Related Infrastructure

• Keep your application code and the application-related infrastructure in the same repo
• Use CI/CD tools to trigger different pipelines based on the folder structure and which files were changed on the latest push

Example application repo:

• If you are using GKE, config-connector is a good way to deploy infrastructure using YAML files instead of Terraform. It provides a more consistent and developer-friendly structure, however, do not use it for “heavy” infrastructure since it can bring problems.
  Our recommendation is to use it for lighter (and application lifecycle-tied) components, like PubSub Topics, Buckets, Service Accounts…

Recommendations for Platform-Related Infrastructure

Platform-related infra includes the following categories:

• IAM (who can access what)
• Cloud organizational structure (folders, groups)
• Security (policies, scanners, rules)
• Networking (VPC, DNS, VPN)
• Shared Compute (multi-tenant GKE)
• Shared Data (databases, Big Query, ML)

Each category listed above must have its own Terraform state and deployment pipelines since their lifecycles are different and possibly permissions needed are different as well.

You need to decide whether to use a single repository for all your platform infra or multiple repositories. Having a single repository with folders per each infrastructure category helps with simplicity and less overhead, but then it becomes harder to control who can access the repo and reduce the blast radius if something goes wrong. A good thing is that you can always start with a single repo and separate it by creating new ones along the way.

So, what will you choose? There is no correct answer because it depends on how your company works, team sizes, etc..

The following questions will help you decide:

Infrastructure scale

For small-scale projects, starting with a single repository is recommended. For large-scale projects, the decision depends on the team shape and size. If there is a single dedicated infrastructure team, a single repository can be considered. If there are multiple dedicated infrastructure teams, multiple repositories are recommended.

• How big is the project? A single repository is the best fit when you are starting since it allows easy visibility, discoverability, and onboarding.

Team shape/size, access controls

• Do you have a single dedicated infrastructure team?
• The recommendation is to start with a single repository and separate if needed.
• Do you have multiple dedicated infrastructure teams?
• It is common for larger companies to have a dedicated networking team, IAM team, etc. In this case, I recommend going with multiple repositories, that way, each team can control/own its repo.

CI/CD Integrations and permissions

• Does your current CI/CD tool easily allow different pipelines to be triggered based on paths? If you can’t trigger different pipelines, then you need multiple repositories.
• Does your current setup allow you to use different credentials or service accounts per folder? You need multiple repositories if the credentials are tied to a repo and want to use different service accounts per type of deployment.

Single repo example:

infra-iam
├── iam.tf
├── service-accounts.tf
├── iam.auto.tfvars
├── modules/
├── README.md
└── .gitignore

Multiple repo example:

my-company-infra
├── iam/
│   ├── iam.tf
│   ├── service-accounts.tf
│   └── iam.auto.tfvars
├── org/
│   ├── folders.tf
│   ├── projects.tf
│   └── org.auto.tfvars
├── modules/
├── README.md
└── .gitignore

Managing Dependencies Between the Different States

Avoid using complex CI/CD tooling to share Terraform outputs. While remote Terraform states can be used, they can lead to cascading problems.

1. Simply write the required values in the Terraform code. ( in .tfvars)

If you created a platform-wide VPC in the networking repo/folder, and you are now deploying, a Kubernetes cluster: you should use a variable in the network field of the Kubernetes cluster, but then simply put the VPC name as a string in the .tfvars file!

Managing Different Environments (Dev/Staging/Prod)

1. Create folders for each environment with repeated code.
2. Each environment has its own Terraform state

infra-iam
├── environments/
    ├── dev/
        ├── service-accounts.tf
        ├── iam.auto.tfvars
        └── iam.tf
    ├── prod/
        ├── service-accounts.tf
        ├── iam.auto.tfvars
        └── iam.tf
├── modules/
├── README.md
└── .gitignore

1. Avoid having huge Terraform state files. It is not future-friendly. After a few years, you will find yourself having to move resources in the tfstate, and believe me, you don’t want to deal with that.

Key Takeaways

Repository approach

Designing future-proof Infrastructure as Code repositories requires careful consideration and planning to achieve maintainability and scalability and be compatible with your team dynamics.

In this post, we emphasize:

• Simplicity: Avoid complex terraform modules and CI/CD pipelines.
• Separation of Concerns: Keep platform and application-specific infrastructure separated.
• Terraform State Management: Keep dependencies within the code itself, avoiding the use of complex CI/CD tools to manage them.
• Environment Management: Use separate folders per environment, with repeated code.

The ideal repository structure depends on the company size, team structure, and automation capabilities.

By following the best practices covered in this post, you can create future-proof Infrastructure as Code repositories that will facilitate efficient development, collaboration and scalability.