Tutorial — Generating IaCs on GCP

A guide to generating IaCs for existing and fresh infrastructure on GCP

According to the people I’ve interacted with recently, IaCs for existing infrastructure have been a task for nearly all DevOps engineers in the recent sprint for some reason. Well, why not? Maintaining individual infrastructure replicability with the same benchmarked configuration and updating seamlessly is just something that cannot go wrong.

Why?

Since infrastructure modifications are versioned and vetted similarly to code, this approach encourages consistency and reliability across environments. Terraform, Google Cloud CLI, and Deployment Manager are just a few of the tools that Google Cloud Platform (GCP) offers to facilitate IaC and enable the management and automation of complex cloud infrastructure. Worry not! We’re going to discuss more, but first some knowledge exchange.

You can experience problems with repeatability, scalability, and consistency if you have already set up infrastructure in GCP without IaC. There are various advantages to switching to IaC from your current infrastructure:

• Version control: Allows you to monitor changes over time and undo them if needed.
• Consistency Across Environments: IaC guarantees that the staging, production, and development environments are all the same.
• Simplified Scaling: By modifying configuration files, infrastructure may be easily scaled and modified.
• Documentation: Your infrastructure’s documentation is created by the code itself.

How?

Although there are methods to write your own IaC for the existing infrastructure on Terraform, Pulumi or Deployment managers, that is a story for a different day. For now, we are going to delve into the toolsets available to make life a little easier and enabling us to export existing environmental configurations as IaC files.

Note: Personally I would always prefer to not solely depend on toolsets and generators for such tasks but use them as a baseline for the final scripts and documents. Tweak as per your needs, verify IaC code and security controls and form a solid end product.

Let’s get started!

Terraformer

You can create Terraform configuration files (templates) from an existing GCP infrastructure using Terraformer, an open-source tool from Google. Multiple resources are supported, and Terraform may be used to manage the infrastructure as code using the created files. Terraformer allows incremental imports and may be used locally, so you can create configuration files for particular resources only.

This can help you:

• Create tf/json + tfstate files for each supported object by resource using the infrastructure that is already in place.
• Upload remote state to a GCS bucket. Terraform_remote_state allows you to connect between resources (local and bucket).
• Use a custom folder tree design to save tf/json files.
• Import by type and resource name.

Download Terraformer from its GitHub releases and set up Terraform and authenticate with your GCP account using the Google Cloud SDK (gcloud init). Following which you can run the below command to generate your terraform files.

terraformer import google --resources=compute,sql,storage --zone=us-central1-a --projects=your-gcp-project

Now, as mentioned earlier, verify configurations, especially sensitive data (which should be stored securely) and organize the code for reusability (e.g., creating Terraform modules). Following which I would highly suggest you to Run terraform plan to preview changes, followed by terraform apply in a test environment to ensure configurations match existing infrastructure. Voila! We are done!

Google Cloud Config Connector

Config Connector enables you to configure GCP resources as Kubernetes Custom Resource Definitions (CRDs), thereby bridging the gap between GCP services and Kubernetes environments.

Config Connector extends Kubernetes to manage GCP resources declaratively. Using Config Connector, you can specify GCP resources in YAML manifests, similar to other Kubernetes resources, making them manageable with Kubernetes tooling and workflows.

Now this might not be IaC by definition but surely a great method to maintain infrastructure by:

• Unified Management: Control both application and infrastructure elements within a single Kubernetes cluster.
• Consistency: Declarative configurations guarantee that resources continue to meet the required standards.
• Automation: As part of your Kubernetes deployments, automate the creation, deletion, and updating of resources.

You can check for all resources having the capability to be managed by this engine by running

kubectl get crds --selector cnrm.cloud.google.com/managed-by-kcc=true

After the initial installation of Config connector following the steps described here

Keep in mind that when you specify where to build your resources, you added an annotation to the same namespace that you defined as your default.
Config Connector enables service APIs via the Service Usage API. You must activate the Service Usage API to carry out these actions. The Google Cloud CLI can be used to enable this API:

gcloud services enable serviceusage.googleapis.com

For example, creating a storage bucket on GCP would require us to configure the following YAML file:

apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  name: my-gcs-bucket
  annotations:
    cnrm.cloud.google.com/project-id: "YOUR_PROJECT_ID"
spec:
  location: US
  storageClass: STANDARD
  versioning:
    enabled: true
  lifecycleRule:
    - action:
        type: Delete
      condition:
        age: 30
  cors:
    - origin: ["https://example.com"]
      method: ["GET", "HEAD", "DELETE"]
      responseHeader: ["Content-Type"]
      maxAgeSeconds: 3600
  labels:
    environment: dev
    team: imrans-team

Simply applying the configuration using

kubectl apply -f my-gcs-bucket.yaml

Deleting is as simple as deleting this resource, now this might be an amazing approach for a GCP native environment. Find more details on monitoring resources, importing and exporting configuration on the official documentation.

Coming back to exporting configurations to IaC GCP Config connector allows bulk exports for projects which is capable of cloning environment configurations to a linux based local machine. This would require you to first configure a terraform backend remotely storing the state to a GCS bucket.

terraform {
  backend "gcs" {
    bucket  = "BUCKET NAME"
    prefix  = "PATH/TO/STATE/FILE"
  }
}

Followed by exporting the configuration with:

gcloud beta resource-config bulk-export \
 --path=OUTPUT_DIRECTORY \
 --project=PROJECT_ID \
 --resource-format=terraform

Caveats: Although the Terraform Google provider supports certain resource types, they are not supported for export to Terraform format. Use the gcloud beta resource-config list-resource-types command to get a list of resource types that can be exported to Terraform format.

Check out the official documentation for prerequisites and setup.

Google Cloud Deployment Manager

Although it works best when resources are created from scratch rather than from an existing deployment, Google Cloud Deployment Manager can assist in creating IaC templates in YAML or Python. However, if you have information about the infrastructure, you can use Deployment Manager templates to duplicate an environment.

Starting off with creating the configuration YAML file

resources:
  - name: my-gcs-bucket
    type: storage.v1.bucket
    properties:
      name: "my-unique-bucket-name"
      location: "US"
      storageClass: "STANDARD"
      versioning:
        enabled: true
      labels:
        environment: "development"
        owner: "team-name"

Followed by deploying the configuration by running

gcloud deployment-manager deployments create my-gcs-bucket-deployment --config bucket-config.yaml

There you go, as simple as that. We can further verify our deployment by running

gcloud deployment-manager deployments describe my-gcs-bucket-deployment

Or update it by running

gcloud deployment-manager deployments update my-gcs-bucket-deployment --config bucket-config.yaml

Get more details on the official getting started guide.

To Conclude

Your cloud management capabilities can be significantly enhanced by moving your current infrastructure to IaC on GCP. Terraform, Terraformer, and the GCP CLI are among the tools that can help you organize, control, and replicate your resources within an IaC framework. You will quickly have a simplified, automated infrastructure management process on GCP if you start small and follow best practices.

Remember, validate before you deploy and secure your IaC files. (Will be discussing this in an upcoming blog)

Connect with me

imranfosec | Instagram | Linktree