IaC 101: Building Reliable Infrastructure with the Power of Code

For many organizations, infrastructure management often presents issues due to the time and effort required to manually provision and configure resources. This can be particularly challenging in large and complex infrastructure environments, as it can be difficult to consistently and accurately set up and configure resources in a repeatable and reliable manner.

Infrastructure as code (IaC) has emerged as an effective solution for preventing headaches related to infrastructure management — and chances are, its benefits are relatively easily attainable for your organization. IaC treats infrastructure in the same way that developers treat software code: by writing it, versioning it and automating the provisioning and management of infrastructure resources.

So let’s quickly catch you up on the basics and see what tools and techniques can help you put your infrastructure to better use with less fuss.

Photo by Tobias Carlsson on Unsplash

Why Treat Your Infrastructure as Code

Automate Infrastructure Deployment

As more and more infrastructure components are deployed in the cloud, organizations are empowered to save time and reduce the risk of error by automating the deployment process. Compared to manual configuration, which can be both time-consuming and error-prone, IaC enables you to deploy infrastructure resources in a repeatable manner, with predictable and effective outcomes.

Manage Infrastructure at Scale

IaC is particularly useful for managing large and complex infrastructure environments, as it allows you to define and manage all of the resources in a single place. It also makes it easier to track changes to the infrastructure, as the scripts can be stored in version control systems like Git. This enables you to easily roll back changes if needed and collaborate with multiple team members on the same infrastructure configuration.

IaC also helps in the standardized deployment of tools within your organization. The code can include not just the core VMs but also the software installation steps.

Version Control Your Infrastructure

By storing IaC scripts in version control systems like Git and thereby enabling change tracking and rollback, you can easily reinstate or create a completely new production-like infrastructure for testing and discard it later.

Your IaC Toolbox

There are several mature tools available for managing infrastructure as code, including AWS CloudFormation, Azure Bicep, Pulumi and Terraform. Each of these tools has its strengths and weaknesses, and the best choice for your organization will depend on your specific needs and preferences. Let’s review them to see which ones might be up your alley.

AWS CloudFormation

CloudFormation is a service offered by AWS that enables you to define your infrastructure as code in JSON or YAML format. It provides a wide range of resource types that can be used to define infrastructure, including EC2 instances, VPCs and RDS databases. One potential downside, however, is that it’s tied to the AWS platform and does not support other cloud providers.

Here’s a simple example code for deploying a compute instance using CloudFormation in YAML format (Let’s name it infrastructure.yaml):

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-0ff8a91507f77f867
      InstanceType: t2.micro
      KeyName: myKeyPair
      SecurityGroupIds:
        - !Ref MySecurityGroup

Outputs:
  InstanceId:
    Description: The ID of the instance
    Value: !Ref MyInstance
    Export:
      Name: !Sub "${AWS::StackName}-instance-id"

You can deploy this YAML file using the command line interface (CLI) or the CloudFormation API.

Here’s an example for deploying while creating a new stack and then updating it:

aws cloudformation create-stack \
    --stack-name my-stack \
    --template-body file://infrastructure.yml \
    --parameters ParameterKey=KeyName,ParameterValue=myKeyPair

aws cloudformation update-stack \
    --stack-name my-stack \
    --template-body file://infrastructure.yml \
    --parameters ParameterKey=KeyName,ParameterValue=myKeyPair

Azure Bicep

Bicep is an open-source IaC tool developed by Microsoft for use with Azure resources. Like CloudFormation, you can use it to define your infrastructure in a simpler and more readable format like JSON or YAML.

See this example code for deploying a compute instance using Bicep in YAML (let’s name this one infrastructure.bicep):

resources:
  - name: MyInstance
    type: Microsoft.Compute/virtualMachines
    properties:
      location: westus
      vmSize: Standard_B1s
      osProfile:
        computerName: myVM
        adminUsername: azureuser
        adminPassword: 
      storageProfile:
        imageReference:
          publisher: MicrosoftWindowsServer
          offer: WindowsServer
          sku: 2019-Datacenter
          version: latest
      networkProfile:
        networkInterfaces:
          - id: resourceGroup().networkInterfaces[0].id

outputs:
  - name: vmId
    value: MyInstance.id

For Bicep deployment, you can use the Azure CLI or Azure PowerShell to deploy your scripts.

Below is an example for deploying while creating a new stack and then updating it:

az deployment create --location westus --template-file infrastructure.bicep

az deployment group update --resource-group my-resource-group --template-file infrastructure.bicep

Pulumi

Pulumi is a cloud-agnostic IaC tool that gives you the flexibility to use a variety of programming languages, including JavaScript, TypeScript and Python to define you infrastructure as code. It provides a wide range of resource types and supports multiple cloud platforms, including AWS, Azure and Google Cloud.

Example code for deploying a compute instance on AWS using Pulumi with Python (we’ll call it infrastructure.py):

import pulumi
import pulumi_aws as aws

security_group = aws.ec2.SecurityGroup("my-security-group", ingress=[{
    "protocol": "tcp",
    "from_port": 22,
    "to_port": 22,
    "cidr_blocks": ["0.0.0.0/0"],
}])

instance = aws.ec2.Instance("my-instance",
    instance_type="t2.micro",
    security_groups=[security_group.name],
    ami="ami-0ff8a91507f77f867",
    key_name="myKeyPair",
)

pulumi.export("instance_id", instance.id)utputs:
  - name: vmId
    value: MyInstance.id

In Pulumi, you can use the command line interface (CLI) to deploy your infrastructure.

Here’s how to deploy while creating a new stack and then updating it:

pulumi up

pulumi up --yes #the flag is optional while running update on an existing resource

Terraform

Terraform is another popular IaC tool that is cloud-agnostic and supports several cloud providers. It uses a DSL based on HashiCorp Configuration Language (HCL) to define the infrastructure and includes a wide range of resource types. Terraform has a large community of users and contributors, which can help get support and find resources.

You can simply deploy a compute instance on AWS using Terraform with the following code (which we’ll call main.tf):

resource "aws_instance" "my_instance" {
  ami           = "ami-0ff8a91507f77f867"
  instance_type = "t2.micro"
  key_name      = "myKeyPair"

  security_groups = [
    "${aws_security_group.my_security_group.name}"
  ]
}

resource "aws_security_group" "my_security_group" {
  ingress {
    protocol    = "tcp"
    from_port   = 22
    to_port     = 22
    cidr_blocks = ["0.0.0.0/0"]
  }
}

output "instance_id" {
  value = "${aws_instance.my_instance

For Terraform, you can once again use the CLI to deploy your infrastructure.

And lastly, here’s an example to deploy while creating a new stack and then update it:

terraform apply

Comparing IaC Tools

Cloud Agnosticism

Both Pulumi and Terraform are cloud agnostic, meaning they can be used to manage infrastructure on multiple cloud platforms. This can be useful for organizations that use multiple clouds or want the flexibility to switch clouds in the future. CloudFormation and Bicep, on the other hand, are tied to a specific cloud provider — AWS and Azure, respectively.

Language Support

Pulumi and Terraform support multiple programming languages, which can be beneficial for organizations that already have a preferred language for development. CloudFormation and Bicep use JSON or YAML, both commonly used for IaC, but may not be as familiar to developers who are more comfortable with a specific programming language.

Ease of Use

This can be highly subjective, as different people may have different preferences for how they like to write code. Some users may find the syntax of a particular tool more intuitive or easier to read, while others may prefer a different tool. It’s worth trying out a few different tools to see which one feels the most comfortable for you.

Secret Management

It’s important to choose the IaC tool that best meets your needs and fits with your organization’s workflow and security practices. It’s also important to follow best practices for managing secrets, such as encrypting sensitive information and using dedicated secrets management services if available.

The tools I introduced above use different ways to manage secrets.

With AWS CloudFormation, you can manage secrets using the AWS Secrets Manager service. Secrets can be stored as plaintext or encrypted and made accessible for resources within the CloudFormation stack using the !GetSecretValue function. CloudFormation also supports the use of encrypted parameters, which you can use to pass secrets to CloudFormation templates at runtime.

Azure Bicep supports the use of Azure Key Vault to manage secrets. Secrets are stored in Key Vault and accessed by resources within the Bicep deployment using the keyVaultSecret() function. Bicep also supports the use of variables, which you can use to pass secrets to Bicep templates at runtime.

Pulumi supports the use of secrets through the pulumi.Secret class. Secrets can be stored as plaintext or encrypted, and they can be made accessible for resources within the Pulumi deployment using the pulumi.getSecret function. Pulumi also supports the use of stack-level and component-level configurations, which you can use to pass secrets to Pulumi deployments at runtime. Pulumi provides a web-based UI called Pulumi Console, which is where you can manage Pulumi deployments and secrets and view real-time logs and updates.

Terraform supports the use of secrets through the terraform.tfvars file and the -var flag. Secrets can be stored as plaintext or encrypted and can be accessed by resources within the Terraform configuration using the var. prefix. Terraform also supports the use of environment variables and the terraform.tfvars.json file to pass secrets to Terraform configurations at runtime.

Resource Type Support and Functionality

Each tool supports a different set of resource types and functionality. For example, AWS CloudFormation includes support for a broad range of resources within its native platform, while Pulumi supports a more limited set of resources but is more flexible in terms of the types of operations that can be performed. The above can also be said for Azure Bicep and Terraform, respectively.

State Management

State management is an important aspect of IaC, enabling you to track the current configuration of your infrastructure and ensure that changes are applied predictably and reliably.

AWS CloudFormation stores state in a JSON-formatted file called a stack file. The stack file is stored in an S3 bucket and is updated whenever changes are made to the CloudFormation stack.

Azure Bicep stores state in Azure Resource Manager (ARM) templates. When changes are made to the Bicep deployment, the ARM templates are updated to reflect the new state of the infrastructure.

Pulumi stores state in a cloud-based backend service. When changes are made to the Pulumi deployment, the state is updated in the backend service to reflect the new configuration of the infrastructure.

Terraform stores state in a local file called the Terraform state file. The state file is updated whenever changes are made to the Terraform configuration. Terraform also supports storing state in a remote backend, which can be helpful for teams working on the same infrastructure configuration.

A Few Best Practices

• Always use version control for your IaC scripts: As stated above, this way you can track changes to your infrastructure, roll back when needed, and makes collaboration easier.
• Use modules to organize your code: Modules can help you break your infrastructure configuration into smaller, more manageable pieces, which can make it easier to write and maintain. Don’t have a single file to deploy your entire infrastructure.
• Test your IaC scripts before deploying to production: It’s important to test your IaC scripts in a staging environment before deploying them to production to make sure everything is working as expected.

If you want to find out how we at Starschema use IaC to deploy a modern data stack platform in the cloud using Pulumi and Python, leveraging leading technologies such as dbt, Airbyte, Fivetran, Dagster, Airflow, Redata, Snowflake, Databricks, Tableau, OpenMetadata and OpenLineage, get in touch — we’d love to talk to you.

About the author

Anjan Banerjee is the Field CTO of Starschema. He has extensive experience in building data orchestration pipelines, designing multiple cloud-native solutions and solving business-critical problems for multinational companies. Anjan applies the concept of infrastructure as code as a means to increase the speed, consistency, and accuracy of cloud deployments. Connect with Anjan on LinkedIn.

REACH OUT TO STARSCHEMA HERE:

Let’s talk | Starschema

READ MORE STORIES FROM STARSCHEMA:

Look Out for These Data Engineering Trends in 2023

So What’s All This Talk about Zero-ETL Integration ?

Building an Open Source Data Platform — Consider these tools!