How we sped up our AWS infrastructure testing using LocalStack

The AppsFlyer platform team builds and develops the infrastructure that serves the entire R&D organization.

With over 1 million incoming HTTP requests per second and 150 billion events per day, building such a massive infrastructure over the AWS cloud is a challenging task.

In recent years, we adopted the Infrastructure as Code (IaC) declarative approach for provisioning our infrastructure.

We use HashiCorp Terraform to define the desired state of our infrastructure, and the infrastructure code is maintained in our version control system.

Using IaC makes the development lifecycle of the infrastructure almost identical to the development lifecycle of an application code.

The gap between the two lies in the field of testing.

Why is it challenging to test cloud infrastructure ?

When you provision a cloud infrastructure on AWS, you interact with a variety of cloud services.

If you don’t have a tool that does mocking to those services you lack the ability to perform tests locally, thus forcing you to run tests directly on your cloud environment.

This results in some major pain points.

4 main pain points

Let’s say for instance that you are a developer creating a Terraform module for establishing a Kubernetes cluster (EKS) on AWS. In order to fully test the cluster, you first have to fully deploy your code on AWS. The deployment to AWS has 4 big downsides:

• Time to deploy - In the case of EKS, provisioning a cluster can take up to 20 minutes; that’s a lifetime in terms of a development lifecycle
• Cloud charges - As you know, consuming cloud resources does not come free of charge, so the testing procedure will add expenses to your monthly AWS bill
• Terraform apply errors - Terraform consists of 3 main steps- Init, Plan, Apply. While the first 2 steps are easy to verify, the last step is tricky. Some AWS or Terraform errors will only be discovered during the last phase, thus significantly slowing down the development lifecycle
• Shared resource bottleneck - Developers need to synchronize in order to use the shared cloud resources, thus slowing down development

Up until recently we had no good solutions to avoid those pain points. Fortunately we started using LocalStack.

What is LocalStack ?

LocalStack is a cloud service emulator that runs in a single container on your laptop or in your CI environment, where it provides an easy-to-use test/mocking framework for developing Cloud applications.

It does so by spinning a test environment on your local machine, and exposing the same API as the real AWS cloud environment.

A developer using LocalStack can test their entire application without having a real AWS account at all.

How did we leverage LocalStack to test our infrastructure?

The VPC use case

We created a Terraform module that contains the full configuration of a VPC, whose state consists of the following prerequisites:

• 3 subnet types
• Private subnets
• Public subnets
• Isolated subnets
• Each VPC subnet should reside in 3 different availability zones
• Tagging of VPC resources (route tables, subnets, security groups, elastic IPs, etc.) should be done according to a strict naming convention

Our goal was to deploy a VPC in a LocalStack environment, and test the VPC’s state locally before deploying it on the real AWS account.

Testing the VPC in LocalStack

Using the Terraform Apply command, we deployed the VPC to the LocalStack environment.

Since LocalStack mocks the AWS services, we could simply use the AWS CLI commands to get any VPC information we desired.

For example, using the describe-vpcs command, we could verify that the VPC was created.

aws --endpoint-url=http://localhost:4566 ec2 describe-vpcs
{
    "Subnets": [
        {
            "AvailabilityZone": "eu-west-1a",
            "AvailabilityZoneId": "euw1-az3",
            "AvailableIpAddressCount": 248,
            "CidrBlock": "10.1.2.0/24",
            "DefaultForAz": false,
            "MapPublicIpOnLaunch": false,
            "State": "available",
            "SubnetId": "subnet-3d9e840a",
            "VpcId": "vpc-02d8d5b1",
            "OwnerId": "711783529907",
            "AssignIpv6AddressOnCreation": false,
            "Ipv6CidrBlockAssociationSet": [],
            "Tags": [
                {
                    "Key": "Vpc",
                    "Value": "vpc-prd-euw1"
                },
                {
                    "Key": "appsflyer.com/access",
                    "Value": "private"
                },
                {
                    "Key": "Name",
                    "Value": "subnet-prd-eu-west-1a-private"
                }
            ],
            "SubnetArn": "arn:AWS:ec2:eu-west-1:711783529907:subnet/subnet-3d9e840a"
        },
          {
            "AvailabilityZone": "eu-west-1b",
            "AvailabilityZoneId": "euw1-az1",
            "AvailableIpAddressCount": 251,
            "CidrBlock": "10.2.4.0/24",
            "DefaultForAz": false,
            "MapPublicIpOnLaunch": false,
            "State": "available",
            "SubnetId": "subnet-efe79e26",
            "VpcId": "vpc-02d8d5b1",
            "OwnerId": "711783529907",
            "AssignIpv6AddressOnCreation": false,
            "Ipv6CidrBlockAssociationSet": [],
            "Tags": [
                {
                    "Key": "appsflyer.com/access",
                    "Value": "isolated"
                },
                {
                    "Key": "Name",
                    "Value": "subnet-prd-eu-west-1b-intra"
                },
                {
                    "Key": "Vpc",
                    "Value": "vpc-prd-euw1"
                }
            ],
            "SubnetArn": "arn:AWS:ec2:eu-west-1:711783529907:subnet/subnet-efe79e26"
        },

Using the describe-subnets command enabled us to test if the desired subnets inside the VPC were created, and that their relevant subnet tags applied to the convention we’d set.

aws --endpoint-url=http://localhost:4566 ec2 describe-subnets
{
    "Subnets": [
        {
            "AvailabilityZone": "eu-west-1a",
            "AvailabilityZoneId": "euw1-az3",
            "AvailableIpAddressCount": 248,
            "CidrBlock": "10.1.2.0/24",
            "DefaultForAz": false,
            "MapPublicIpOnLaunch": false,
            "State": "available",
            "SubnetId": "subnet-3d9e840a",
            "VpcId": "vpc-02d8d5b1",
            "OwnerId": "711783529907",
            "AssignIpv6AddressOnCreation": false,
            "Ipv6CidrBlockAssociationSet": [],
            "Tags": [
                {
                    "Key": "Vpc",
                    "Value": "vpc-prd-euw1"
                },
                {
                    "Key": "appsflyer.com/access",
                    "Value": "private"
                },
                {
                    "Key": "Name",
                    "Value": "subnet-prd-eu-west-1a-private"
                }
            ],
            "SubnetArn": "arn:AWS:ec2:eu-west-1:711783529907:subnet/subnet-3d9e840a"
        },
          {
            "AvailabilityZone": "eu-west-1b",
            "AvailabilityZoneId": "euw1-az1",
            "AvailableIpAddressCount": 251,
            "CidrBlock": "10.2.4.0/24",
            "DefaultForAz": false,
            "MapPublicIpOnLaunch": false,
            "State": "available",
            "SubnetId": "subnet-efe79e26",
            "VpcId": "vpc-02d8d5b1",
            "OwnerId": "711783529907",
            "AssignIpv6AddressOnCreation": false,
            "Ipv6CidrBlockAssociationSet": [],
            "Tags": [
                {
                    "Key": "appsflyer.com/access",
                    "Value": "isolated"
                },
                {
                    "Key": "Name",
                    "Value": "subnet-prd-eu-west-1b-intra"
                },
                {
                    "Key": "Vpc",
                    "Value": "vpc-prd-euw1"
                }
            ],
            "SubnetArn": "arn:AWS:ec2:eu-west-1:711783529907:subnet/subnet-efe79e26"
        },

Accordingly, the entire VPC state could now be tested simply by using the AWS CLI on the local developer’s laptop.

LocalStack tests integrated into the CI/CD pipelines

We created a test suite written in Golang that is now an integral part of our CI/CD pipelines.

It’s similar to application tests; if all tests pass, the tested VPC module can be safely deployed to the real AWS account.

Did we solve the 4 main pain points ?

Well, the answer is yes!

• Time to deploy — We moved from order of minutes to order of seconds!
• Cloud charges — No cloud charges, since there is no real AWS account involved
• Terraform apply errors — We can now catch Terraform errors on LocalStack and fix them quickly, therefore we rarely see Terraform errors when deploying to the real AWS account
• Shared resource bottleneck — No synchronization is needed between developers, each developer can run the test suite locally

Future thoughts

LocalStack helps us create a fully tested infrastructure and speed up the development lifecycle.

However, not all AWS services are fully supported in LocalStack, EKS is an example of such a service.

Since Kubernetes is the container orchestrator we use for deploying our services, EKS plays a vital role in our production environment.

Today we lack the ability to test EKS locally, so the tests run using a dedicated AWS account and the pain points I described in this article are something we still suffer from when testing EKS.

In the future, when the EKS service inside LocalStack gets a more mature version, we will definitely be interested in adopting it. Using LocalStack alongside tools, such as ‘kind’ for interacting with a Kubernetes cluster will give us the ability to fully test our Kubernetes clusters locally.

In terms of saving time and costs, and early bug detection, this will be a major milestone for us in the platform team.