On-demand QA environments with AWS Fargate

Published in

Wonder Engineering

7 min readAug 6, 2018

A big part of building product is making sure the app does what you want it to. At Wonder, we do this with unit and integration tests covering our code, as well as manually testing the app in a staging environment.

When our product and engineering teams were relatively small, using a single staging environment worked well. Engineers could easily synchronize what code was being deployed and deploys would rarely cause conflicts as the feature set was manageable. In turn, anyone could easily visit staging to ensure that everything worked as expected.

As our teams and product scaled however, managing a single staging environment became increasingly complex. To solve this, we redesigned our QA process to make it easy for engineers to create feature-specific disposable testing environments.

The end result is wonqa, a library which creates SSL-enabled testing environments on the fly. You can start using it now through npm, or read on to learn more about how it came to be.

Problem(s) with a single staging environment

Single staging environments work well when a team is tiny, but become the source of inefficiencies and bottlenecks as a team grows.

At Wonder, we noticed that having a single staging environment forced every engineer to have to synchronize deploys. Before pushing code, engineers had to ensure that staging was available and that their new commits wouldn’t conflict with the staging environment’s current state. Failure to do so could result in comprising the QA process of a colleague’s feature. When conflicts arose between features, engineers had to spend time resetting the staging environment to a given state.

more dependencies = more uncertainty = more friction

Our single staging environment also made the day-to-day workflow of product and QA teams more complicated as the root cause of a bug on that environment could not be be found quickly. Was a bug due to the feature being tested or was it caused by a different commit coming from a different branch that happened to be deployed at the same time? This uncertainty led to unnecessary back-and-forths between product and engineering and, ultimately, time lost.

Solution

Our ideal solution was to have a testing environment per branch. That way, engineers could deploy at will as the environment was dedicated to the current feature, and we could QA features efficiently.

This idea isn’t new, but making it work seamlessly given the intricacies of a particular stack can be challenging. For example, we initially looked into using existing tools such as Heroku’s Review Apps which was created for this purpose. Unfortunately, we manage multiple apps that need to be deployed in tandem for the Wonder site to work and Heroku’s product is designed to work with a single app. Ultimately, we decided to build a tool in-house and open source it for anyone to use.

When thinking of an ideal solution, a few priorities stood out:

each environment should be isolated to a given branch and be hosted on a user-friendly URL such as <branchName>.<domain> to make sharing easy.
environments should mimic production as closely as possible — for example by using HTTPS.
environments should be quick to spin up and tear down to make them both easy-to-use and cost-effective, and managing them should be done automatically (when PRs are created or closed, for example)

an actual test environment created in minutes using wonqa

To make this work, various puzzle pieces need to come together.

1. Docker

First we need an easy way to package the code of a given branch so that it can be deployed on the fly as one or many containers.

Here, we used Docker to turn a given branch into an image.

Create Dockerfile(s) to containerize your app code

# myApp/DockerfileFROM node:8.11
COPY . ./
RUN npm install && npm run build
EXPOSE 3000
CMD [ “npm”, “run”, “start” ]

2. Use these Dockerfiles to build images

$ docker build .

2. Configure HTTPS

Second, we need a way to generate SSL certificates so that the environment can run HTTPS and mimic production.

Here, we used LetsEncrypt and Certbot to generate certificates on the fly. We used Certbot’s DNSimple plugin as it’s also our DNS provider but Certbot has plugins for other DNS providers as well.

We also needed to make sure that the containers could consume HTTPS. We placed nginx in-front of the containers running our app code to proxy incoming HTTPS traffic from the app’s entry-point to the various subdomains and ports used by the app:

Generate SSL certificates and copy them locally

$ docker run -i certbot/dns-dnsimple \
    certonly \
    --dns-dnsimple \
    --dns-dnsimple-credentials $pathToCredentialsFile \
    --agree-tos \
    --no-eff-email \
    -m $email \
    -d $domains$ docker cp $containerID:$pathToCerts. $localPathToCerts

2. Configure the nginx server to use these SSL certificates

# myApp/nginx-config.confhttp {
  # Use SSL certificates:
  ssl_certificate     /etc/ssl/fullchain1.pem;
  ssl_certificate_key /etc/ssl/privkey1.pem;  map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
  }  # Create server blocks for each subdomain/port used by your app
  server {
    listen 443 ssl default_server;
    location / {
      proxy_pass http://localhost:<myAppPort>/;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header Host $http_host;
      proxy_redirect off;
      proxy_http_version 1.1;
      proxy_set_header    Upgrade           $http_upgrade;
      proxy_set_header    Connection        "upgrade";
    }
  }
}events {
  accept_mutex on;
  worker_connections 1024;
}

3. Build an nginx image using this config

# myApp/Dockerfile-nginxFROM nginx
COPY nginx-config.conf /etc/nginx/nginx.conf
# assumes certs/ is your $localPathToCerts
COPY certs/ /etc/ssl/

3. Host images in an online registry

Once you have images of your app code and the nginx server, push them to an online registry so they can be used to spin up containers in the cloud. We chose AWS’s ECR but DockerHub works great too.

# tag each image with the registry URI
$ docker tag <imageID> <awsAccountId>.dkr.ecr.<awsRegion>.amazonaws.com/<registryID># push them up to the registry
$ docker push <tag>

4. Spin up containers

Once images are up in the cloud, you can use a container platform to spin up containers and create your environment.

We picked AWS Fargate, a relatively new AWS product which allows you to run containers without having to worry about the underlying infrastructure. We picked Fargate as our stack is moving towards using more AWS products but other container platforms such as Docker, Google Cloud or Microsoft Azure will work too.

Fargate abstracts away most of the complexity that comes with managing the virtual machines needed to run containers but you’ll need still to define the main configurations. For example, you’ll need to setup IAM permissions, a Fargate-enabled ECS cluster running on a VPC with the Internet access and security groups to allow incoming traffic to the various protocols/ports used by your app, etc. You’ll also need to create task definitions to tell Fargate how to spin up each environment.

You can create all of these directly from the AWS console or programmatically using the AWS SDK. Better yet, use wonqa — its init() method will create all required AWS resources for you, and its create() method will create task definitions and spin up an environment 😎.

// using the AWS NodeJS SDK to run a Fargate task and create an environmentconst AWS = require('aws-sdk');const runTask = taskDefinitionArn => {
  const ecs = new AWS.ECS({ region: <awsRegion> });
  const params = { 
    cluster: <clusterName>,
    launchType: 'FARGATE',
    networkConfiguration: {
      awsvpcConfiguration: {
        subnets: [<subnetID>],
        securityGroups: [<securityGroupID>],
        assignPublicIp: 'ENABLED'
      }
    },
    taskDefinition: taskDefinitionArn,
  };
  return ecs.runTask(params).promise();
}

5. Create DNS records

Once containers are up and running in the cloud, you can map the environment’s public IP to a user-friendly URL.

We used DNSimple’s API to create A and CNAME records and point <branchName>.<rootDomain> to the environment, but any DNS provider which exposes an API to edit records should do the trick.

const dnsimple = require('dnsimple');const dns = dnsimple({ accessToken: <apiToken> });const aRecord = { 
  type: 'A', 
  ttl: 60,  
  name: <subDomain>, 
  content: <publicIP> 
};
dns.zones.createZoneRecord(<accountID>, <rootDomain>, aRecord);const cNameRecord = { 
  type: 'CNAME', 
  ttl: 60,  
  name: `*.<subDomain>`, 
  content: <subDomain>.<rootDomain> 
};
dns.zones.createZoneRecord(<accountID>, <rootDomain>, cNameRecord);

6. Have CI and Github manage the workflow

Last but not least, we plugged this process into our CI workflow so that an environment could be generated when a PR was opened, and resources could be cleaned up when the PR was closed.

We used CircleCI’s workflow feature to create the environment and push a status update to the GitHub PR once the environment is up and running.

CircleCI runs a workflow to create the environment which then pushes a status update on the GitHub PR

Conclusion

The ability to create on-demand testing environments has been a boon for our development and testing workflows. As developers and product leads, we love having dedicated environments to test isolated features with confidence. Better yet, we’ve regained the time we were losing managing a single staging environment.

To create these environments, multiple steps and tools need to be orchestrated to turn app code into images, images into containers and containers into secure environments that can be accessed through easily recognizable URLs.

If you use a similar stack as us (NodeJS, Docker, AWS), try using wonqa to create these environments in a matter of minutes.

Want to work with the team that made this? Wonder is hiring!