Terraformin’

Building a private cloud requires a comprehensive infrastructure-as-code toolkit alongside the requisite hardware, operating system and hypervisor software.

As part of the THG warehouse management system’s evolution we’re moving from running on a single cloud provider to running both in the public cloud and on our nascent private cloud.

Google image search for terraform brings up some interesting results…

In AWS, we are heavy users of CloudFormation, an infrastructure-as-code tool that maintains the system state of your ‘Virtual Private Cloud’ and other resources alongside a yaml (or json) format for describing the resources you wish to provision.

As part of research into our next platform move, I have also rebuilt some of our infrastructure in Google Cloud (GCP) using Google’s Deployment Manager — another Infrastructure-as-code tool similar to CloudFormation.

As we worked through the various options available to us, it was immediately obvious that CloudFormation (an AWS specific technology) would no longer “cut the mustard” due to it’s inherent “platform specificness” which is at odds with the goal of flexibility across multiple cloud vendors, that we were aiming for.

Google’s Deployment Manager has the same deficiency as CloudFormation (from now on cfn for brevity). We can use it effectively when interacting with GCP; however, it has no flexibility for use with other providers.

So, we were looking for a tool that could be used across clouds and has support for working with multiple vendors. The obvious choice for this is HashiCorp’s Terraform. It supports multiple backends (providers) including AWS and GCP and also offers support for OpenStack which we’re moving to for our internal or private cloud.

From cfn and gcp deployment manager to terraform

When transitioning from one platform to another there will usually be a loss of capabilities, and although terraform is an excellent tool, it is only as good as the “provider” that connects to your cloud platform.

As we are building our private cloud on top of OpenStack, we’re currently limited by the capabilities of the terraform openstack provider — luckily it’s open source so any limits can be removed by judicious use of donating development time to add missing features!

Converting our cfn definitions should be a simple case of finding the equivalent resource in the terraform provider and doing some fairly mechanical transformations from the yaml to the terraform descriptions (in HashiCorp Configuration Language or hcl).

However, the differences between cfn and terraform are more than just surface-level syntax. For example, capabilities of AWS that are not present in OpenStack are obviously missing, but there are also some subtle differences — where to apply firewall rules (Security Groups in AWS, on individual hosts, Security Groups or network ports in OpenStack) that must be navigated during the migration.

With Deployment Manager, the changes are much more stark. Cloudformation’s yaml is relatively easy to convert to the corresponding resources/data queries that Terraform provides. Deployment Manager, however, provides a more complex (or, more kindly, “sophisticated”) infrastructure-as-code tooling that uses Jinja2 templates which can contain more complex relationships and logic that are obviously impossible to model entirely in Cloudformation’s yaml.

When we were working with the Deployment Manager code, we made a conscious effort to avoid using any logic within the templates which we managed to do as the amount of hosts we needed to deploy was limited. A large estate or fleet of GCP resources would, however, be more irritating to manage (even using infrastructure-as-code!) without taking advantage of loops and other control structures available in the more complex Jinja2 template versions of deployment manager code.

Waiters

Eric Idle as a different kind of Waiter

The most complex feature of Deployment Manager we had to use was a Waiter. A waiter allows you to construct dependencies between resources where a secondary resource must not try to instantiate until the primary resource returns success.

In our case we needed to rely on the creation of a NAT instance to allow egress of traffic to the internet (where some of our installation dependencies live):

- type: runtimeconfig.v1beta1.waiter
  name: vgr-shared-nat-instance-waiter
  properties:
    waiter: vgr-shared-nat-instance-waiter
    parent: $(ref.vgr-shared-nat-instance-config.name)
    timeout: 120s
    success:
      cardinality:
      path: vgr-shared-nat-instance
      number: 1
  metadata:
    dependsOn: [vgr-shared-nat-igm]

Which we can then consume in our InstanceGroupManager (similar to an AWS AutoscalingGroup) definition:

- name: vgr-shared-nat-get-igm-instances
  action: gcp-types/computev1:compute.instanceGroupManagers.listManagedInstances
  properties:
    instanceGroupManager: vgr-shared-nat-igm
    project: thg-voyager
    zone: europe-west2-a
  metadata:
    dependsOn: [vgr-shared-nat-instance-waiter]

After investigation into our requirements, we decided that we didn’t immediately need an identical facility in our OpenStack private cloud and we would be able to migrate the rest of our simple GCP Deployment Manager configuration to terraform in a straight-forward fashion.

With these queries over compatibility and the “lowest-common-denominator” which terraform imposes on our cloud provisioning, we were ready to investigate some terraform and OpenStack specific idiosyncrasies.

Terraform Gotchas

As a team we’re used to deploying infrastructure changes via cfn as and when these changes are required. We have never had to think about co-ordination of these changes as cfn would prevent one engineer from overwriting or destroying another engineer’s work.

Terraform (by default) however doesn’t maintain the current state of your infrastructure in a shared location. Instead the state is captured in a local terraform.tfstate file. Losing this file results in anguish and pain (and yes 3/4 of the engineers on the project experienced this). Fortunately this state can be moved to a remote location such as S3 (ironically) or, in our case, using Swift.

After solving the issue of shared infrastructure state, our next problem was managing multiple engineers running terraform at once producing conflicting changes to the live resources state.

HashiCorp offer a solution to this via one of their terraform offerings (Terraform Cloud or Terraform Enterprise), which as a team we will be moving to in the future — however we worked out a reasonable solution without running a different terraform setup.

Briefly, we split out our terraform resources into sections based on the lifecycle of the resources. This allows us to run terraform for different directories within our repository at different cadences, depending on the rate of change. For example, we typically request floating_ip addresses from the external ip address pool, at the start of provisioning an OpenStack project and once requested (and granted) we don’t change these. So the terraform files for these (and other similar resources) are in a logical “run-once” directory.

On the other hand some resources could (and should) change frequently and these need to be separated so that when we run this terraform, we don’t accidentally re-provision resources that should be only “run-once”.

Self-provisioning

One of the key ideas we wanted to carry over from our AWS infrastructure provisioning was the concept of a virtual machine “self-provisioning”. This takes advantage of the linux facility of cloud-init. This facility allows a virtual machine booted in a cloud environment to take on a “personality” via a supplied user-data block:

Cloud images are operating system templates and every instance starts out as an identical clone of every other instance. It is the user data that gives every cloud instance its personality and cloud-init is the tool that applies user data to your instances automatically.

Typically cloud-init is used to ensure that a newly booted VM is configured with correct ssh keys, remounts persistent storage volumes, configures hostnames etc. Cloud-init is critical for allowing the use of an Autoscaling Group where a vm will automatically provision itself with the correct image and configuration (provided by user-data) to allow it to join an already running cluster of machines; enabling elastically up-scaling the amount of resources.

We have experimented (with a reasonable success rate) at performing more complex cloud-init processes by running ansible with a local connection.

Self-provisioning workflow example: Bastion

A “bastion”

We start with the ansible playbooks and roles in a github repository — allowing us to plugin linting, validation, etc of our infrastructure-as-code as steps in a “build” pipeline (obviously with interpreted code there’s no actual compilation/build happening here).

When we’re happy with the state of the ansible, a “build” is executed which creates an infrastructure “artefact” which we publish to S3 for safe-keeping and to allow simple global access later on.

The next step is the provisioning of a virtual machine in the cloud platform of choice (AWS, GCP or OpenStack). As we provision the virtual machine, we pass user-data that contains the needed instructions and credentials to pull the latest ansible playbooks and roles from the specified S3 bucket and then run ansible using a local connection to install all the specified software.

In this example terraform configuration file we define some templates; cloud-init.cfg and a start-script. These templates are rendered into a user_data block in the bastion host resource. We pass configuration parameters into these templates so that they can be customised for each region etc. we are provisioning instances in.

After executing terraform apply -var-file="" with our region and environment specific variables, and waiting for the underlying host to boot, we can see the following start-script executing in the logs:

[  140.926978] cloud-init[1101]: Collecting pip
[  140.961408] cloud-init[1101]: Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB)
[  141.176096] cloud-init[1101]: Collecting wheel
[  141.182934] cloud-init[1101]: Downloading https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
[  141.227608] cloud-init[1101]: Installing collected packages: pip, wheel
[  142.033080] cloud-init[1101]: Successfully installed pip-19.3.1 wheel-0.33.6

After installing pip and all the dependencies for ansible, the start-script downloads the infrastructure bundle from S3, unpacks it then executes to install our public keys on this bastion instance:

[  193.711225] cloud-init[1101]: + cd /tmp/ansible/playbooks/voyager-main/selfprovision/vgr/
[  193.713140] cloud-init[1101]: + ansible-playbook bastion.yaml -i /tmp/ansible/inventories/openstack.yaml --connection=local
[  195.614132] cloud-init[1101]: [WARNING]: Invalid characters were found in group names and automatically
[  195.618963] cloud-init[1101]: replaced, use -vvvv to see details
[  195.945109] cloud-init[1101]: PLAY [localhost] ***************************************************************
[  195.962891] cloud-init[1101]: TASK [Gathering Facts] *********************************************************
[  196.875096] cloud-init[1101]: ok: [localhost]
[  196.885734] cloud-init[1101]: TASK [users : Create group] ****************************************************
[  197.242077] cloud-init[1101]: changed: [localhost]
[  197.249742] cloud-init[1101]: TASK [users : Create users] ****************************************************

After the ansible has finished, this instance has bootstrapped itself into the type of host we need (in this example a bastion host).

With this practice of a virtual machine installing its own software as it boots, we only need to make a couple of changes to have a bootstrapping method for both the initial installation and for providing software updates:

• change the start script to look up a version number from consul; and
• trigger running the start script from an external controller (Jenkins in its role as a deployment tool for example)

This ensures our initial creation of an environment and updates to the environment follow the same overall process of self-provisioning.

Next Steps & Future Plans

As our experience using terraform with OpenStack has increased, we’ve, obviously, had some thoughts on what we would like to improve to our processes, tools and how we could optimise our work.

Terraform cloud

When we started the project we weren’t sure if we would have access to Terraform Enterprise in the near future. This didn’t pan out and we used various workarounds to make our lives easier. We now plan to transition to Terraform Cloud to allow us to work as a team more effectively (and this would potentially allow us to simplify our directory structures).

Bastion-on-demand

During normal everyday operations, there should be no need to have SSH access via a bastion server. Our current configuration has a bastion server always online.

An optimisation we could introduce would be to have an external controller that would be able to provision a bastion instance on-demand, providing better security and reducing our VM estate (and carbon footprint 🌳).

Caching Proxy

Currently each host downloads the assets it requires directly from S3. In the case of multiple hosts requesting the same assets we are making multiple trips over the “wild internet”.

An obvious optimisation here is to provision a squid caching proxy and make all requests for external assets (ie. requests to S3) available via this proxy. This would probably improve our deployment times and would allow us to tighten up our egress security group rules.

With a squid proxy we should also be able to simplify the start script and remove the requirement to install awscli and to inject AWS credentials onto the hosts, another small improvement.

Packer

The “self-provision” mechanism we use, both in AWS and now in OpenStack gives us some of the benefits of cloud infrastructure without fully diving into immutable infrastructure.

The next logical step would be to embrace immutable infrastructure and create a new image which contains both the underlying operating system (in our case Centos7) as part of the build instead of just building RPMs which is the current output from Jenkins. In this scenario, the deployment would simply be instantiating a new VM with a new image_id instead of using the base Centos7 image_id. The downside of this approach is how we handle storage of the many images required for the entire system and maintaining a back catalog of images to allow roll back.

The alternative to full-fledged virtual machine images would be to look at swapping some of the services out for docker images — Packer (yet another HashiCorp tool) can handle both vm images and docker images, allowing us to have a mix of both in play at once (eg. during migration).

Read-only disks

After implementing immutable infrastructure for our hosts, then next step would be to make the boot volumes of the hosts read-only. Only allowing data to be written to the mounted long-term storage volumes. This seems to be just about possible, depending on the exact version of Linux and the applications you need to run — one for some R&D!

Locked-down UI

Horizon UI showing instances view

OpenStack provides a web console (Horizon) which can be used to administrate an OpenStack environment. Currently the configuration of Horizon allows a user to make modifications to instances, volumes, secrets etc. It has been suggested that the UI should only allow a read-only view of the environment and that all modifications would need to be performed via the API only. This would work perfectly for us (as terraform essentially calls the OpenStack APIs)

However, there have been cases in the past in AWS where a host SSH daemon has crashed preventing anyone from fixing it locally, leaving the only recourse to use the AWS Web Console to restart the host. If we go down the route of having a read-only Horizon, we would probably need an Admin role that does have modification access (at least to reboot instances in such cases).

OpenStack Provider

As terraform must service many different cloud providers, it has a plugin architecture where the correct provider backend is downloaded upon uttering terraform init in an appropriate directory.

We soon discovered that the currently available OpenStack provider is missing some features that we would find very useful. As of writing, the version of the provider for OpenStack doesn’t allow for looking up previously created volume resources. This has led to us splitting volume creation into a separate directory and having a strict workflow of:

1. Run terraform apply to create volumes
2. Browse Horizon (OpenStack UI) to lookup volume ids of just created volumes
3. Edit .tfvars file to add volume ids
4. Run dependent terraform apply

This is obviously prone to error (and thankfully we only care about these volumes for our persistent data storage and not for our compute boot volumes). As we required this functionality and the terraform OpenStack provider is open source, THG has contributed back to this project so that we can make use of this new data resource for volumes as soon as the next version of the provider is publicly released.