DigitalOcean Spaces as a Terraform Backend

Amazon Web Services S3 object storage API compatibility is a very cool interface for a lot of great object storage options like Minio to run an S3-tooling compatible object storage service on your infrastructure, and likewise for hosted options like DigitalOcean Spaces, where the common tooling around AWS S3 applies to new endpoints, using the same components like authentication scheme, bucket namespacing, etc.

A common use-case that might concern DigitalOcean (and other IaaS consumers) users is storing Infrastrucuture-as-Code state data, consistently, in a reliable/durable manner, either in their infrastructure (if they run things like the Terraform-generated resources on DigitalOcean) or in-resource (if you run, for example, Minio in something like a standalone set of instances or inside of a Kubernetes cluster, etc.) among your other options for backends for state data that does not necessarily couple with your infrastructure in the same way.

Your use of Terraform in existing configurations doesn’t need to change drastically; in this case, you’re just adding a Terraform data block, alongside your usual provider block, to supply your data. What you’ll need is your DigitalOcean Space endpoint URL:

and your access token and secret token from https://cloud.digitalocean.com/settings/api/tokens under the Spaces section:

Keep in mind this will be separate from your DigitalOcean API token for resources like compute and block storage resources, and use only for access to object storage buckets themselves.

For our purposes here, we’ll cover, first, managing these credentials within Terraform (since you may be using S3-compatible tooling elsewhere, i.e. awscli or boto to manage Object Store data, I'll cover this as well afterwards).

Create (or update) your vars.tf (Note: this file should be in your .gitignore if you are version controlling Terraform) file with these keys:

space=terraform
spaces_endpoint=nyc3.digitaloceanspaces.com
deployment_name=infrastructure/memeCluster.tfstate
spaces_access_key="<your spaces API access key>"
spaces_secret_key="<your spaces API secret token>"

In your Terraform script, or in your repo in a file called provider.tf, you'll have a block like this to connect up to DigitalOcean's API:

provider "digitalocean" {
token = "${var.do_token}"
}

Great. But on Terraform run, it will dump all stateful data to your project root, which would remain locally stored (and a huge pain to check into version control reliably). So, let’s setup a block to store our state data remotely:

terraform {
backend "s3" {
endpoint = "${var.spaces_endpoint}"
bucket = "${var.space}"
key = "${var.deployment_name}"
access_key= "${var.spaces_access_key}"
secret_key= "${var.spaces_secret_key}"
}
}

and then fire off your Terraform run.

There are also alternate methods to handling credentials:

You can also set these 3 items as environment variables on your local machine (AWS_S3_ENDPOINT, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY respectively), and when you reference the S3 backend in terraform, these values can be detected; you may be familiar with this approach if you also use tooling like awscli or the botoclient package (which, because of s3-compatibility) which will also work with Spaces for s3 related usages. This can be used in place of the above instantiation (they only need to be sourced in your shell environment when the Terraform CLI is run), or alternatively, using a combination of the two approaches, you can import a ~/.aws/credentials file that you may already be using locally for these S3 tools), and then make the following modification to your Terraform configuration with the shared_credentials_file key:

terraform {
backend "s3" {
shared_credentials_file = "${var.shared_credentials_file}"
bucket = "${var.space}"
key = "${var.deployment_name}"
}
}

and point your vars entry to your credentials file.