Terraform RemoteState Server

Terraform is a pretty nifty tool to layout complex infrastructures across cloud providers. It is an expressway to overcome the otherwise mundane and tedious task of going through insane amount of API documentations.

The output of terraform runs is a JSON which carries an awesome lot of information that the cloud platform provides about a resource; like instance_id, public_ip, local_ip, tags, dns, security groups etc and often it has left me wondering If I could search/access these JSON document from configuration management recipes, playbooks, or modules.

Example: While provisioning a zookeeper instance, I wan the local-ip of all the peer nodes. I could run a query that would fetch me local_ips of all the nodes in this VPC that have the same security group. Or while applying a security patch to all the Redis nodes, I need the public-ip of all nodes that carry the tag `node_type: redis`.
I hope you get the idea of use cases by now and It definitely sounds like something that a document DB should be able to handle with relative ease.

To be able to achieve this, Terraform does not expose any pluggable backends to have custom formatters, however it does provide an ability to talk to a RESTful server. Every time a state needs to be read terraform makes a GET call on the /path specified while setting up the remote config. A save operation corresponds to a POST call on the same /path and a DELETE method call for a delete operation.

Here’s how you add a remote config to your terraform project:

terraform remote config \
    -backend=http \
    -backend-config="address=http://my.rest.api.com/hello"

While I wanted to export the information to MongoDB, others might want to store it somewhere else, maybe a Redis? Capitalising on terraforms ability to talk to a RESTful state server, I decided to write a implementation that would take data from the RESTful endpoint and save it to a MongoDB. Once it reaches MongoDB it’s fairly convenient and easy to use that information in the configuration manager code.

So I quickly put together a RESTful server, less than a day’s effort, written in Golang And it is available at http://github.com/meson10/tfstate

Given that you have GOPATH etc configured properly (In case you are new to Golang I suggest reading more about it here), You can download tfstate as simply as:
$: go get github.com/meson10/tfstate
This should provide you with a bianry file that you can execute as:
$: tfstate -config=/path/to/config.yaml

A sample configuration looks like this:

mongo:
  host: hello.mlab.com:15194
  database: terraform
  username: transformer
  password: 0hS0sw33t

Although tfstate by default talks to MongoDB but implementing your own backend is fairly easy. Each provider has to implement the Storer interface that looks like this:

type Storer interface {
    Setup(cfgpath string) error
    Get(ident string) ([]byte, error)
    Save(ident string, data []byte) error
    Delete(ident string) error
}

Look at https://github.com/meson10/tfstate/blob/master/mongo.go for a sample implementation of this Interface.

Here’s an output from a working use case:

piyush:infra-monk: master λ tfstate -config tfstate.yaml
2016/06/15 22:19:02 Getting ident azure-state-zookeeper
2016/06/15 22:19:07 Saving ident azure-state-zookeeper to DB
2016/06/15 22:19:27 Saving ident azure-state-zookeeper to DB

2016/06/15 22:20:39 Getting ident aws-state-cassandra
2016/06/15 22:20:41 Saving ident aws-state-cassandra to DB
2016/06/15 22:23:52 Saving ident aws-state-cassandra to DB

Feel free to leave a comment or send Pull Requests :)