Writing Code Directly On A Production Server

No really. This is a follow-up to a presentation I made at a Ruby Meetup in Ottawa on April 26, 2016. The premise was to use the exact same tools from infrastructure as code to drive your development environment, not just your production deploys.

What is Infrastructure As Code

So let’s start by talking a little bit about what I mean by infrastructure as code. Feel free to learn more from Martin Fowler (video and text), or Kief Morris (book).

First and foremost, our servers (aka the infrastructure) are defined by executable scripts generalized as Definition Files (aka executable code). By using code our goal is to achieve penultimate consistency (through immutability) between all of our servers.

The Definition File is a script that is used to generate your infrastructure (e.g. providing a server, setup DNS records, register with an existing cluster, setup SSL certs etc)

Configuration Synchronization

There are two ways to achieve this, the first is Configuration Synchronization.

You might also hear the term idempotent when referring to configuration synchronization; which means that if you apply a change to something the resulting state is indistinguishable whether you applied it once or several times. For example,

$ mkdir -p /tmp/hello # this is idempotent
$ mkdir -p /tmp/hello # as you can run it again, and again

Not so much for the following

$ mkdir /tmp/world # not idempotent
$ mkdir /tmp/world # as you get an error here
mkdir: cannot create directory ‘/tmp/world’: File exists

Having idempotency in your definition files makes it easier to support configuration synchronization as the tooling does not necessarily need to know what changed, as applying the definition instructions multiple time should not change the resulting state of your server had you run for the first time.

Tools that offer this kind of support include chef, puppet and ansible (there are a to more).

Tools for achieve configuration synchronization

As soon as you allow your servers to be changed, you lose confidence (well I do, anyway) in the current state of the system. This is known as configuration drift as the the actual current state of your server and that which would be produced by your definition file on a new server are no longer equivalent (look at how leftpad broke the internet).

Configuration drift (as well as plain old regular hack on the server drift) results in snowflake servers that are unique for your environment and extremely difficult to replicate.

Immutable Servers

A second mechanism to achieve infrastructure as code (which mitigates server drift) is with the use of Immutable Servers.

Instead of trying to synchronize changes to existing servers; you just create new ones and burn your old ones. This alleviates the complexities of trying to make your stuff idempotent, but also provides and a true(r) reflection of your current definition file.

Additional tools that help with immutable servers (not an exhaustive list) include:

Docker provides a great mechanism for packaging up infrastructure as images, as well as vagrant and finally plain on scripts (whether they are in ruby, python, bash) can work great for your needs.

Towards an Immutable Stack

So we now have a theoretical baseline from which we can start to confidently and consistently build our infrastructure. Next, let’s look at extending this notion to our development environment. Because right now, most of us develop on snowflakes.

Development is mostly done on snowflake servers, but it doesn’t have to be.

I am proposing that we develop on a server that is the equivalent to a production setup. This would be possible by having extra services available on your servers (e.g. brunch and sass for asset pre-compilation), but we are really just adding to disk, these services could easily be turned on when in development mode and then off production.

In the above, we produce our immutable stack (aka our development environment), work directly on it and when happy we release it to production. We then create a new instance of the immutable stack (which may or not have changed), work directly on it some more and when ready it becomes our new production. This allows us, as developers, to change our core tooling (e.g. new version of elixir, postgres, redis, etc) as well as our code (i.e. that is what we do, right?) in complete isolation.

This approach can work equally well for teams, and it becomes more of question of your development pipeline for which server(s) are designated for development, demo, staging, pre-prod, and/or production (but its just a label and slight config tweak). The point is that any server could be a production server (but it doesn’t have to be).

In the follow-up article, I will demonstrate how this could be achieved using Digital Ocean’s API, Bash (dio and bits) and Floating IPs.