Continuous Delivery for Elixir (Part 5 — Use a package to get trackage)

Quick Recap

As of the prior section, we have a build of our application containing an external configuration, all our our dependencies (including an Erlang runtime), and an initial start/stop script. That’s pretty good; however, how do we, our Ops folks, or an automated system know if we have the latest version of the application? Or, even, which version of the app we have? A fairly tractable problem for 1 app in 1 environment with a small and focused Ops team. Start adding apps and QA and Integration environments, and a host of other services for Ops to support and suddenly being a special snowflake means melting while others fight fires. This section takes the application artifacts we already have and layers on a system package. Because a world of tools have been built around system packages, this really opens up our adjacent possible.

System packages are the Sour Cream layer of our 7-layer deployment dip.

The author clearly has a metaphor problem. Stahp, author, stahp.

Also, if you haven’t checked out part 3.1, we ran into a minor snag with our release + conform. Head over and take a quick read. Otherwise, expect an untimely, fiery end for our app.

Continuous Delivery for Elixir (Part 3.1 — Conform Addendum)

Assumptions

Before we dive into packaging, let’s discuss some assumptions that I have made:

1. You’ve never built a system package before.
2. You don’t know and/or don’t care about the philosophical differences between various Linux distributions.
3. You do not intend to attempt to distribute your application as part of the default distribution (i.e. bundled with all Debian installs everywhere, because if you do, you should definitely know and care about 2.)
4. You don’t have byzantine policies about where your application is deployed in the production filesystem.

Our Target Environment

As we discussed in part 1, our target environment is Debian 8 (Jessie). We’ll use the LSB functions it provides to generate an init script that contains status as well. We’ll use the operating system process as proxy for whether our system is alive or not. In more sophisticated applications, you may wish to do direct application communication, but this will work for our purposes.

The LSB functions contain helpers for starting our application and putting the OS PID into a file. We’ll wrap start, stop, and restart with those helpers. We’ll then generate a status, and the remainder of commands we’ll pass straight through to our application.

You may have noticed that we’re assuming our application will be deployed to /opt/alfred. For the purposes of this tutorial, this will effectively be hard-coded. Our application will be in /opt, but our configuration will be in the standard /etc.

Because our canonical configuration file will be in /etc, we’ll need some way to point conform to it so that we’ll get the correct details into sys.config. Because symlink reads are transparent, we can change our .conf file in the release to a symbolic link to /etc/alfred.conf and conform will never know the difference. Since we want to make this symlink, we’ll create a packaging staging area for our project where we will layout the files we want included in our package.

packaging
├── etc
│   └── alfred.conf
└── opt
    └── alfred
        ├── bin
        ├── erts-7.3
        ├── lib
        └── releases
            ├── 0.0.1
            │   ├── alfred.bat
            │   ├── alfred.boot
            │   ├── alfred.conf -> /etc/alfred.conf
            │   ├── alfred.rel
            │   ├── alfred.schema.exs
            │   ├── alfred.script
            │   ├── alfred.sh
            │   ├── conform
            │   ├── start.boot
            │   ├── start_clean.boot
            │   ├── sys.config
            │   └── vm.args
            ├── RELEASES
            └── start_erl.data

Since this file structure contains both our application name and our application version, I’m inclined to be lazy and pull that information from our existing Mix project. With a little file and directory shuffling and calling out to tar and fpm, we have the perfect case for a Mix task.

Our overall set of steps will be

1. Create our working areas
2. Unpack our release
3. Move the .conf to /etc
4. Symlink the old location to /etc
5. Use fpm to build the package

Brief Aside

Because we only want to package a single versioned release, but several release versions may exist in the same rel directory, we’ll use the tarball generated by exrm instead of the file structure.

The contents of lib/mix/tasks/alfred.package.ex

Right now we have two problems

1. “Where is fpm coming from?”
2. “I thought you said we’d need an ERTS for our target environment.”

Let’s tackle those in reverse order and get a Debian 8 virtual machine running first.

Creating Our Debian 8 Virtual Machine

This local virtual machine will eventually become the basis of our CI build and test worker. Therefore, we know a few things we’ll need to bootstrap it with:

• Erlang
• Elixir
• Git (for dependencies from GitHub instead of Hex)
• XML development libraries (used by our XMPP dependencies)

We might as well make this VM our packaging machine and add

• Ruby and its development libraries
• fpm

I tend to make a lot of mistakes and like my finished processes to be repeatable. Because I plan on making a lot of mistakes, I don’t want to manually configure the VM every time I screw it up. I want to reuse the final configuration for a Jenkins worker later, so I definitely want an automated configuration. And, if we recall from part 1, I want to interact with the Debian-specific portions as little as possible. As a result, I reach for the specific combination of Vagrant and Puppet. Vagrant will allow me to quickly cycle through building and tearing down the VM, and Puppet will automate its configuration and give me an artifact to share with the Jenkins worker later.

I’m using a prebuilt Debian 8 Vagrant box that already has Puppet pre-installed.

The Puppet manifest will add the Erlang Solutions package repository, which houses the latest versions of Erlang and Elixir. The manifest will then ensure we have the latest versions of the dependencies we outlined above.

Please thank Erlang Solutions for their Erlang/Elixir package repository.

We also use Puppet to install fpm, with the added detail that fpm is a gem and it needs the Ruby development libraries.

We embed a couple Puppet modules (apt, concat, and stdlib) — not terribly good practice, but for our purpose of a quick and dirty build vm without further Puppet infrastructure or knowledge, it will suffice. For the full contents, take a peek at the commit. A longer-term solution will involve librarian-puppet.

Down to Business

We’re ready to fire up our build VM!

$ vagrant up
Bringing machine 'build' up with 'vmware_fusion' provider...
==> build: Cloning VMware VM: 'puppetlabs/debian-8.2-64-puppet'. This can take some time...
==> build: Checking if box 'puppetlabs/debian-8.2-64-puppet' is up to date...
==> build: Verifying vmnet devices are healthy...
==> build: Preparing network adapters...
==> build: Starting the VMware VM...
==> build: Waiting for machine to boot. This may take a few minutes...
...
==> build: Running provisioner: puppet...
==> build: Running Puppet with environment test...
... [this may take time while it downloads/installs dependencies]
==> build: Notice: /Stage[main]/Main/Package[fpm]/ensure: created
==> build: Notice: Applied catalog in 117.10 seconds

What we see is that Vagrant and Puppet have brought up the machine and configured it as we requested.

As in part 3.1, we’ll need to replace the conform escript.

$ vagrant ssh 
$ cd /vagrant
$ mix local.hex --force
$ mix local.rebar --force
$ mix deps.get
$ cd deps/conform
$ mix deps.get
$ MIX_ENV=prod mix do compile, escript.build
$ mv conform priv/bin
$ cd /vagrant

Now we can create our system package.

$ MIX_ENV=prod mix do compile, release, alfred.package

Woah! 🍻 We have a .deb system package! We can install it on this build machine if we want to try it out.

$ sudo dpkg -i alfred_0.0.1*.deb
$ sudo /etc/init.d/alfred status
$ sudo /etc/init.d/alfred stop
$ sudo nano /etc/alfred.conf
$ sudo /etc/init.d/alfred console
$ sudo /etc/init.d/alfred start

What’s next?

Let’s take a moment to recognize how far we come. Right now we have a package that we can hand to Ops. This package has our application and everything that’s need to run our application. The package has an init script that’s compatible with automation tools and a key=value config file in /etc. Our application looks, configures, and starts just like a vast majority of Linux services. If you’re looking for ways to “sneak” Elixir into your environment, this is it, my friends.

Don’t rest on your laurels quite yet. We still have a fair amount of automation left:

• Publishing to a package repository
• Automatically installing our package
• Automatically configuring our package