Continuous Delivery for Elixir (Part 3 — conform is the new norm)

Where we’re headed

When we generate an OTP release (a tarball of all the things our application needs, its dependencies, and, often, Erlang itself), the release has a sys.config that contains the configuration as it was at compile time.

[{alfred,
     [{'Elixir.Alfred.Robot',
          [{adapter,'Elixir.Hedwig.Adapters.HipChat'},
           {aka,<<"!">>},
           {jid,<<"12345_123456@chat.hipchat.com">>},
           {name,<<"Alfred">>},
           {password,<<"password">>},
           {responders,
               [{'Elixir.Hedwig.Responders.GreatSuccess',[]},
                {'Elixir.Hedwig.Responders.Help',[]},
                {'Elixir.Hedwig.Responders.ShipIt',[]}]},
           {rooms,
               [[<<"12345_some_room@conf.hipchat.com">>,[]],
                [<<"12345_another_room@conf.hipchat.com">>,[]],
                {<<"12345_last@conf.hipchat.com">>,[]}]}]}]},
 {sasl,[{errlog_type,error}]}].

While entirely possible to change this later, the format leaves much to be desired when it comes to mere mortals updating it or if you wish to integrate with an automated configuration management system.

Also important to note, sys.config is in Erlang Term Format, so atoms aren’t prefaced with a “:” , binaries/strings require <<””>>, and don’t forget the trailing “.”!

Since we want to automate as much as possible about the deployment, we’ll want a format that’s easier to integration with automated configuration management systems.

alfred.Elixir.Alfred.Robot.adapter = Elixir.Hedwig.Adapters.HipChat
alfred.Elixir.Alfred.Robot.name = "Alfred"
alfred.Elixir.Alfred.Robot.aka = "!"
alfred.Elixir.Alfred.Robot.jid = "12345_123456@chat.hipchat.com"
alfred.Elixir.Alfred.Robot.password = "password"
alfred.Elixir.Alfred.Robot.rooms = "12345_some_room@conf.hipchat.com", "12345_another_room@conf.hipchat.com", "12345_last@conf.hipchat.com"

Conforming

Normally, adding conform to our project would be relatively straightforward: simply add the hex dependency and run “mix deps.get”. However, our project and its configuration will produce list of tuples for the rooms, which exposes a bug in conform 2.0.0. The issue has been fixed on master, but not yet released as 2.0.1.

defp deps do
  [
    {:exml, github: “paulgray/exml”, override: true},
    {:hedwig_hipchat, “~> 0.9”},
    {:romeo, “~> 0.4.0”, override: true},
    {:conform, github: “bitwalker/conform”, override: true},
  ]
end

We’ll also add conform to our list of dependent applications to launch at startup. This isn’t strictly necessary in this section, but will be required in the later release/exrm section. I have found a good habit to be that when I add a dependency, by default I also add it to the list of applications to start.

def application do
  [ applications: [
      :logger,
      :exml, :romeo, :hedwig, :hedwig_hipchat,
      :conform,
    ],
    mod: {Alfred, []}
  ]
end

Ok, we have now conformed, but what does it do, and how does it help us?

Conform adds three new mix tasks:

conform.configure # Create a .conf file from schema & project config
conform.effective # Print the effective configuration
conform.new # Create a new .schema.exs file for configuring your app 

Stay with me; I know what you’re thinking, “Yeah, um, that still doesn’t help me.” We’re going to walk through all of them.

The first thing we want to do is run the conform.new task

$ mix conform.new
==> The schema for your project has been placed in config/alfred.schema.exs

The generated file should look something like this:

We see that conform has pulled the contents of config/config.exs into this schema, and that the schema is mostly correct. Rooms aren’t a list of tuples of atoms and binaries, but we’ll fix that right now.

We know we want our rooms to be specified as a list of strings/binaries and that we need to output a list of tuples of binaries and lists for sys.config. We’ll change the datatype and create a custom transform to make that happen.

transforms: [
  “alfred.Elixir.Alfred.Robot.rooms”: fn conf ->
    conf
    |> Conform.Conf.get(“alfred.Elixir.Alfred.Robot.rooms”)
    |> Enum.map(fn {key, rooms} -> Enum.map(rooms, fn (name) -> {name, []} end) end)
    |> List.last
  end
],

We take the configuration identified we’ve been given, ask conform to retrieve from its ETS entry the rooms key. We then have something that looks like:

[
  {'alfred.Elixir.Alfred.Robot.rooms', ["587458_testing@conf.hipchat.com", "587458_elixircontinuousdelivery@conf.hipchat.com"]}
]

We then convert each of the rooms to the format Hedwig expects.

[
  [
    {"587458_testing@conf.hipchat.com", []},
    {"587458_elixircontinuousdelivery@conf.hipchat.com", []}
  ]
]

Finally, we only need the inner list.

[
  {"587458_testing@conf.hipchat.com", []},
  {"587458_elixircontinuousdelivery@conf.hipchat.com", []}
]

The full config/alfred.schema.exs for reference:

Now that we have a schema, a transform, and updated datatypes and defaults, we’ll use conform to generate the key=value .conf file.

$ mix conform.configure
==> The .conf file for alfred has been placed in config/alfred.conf

Fantastic! This key=value format is exactly what we’re after. This simple format is well understood by both humans and automation tools.

How do we know if this gives us what we need for sys.config though?

$ mix conform.effective
[alfred: [{Alfred.Robot,
   [adapter: Hedwig.Adapters.HipChat, aka: "!",
    jid: "587458_3985539@chat.hipchat.com", name: "Alfred",
    password: "not_my_real_password",
    responders: [{Hedwig.Responders.GreatSuccess, []},
     {Hedwig.Responders.Help, []}, {Hedwig.Responders.ShipIt, []}],
    rooms: [{"587458_testing@conf.hipchat.com", []},
     {"587458_elixircontinuousdelivery@conf.hipchat.com", []}]]}]]

Great Success!

The variance from what conform.effective outputs and the sys.config example above is a matter of Elixir default representations and Erlang default representations. Because conform assumes (correctly) you’re using Elixir toolchain, it displays the effective configuration using Elixir idioms: no explicitly <<>> for strings, ‘Elixir.’ dropped from the beginning of module names, no trailing . to denote a complete term, etc.

You should try changing a few values in config/alfred.conf and then rerun “mix conform.effective” so you can see the changes propagating.

Let’s commit our changes. Remember please don’t commit your password. Your commit should look something like:

add conform configuration · jeffweiss/alfred@2bebe4a

What’s next?

Now that we have a configuration schema, a friendly .conf file, and conform can translate to what’s needed by an OTP release, in the next section we will generate an OTP release. We’ll also add a plugin so that the .conf file will be examined just prior to application startup and a new sys.config generated so that our application starts with the latest content of alfred.conf.

Continuous Delivery for Elixir (Part 4 — Use a release, troubles cease)