Hot Upgrades! Whaaaaaaaat?

Junaid Farooq
Sep 16, 2018 · 12 min read

Hot upgrades are quite a difficult task in deployments.

In past, we were using Exrm in Evercam.io and we were never bothered by any difficulty as well as never concerned about how things work when you deploy a phoenix application. The moment, Marco Herbst brought up the topic of hot upgrades, I was totally unaware of what it is, and how am I going to deal with this.

But we somehow achieved hot upgrades for our application.

In this article, I will try to some up my work of 3 weeks with all the difficulties I faced.

Keeping that in mind your project is a fresh repository and not using any kind of deployment tool yet now.

1. Add/Switch to Distillery.

This part is quite simple. Add distillery in your dependency list.

defp deps do
[
...
{:distillery, "~> 2.0"},
...
]
end

and run mix deps.get, compile

This part will go as smooth as expected. Now the fun part start here, you need to run mix release.init This will create these files and folders in the root of your project.

The config file is quite simple. But you can find out more interesting things about config here.

NOTE: for Hot upgrades, this part of config file should always be true.

environment :prod do
set include_erts: true

2. Incremental Application Versions.

Update:

The logic I have been using for application versions don't really work there is this issue: When you are doing your work in a branch and have deployed it many times, and then you want to deploy your master again with the hot upgrade, it will break because your master will not have the latest commit or version. So instead of using the old one, use this one, and it's better. It will change on each deploy either it's a branch or master again.

version: "1.0.#{DateTime.to_unix(DateTime.utc_now())}"

Deprecated Logic: One thing for hot upgrades, you should keep in mind, your application versions should be incremental. e.g. 1.0.1, 1.0.2, 1.0.3and so on.

So how are you going to do that???

Be my friend 😍

There are several approaches to do that we will do the best way possible.

defp versions do
{epoch, _} = System.cmd("git", ~w|log -1 --date=raw --format=%cd|)
[sec, tz] =
epoch
|> String.split(~r/\s+/, trim: true)
|> Enum.map(&String.to_integer/1)
sec + tz * 36
end

curious enough?

We are actually trying to use last commit’s date.

Interactive Elixir (1.7.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> {epoch, _} = System.cmd("git", ~w|log -1 --date=raw --format=%cd|)
{"1536894554 +0500\n", 0}

after making a few changes to the above results we will have our resulting version number. eh? Put the above defp version in your mix.exs file and then your project part should look like this.

def project do
[app: :evercam_media,
version: "1.0.#{versions()}",
.....
end

and for Evercam Server the app version will look like this.

iex(2)> EvercamMedia.Mixfile.project[:version]
"1.0.1536912554"
iex(3)>

and each time with your latest commit your version number for the app will change in an incremental way as its date and will be increasing and changing, Perfect!

Now let's create your first production release with the command MIX_ENV=prod mix release.

NOTE: For Evercam Server we have a lot of ENVs so each time when our ansible script compiles or create a release for us, we prepend those envs.

The results of above release command will give you such results for your application.

Generated evercam_media app
==> Assembling release..
==> Building release evercam_media:1.0.1536912554 using environment prod
==> Including ERTS 10.0 from /home/junaid/.asdf/installs/erlang/21.0/erts-10.0
==> Packaging release..
Release succesfully built!
To start the release you have built, you can use one of the following tasks:
# start a shell, like 'iex -S mix'
> _build/prod/rel/evercam_media/bin/evercam_media console
# start in the foreground, like 'mix run --no-halt'
> _build/prod/rel/evercam_media/bin/evercam_media foreground
# start in the background, must be stopped with the 'stop' command
> _build/prod/rel/evercam_media/bin/evercam_media start
If you started a release elsewhere, and wish to connect to it:# connects a local shell to the running node
> _build/prod/rel/evercam_media/bin/evercam_media remote_console
# connects directly to the running node's console
> _build/prod/rel/evercam_media/bin/evercam_media attach
For a complete listing of commands and their use:> _build/prod/rel/evercam_media/bin/evercam_media help

Easy enough for creating your first production release.

This one command is good enough for you to start your application

_build/prod/rel/evercam_media/bin/evercam_media start

but that is not the right way to do it, or its not a right way for hot upgrades. It's a convention that when you deploy your application for production. There should be two places on your remote server.

  1. where you clone, compile and create a release.
  2. where you place your production release to start it.

As I have mentioned above we are using ansible-playbook for all deploy purposes where we have mentioned the remote build directory and remote application directory. i.e /tmp/build_media and /opt/evercam_media respectively.

So once you have created a release for you application you will have these kind of files in your _build directory

In all of the above files, the most important one is tar.gz file. This file will be in your remote build directory and you will unarchive it to your remote application directory, in our case it was

  1. /tmp/build_media/
  2. /opt/evercam_media

and now in your remote application directory you will have files like this.

root@evercam-release-test:/opt/evercam_media# ls
bin erts-10.0 lib releases var

All above commands which were stated above now will work from this directory such as this command

/tmp/build_media/_build/prod/rel/evercam_media/bin/evercam_media start

will become like this for the new unarchived remote application directory

/opt/evercam_media/bin/evercam_media start

Now as you are working on Ubuntu server (I hope so.) you will need an upstart job or a systemd job to run your application as a daemon.

For a brief on systemd (Ubuntu 16.04 and above) you need to create a file in this directory

/etc/systemd/system/yourappname.service

after creating the file paste this minimal template for your application, you can replace the application name and your envs as you like.

[Unit]
Description=Evercam Media
After=network.target
[Service]
Type=forking
LimitNOFILE=1000000
User=root
Group=root
Environment=HOME=/home/root
Environment=LANGUAGE=en_US:en
Environment=LS_ALL=en_US.UTF-8
Environment=ERL_MAX_PORTS=10240
Environment=ERL_MAX_ETS_TABLES=7000
Environment=PORT=4000
Environment=MIX_ENV=prod
Environment=START_CAMERA_WORKERS=false
Environment=DATABASE_URL=postgres://localhost/evercam_dev
Environment=SNAPSHOT_DATABASE_URL=postgres://localhost/evercam_dev
WorkingDirectory=/opt/evercam_media
ExecStart=/opt/evercam_media/bin/evercam_media start
ExecStop=/opt/evercam_media/bin/evercam_media stop
Restart=always
RestartSec=5
Environment=LANG=en_US.UTF-8
SyslogIdentifier=evercam_media
[Install]
WantedBy=multi-user.target

Now save this file with a name you like to call your application and run this command from the same directory .i.e /etc/systemd/system/ where you have saved the file.

systemctl enable yourfilename.service

this command will result in

Created symlink from /etc/systemd/system/multi-user.target.wants/yourfilename.service to /etc/systemd/system/yourfilename.service.

Now you can just start and stop your application with 2 simple commands

systemctl stop evercam_media.service && systemctl start evercam_media.service

Now the most fun part is going to start which is creating an upgrade release for your application, the things you need to worry about.

  1. Upgrade will only work if there will be an older version of the application is present in build as we as the working directory.
  2. Your new version should be in incremental form and always getting changed.

Now make a few changes to your application (don't forget to commit the changes) and create an upgrade release as

MIX_ENV=prod mix release --upgrade

and this command will result in

Generated evercam_media app
==> Assembling release..
==> Building release evercam_media:1.0.1537045416 using environment prod
==> Generated .appup for evercam_media 1.0.1536912554 -> 1.0.1537045416
==> Relup successfully created
==> Including ERTS 10.0 from /home/junaid/.asdf/installs/erlang/21.0/erts-10.0
==> Packaging release..
Release succesfully built!
To start the release you have built, you can use one of the following tasks:
# start a shell, like 'iex -S mix'
> _build/prod/rel/evercam_media/bin/evercam_media console
# start in the foreground, like 'mix run --no-halt'
> _build/prod/rel/evercam_media/bin/evercam_media foreground
# start in the background, must be stopped with the 'stop' command
> _build/prod/rel/evercam_media/bin/evercam_media start
If you started a release elsewhere, and wish to connect to it:# connects a local shell to the running node
> _build/prod/rel/evercam_media/bin/evercam_media remote_console
# connects directly to the running node's console
> _build/prod/rel/evercam_media/bin/evercam_media attach
For a complete listing of commands and their use:> _build/prod/rel/evercam_media/bin/evercam_media help

So fair enough, It created an appup file for you and also a new version of the application.

3. The distillery is not using the new version number

So the above upgrade step is not as simple as it looks, I tried millions of times with always a new version number but Distillery never picked a new version number.

NOTE: It's not a bug in the distillery. the issue is that Mix is compiling the module, and because the module isn’t modified on disk, the result of that compilation is cached (namely the .app file which contains the version).

So for this purpose before doing the compile and mix release. you always need to touch your mix.exs file such as

touch mix.exs

that's enough for the compiler to realize that something got changed in mix.exs and it will consider it for compilation. In this way, your new version will always be picked up. handy enough? 😍

4. An upgrade is not working for me I am having a lot of Errors.

So as in the 2nd step, we successfully created an upgrade release of our application, but that’s not an ideal case always. You are going to hit many errors for sure.

  1. which may come during mix release --upgrade
  2. or the time you are doing an upgrade

You can always ask Paul Schoenfelder, I have been to many repos and asked questions but I never found anyone that much interested in solving your problems as this guy is.

The errors could be of any kind, At first, let's do an upgrade of our application as we were running 1.0.1536912554 version and our new upgrade release is 1.0.1537045416 as in past we just took the tar.gz file and unarchive it to your application directory i.e /opt/evercam_media but this time you will do differently. As I highly suggest you use ansible-playbook, you will get the new release version number this way.

cat /tmp/build_media/_build/prod/rel/evercam_media/releases/start_erl.data | awk '{print $2}'

this will give you the latest version number, If you have used ansible ever, then you can register this version number to a variable for future use. After getting the latest version number you will copy that version directory from your build directory to application directory’s release folder. Confused? Eh?

Your _build directory after an upgrade should look like this

from here you will copy the latest release folder to this directory

/opt/evercam_media/releases

and after copy your release folder will look like this

1.0.1536912554 1.0.1537045416 RELEASES evercam_media.rel start_erl.data

Now you will upgrade to a newers version , You can be anywhere in directory level but root directory is preferable, then run this command.

/opt/evercam_media/bin/evercam_media upgrade 1.0.1537045416

If everything goes right. you will see a message like this.

Release evercam_media:1.0.1537045416 not found, attempting to unpack releases/1.0.1537045416/evercam_media.tar.gz
Unpacked '1.0.1537045416' successfully!
Release evercam_media:1.0.1537045416 is already unpacked, installing..
Release evercam_media:1.0.1537045416 is already installed, current, and permanent!

Congratulations! you have done a hot upgrade.

NOTE: This is not as simple as it looks, You will hit so many hurdles during all this. and so many errors related relup files and many more. but you need to consider few things.

  1. Your old successive release is there when you again run mix release --upgrade and your old releases are also there when you run /opt/evercam_media/bin/evercam_media upgrade version
  2. You cannot do upgrade within your build directory, So that’s why you always need to create a release in other directory and then copy it to another to run your application from there.
  3. A hot upgrade is not unarchiveit's copying the new release folder to the releases folder in the application directory and then upgrade.

You are always going to face hurdles only when doing upgrades but all the errors will get connected to one place relup and appup but they will be in different forms always.

A few errors could look like these

▸  Received 'pang' from evercam_media@127.0.0.1!
▸ Possible reasons for this include:
▸ - The cookie is mismatched between us and the target node
▸ - We cannot establish a remote connection to the node
Node evercam_media@127.0.0.1 is not running!
Release evercam_media:1.0.1-a42e5917 not found, attempting to unpack releases/1.0.1-a42e5917/evercam_media.tar.gz
Unpacked '1.0.1-a42e5917' successfully!
Release evercam_media:1.0.1-a42e5917 is already unpacked, installing..
▸ Release handler check for evercam_media:1.0.1-a42e5917 failed with: {:no_matching_relup, '1.0.1-a42e5917', '1.0.1-a25483c8'}
"Could not locate code path for release-evercam_media\",\"1.0.1-a2ac4195!"

5. Hot upgrade happened? somethings have stopped?

So you have done everything possible in a mannered way make things work but there are few applications which have been stopped and they are not working after hot-upgrade, I have one example of Porcelain.

After hot-upgrade, the porcelain application stopped working. And its error was so misleading as

defp driver() do
case Application.fetch_env(:porcelain, :driver_internal) do
{:ok, mod} -> mod
_ ->
raise Porcelain.UsageError, message: "Looks like the :porcelain app is not running. " <>
"Make sure you've added :porcelain to the list of applications in your mix.exs."
end
end

Porcelain was there in my application list but the error was still coming. In short porcelain application’s envs which set by it on start was missing, So you need you re-init porcelain application.

Now the question is how to do it? How can you know or tell that at this point hot-upgrade just happened? No ideas? So be my guest.

code_change/3 this is the most interesting part of GenServer in case of a hot upgrade as written.

Invoked to change the state of the GenServer when a different version of a module is loaded (hot code swapping) and the state’s term structure should be changed.

So what you will do now? 😖

You will create a new module, I would name it aJanitor and it will look like this

defmodule EvercamMedia.Janitor do

use GenServer
require Logger

def start_link() do
GenServer.start_link(__MODULE__, :ok, [])
end

def init(_args) do
{:ok, 1}
end

def code_change(_old_vsn, state, _extra) do
Logger.info "Re-init Porcelain"
ensure_porcelain_is_init()
{:ok, state}
end

defp ensure_porcelain_is_init do
Porcelain.Init.init()
end
end

and add it to your supervisor tree as a worker, in our case

worker(EvercamMedia.Janitor, [])

NOTE: After adding this Janitor , the very first deploy should be from scratch. If you will hot deploy it, It will come up in your appup but it will not work, because of your supervisor tree might have gone in load_module state instead of update in appup.

To understand it more click me.

So after the first scratch deploy, you will do a hot-upgrade but wait??? code_change/3 is not working? Nothing is being called?? Also Janitor module is not coming in appup.

So Distillery is not considering it to be in appup and why it would be? Have you changed anything in it? And would it be logical to change few lines in Janitor to make it appear in appup file? 😧

So here is the magic trick, Module Attributes as its stated.

Module attributes as annotations are used by the code reloading mechanism in the Erlang VM to check if a module has been updated or not.

And we have not defined any version of our Janitor. So we will add a module attribute as @vsn

@vsn DateTime.to_unix(DateTime.utc_now())

Add the above line beneath require Logger.

Now whenever you will do a hot upgrade this @vsn will be changing each time and telling distillery appup creator that Janitor has been changed and place it in appup as

{update,'Elixir.EvercamMedia.Janitor',{advanced,[]},[]},

Each time when you do a hot upgrade the code_change/3 callback will get called and rescue you. (It rescued us). ✋

Suggestions: People mostly say not to use hot-upgrade but I will suggest to use it, It’s an awesome feature. But in any case, while hot-upgrade you hit any error, Just deploy from scratch and then do hot-upgrade again. If you will fall for errors came across while hot-upgrade then you will get frustrated. i.e. errors when you created an upgrade release or when you did upgrade.

There are so many things to learn from all this experience, I will highly suggest using Ansible Playbooks for deployment purposes. And read Appup Book along with Distillery’s guide. I faced many errors while all this process, But Distillery’s author was a big help to me.

This is my very first article on the internet. You can disagree with many things, I am open to any suggestions or help you need.

Expelliarmus.

Junaid Farooq

Written by

failure. programmer. failure. ..Ashfaq Ahmed **ideal** I am from Sahiwal Pakistan. https://github.com/ijunaid8989

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade