Hot code reloading with Erlang and Rebar3

Vanshdeep Singh
Dec 2, 2015 · 6 min read

In this post I am going to discuss about hot code loading which is a crucial feature for production environments in order to achieve high availability. For people who don't know what hot code loading is following is my formal definition,

Hot code loading is the art of replacing an engine from a running car without having to stop it.

In simple words the aim is to update the current running code with new code without stopping the service. Now that we are clear on what hot code loading is let's try to see how we can achieve this with Erlang.


We will create a sample project and then learn how to do hot code loading. The source code for the project is located here. Let’s create a template project using rebar3,

$ rebar3 new release nine9s

Now we add and as dependencies in our rebar.config. For more realistic feel make the following change to your rebar.config.

You might be wondering what is this “nine9s” app going to do ? The idea is to create a hello world web server which will later be upgraded to serve new code. Modify you nine9s_app.erl so that the function looks as follows,

start(_StartType, _StartArgs) ->
Dispatch = cowboy_router:compile(
[{‘_’, [
{“/”, default_handler, []}
]}
]),
{ok, _} = cowboy:start_http(http, 10, [{port, 9090}],
[{env, [{dispatch, Dispatch}]}]),
‘nine9s_sup’:start_link().

Now lets create a module called ,

-module(default_handler).
-export([init/2]).
init(Req, Opts) ->
Req2 = cowboy_req:reply(200, [ {<<”content-type">>,
<<”text/plain”>>} ],
<<”Hello world!”>>, Req),
{ok, Req2, Opts}.

Next, we compile and run the app,

$ rebar3 compile && rebar3 release
$ _build/default/rel/nine9s/bin/nine9s-0.1.0 console

Now you have your app running, you can verify by browsing to Keep this app running as we will create a new version of this app and then try to hot load that code in this shell.

The above code is located in branch 0.1.0 of the project.


Now will add some new features to our project which will form the version 0.2.0 for this project after which we will try to hot load this code into the shell which is running version 0.1.0 . The code for version 0.2.0 is located in branch 0.2.0 of the project.

Let’s say that we want to count the number of times our has served requests. This is simple to solve, we create a gen_server which will store the number of times the was invoked,

-module(state_handler).
-behaviour(gen_server).
%% API functions
-export([hello_world/0,
get_hello_world_count/0,
start_link/0]).
%% gen_server callbacks
-export([init/1,
handle_call/3,
handle_cast/2,
handle_info/2,
terminate/2,
code_change/3]).
-record(state, {count = 0}).%%%==========================================================
%%% API functions
%%%==========================================================
hello_world() ->
gen_server:cast(?MODULE, hello_world).
get_hello_world_count() ->
gen_server:call(?MODULE, hello_world_count).
start_link() ->
gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
%%%===========================================================
%%% callback functions
%%%===========================================================
init([]) ->
{ok, #state{}}.
handle_call(hello_world_count, _From, State) ->
{reply, State#state.count, State};
handle_call(_Request, _From, State) ->
Reply = ok,
{reply, Reply, State}.
handle_cast(hello_world, State) ->
Count = State#state.count,
{noreply, State#state{count = Count + 1}};
handle_cast(_Msg, State) ->
{noreply, State}.
handle_info(_Info, State) ->
{noreply, State}.
terminate(_Reason, _State) ->
ok.
code_change(_OldVsn, State, _Extra) ->
{ok, State}.

We modify ourso that each time it receives a request it will notify ,

-module(default_handler).
-export([init/2]).
init(Req, Opts) ->
state_handler:hello_world(),
Req2 = cowboy_req:reply(200, [ {<<”content-type”>>,
<<”text/plain”>>} ],
<<”Hello world 2 !”>>, Req),
{ok, Req2, Opts}.

Our state_handler will be a worker process under nine9s_sup supervisor,

-module(‘nine9s_sup’).
-behaviour(supervisor).
-export([start_link/0]).
-export([init/1]).
-define(SERVER, ?MODULE).
-define(CHILD(Id, Mod, Args, Restart, Type), {Id, {Mod, start_link, Args}, Restart, 60000, Type, [Mod]}).
start_link() ->
supervisor:start_link({local, ?SERVER}, ?MODULE, []).
init([]) ->
State_Handler = ?CHILD(state_handler, state_handler, [], transient, worker),
{ok, { {one_for_all, 0, 1}, [State_Handler]} }.

Since we are recording the count for the we would like to have a cowboy route which would serve the current count. So we modify the nine9s_sup.erl,

-module(‘nine9s_app’).
-behaviour(application).
%% Application callbacks
-export([start/2
,stop/1]).
-export([set_routes_new/0
,set_routes_old/0 ]).
start(_StartType, _StartArgs) ->
Dispatch = cowboy_router:compile([{‘_’, get_new_routes()}]),
{ok, _} = cowboy:start_http(http, 10, [{port, 9090}],
[{env, [{dispatch, Dispatch}]}]),
‘nine9s_sup’:start_link().
stop(_State) -> ok.get_new_routes() ->
[{“/count”, count_handler, []}] ++ get_old_routes().
get_old_routes() ->
[{“/”, default_handler, []}].
set_routes_new() ->
CompileRoutes = cowboy_router:compile([{‘_’, get_new_routes() }]),
cowboy:set_env(http, dispatch, CompileRoutes).
set_routes_old() ->
CompileRoutes = cowboy_router:compile([{‘_’, get_old_routes() }]),
cowboy:set_env(http, dispatch, CompileRoutes).

Notice that I have divided the routes into two parts, the ones which were there in version 0.1.0 i.e. old routes and new routes. The functions and will be explained later.

Following is for the route “”,

-module(count_handler).
-export([init/2]).
init(Req, Opts) ->
Count = state_handler:get_hello_world_count(),
BCount = integer_to_binary(Count),
Req2 = cowboy_req:reply(200, [ {<<”content-type”>>,
<<”text/plain”>>} ],
BCount, Req),
{ok, Req2, Opts}.

Lastly we will update the version number in nine9s.app.src and rebar.config. This completes the features for version 0.2.0 . Now we will try to upgrade the code running in the shell from 0.1.0 to 0.2.0


In order to upgrade to new version we need to create a appup file i.e. nine9s.appup.src which describes how to update from version number 0.1.0 to 0.2.0,

{"vsn in app.src", 
[ {"upgrade from vsn", Instructions1}],
[ {"downgrade to vsn", Instructions2}]
}

The appup file is a tuple of three elements where the first element is the version as defined in the .(current version). The second argument is a list of tuples where the first element in each of these tuples is the version number from which to upgrade and the second argument is a list of instructions of how to upgrade from that version. These tuples direct how to upgrade from a given version to current version. The third argument in appup file is a list of tuples where first argument of each of these tuples is version number to downgrade to and second argument is a list of instructions directing how to downgrade to this version.

Following is the appup file for nine9s.

{“0.2.0”, 
[{“0.1.0”, [
{add_module, state_handler}
,{update, nine9s_sup, supervisor}
,{apply, {supervisor, restart_child, [nine9s_sup, state_handler]}}
,{load_module, default_handler}
,{add_module, count_handler}
,{load_module, nine9s_app}
,{apply, {nine9s_app, set_routes_new, [] }} ] }],
[{“0.1.0”, [
{load_module, default_handler}
,{apply, {supervisor, terminate_child, [nine9s_sup, state_handler]}}
,{apply, {supervisor, delete_child, [nine9s_sup, state_handler]}}
,{update, nine9s_sup, supervisor}
,{delete_module, state_handler}
,{apply, {nine9s_app, set_routes_old, [] }}
,{delete_module, count_handler}
,{load_module, nine9s_app}
]
}]}.

Now we will discuss the upgrading instructions here. NOTE that the instructions are executed in the order they a specified,

  • : directs to add the file to the shell.
  • : this will update the supervisor changing its internal i.e. changing the restart strategy and maximum restart frequency properties, as well as changing the existing child specifications. This will add our child spec to the supervisor.
  • : takes as arguments and executes . So we execute which will start the as a worker process under supervisor. Notice the order of instructions here, we have added the first then changed the supervisor state and then spawned the process.
  • :this will reload themodule which replace the old code.
  • : now we add the
  • we reload the module so that new function which we added are loaded.
  • :since we have loaded the new functions we will now execute to add new routes to our server.

Next argument gives instruction on how to downgrade which works similarly as above but old modules are loaded in place of new modules.

  • : will load the old default_handler
  • : terminate process.
  • : delete the child spec for from the
  • : update internal state of the supervisor.
  • : remove the module
  • : set the routes to the older version.
  • : remove the module.
  • : load the older module.

Since our appup file is complete, we are now ready to upgrade to latest code.

# Firstly we copy the appup file to nine9s src folder under lib
$ cp apps/nine9s/src/nine9s.appup.src _build/default/lib/nine9s/ebin/nine9s.appup
# Next we compile and release the new code
$ rebar3 compile
$ rebar3 release
# generate relup w.r.t to the previous release
$ rebar3 relup -n nine9s -v "0.2.0" -u "0.1.0"
# genereate tar file of the new release
$ rebar3 tar -n nine9s -v "0.2.0"
$ mv _build/default/rel/nine9s/nine9s-0.2.0.tar.gz _build/default/rel/nine9s/releases/0.2.0/nine9s.tar.gz# upgrade to the new release
$ _build/default/rel/nine9s/bin/nine9s-0.1.0 upgrade "0.2.0"

That's it ! If everything executed successfully then we have upgraded to version 0.2.0. You can check it by browsing to http://localhost:9090 and http://localhost:9090/count

The project source has two branches i.e. 0.1.0 and 0.2.0 . You can compile and run the code in 0.1.0 and then checkout 0.2.0 and use the python script to upgrade to version 0.2.0

Vanshdeep Singh

Written by

https://github.com/kansi