Hot code reloading with Erlang and Rebar3
In this post I am going to discuss about hot code loading which is a crucial feature for production environments in order to achieve high availability. For people who don't know what hot code loading is following is my formal definition,
Hot code loading is the art of replacing an engine from a running car without having to stop it.
In simple words the aim is to update the current running code with new code without stopping the service. Now that we are clear on what hot code loading is let's try to see how we can achieve this with Erlang.
We will create a sample project and then learn how to do hot code loading. The source code for the project is located here. Let’s create a template project using rebar3,
$ rebar3 new release nine9sNow we add cowboy and lager as dependencies in our rebar.config. For more realistic feel make the following change to your rebar.config.
You might be wondering what is this “nine9s” app going to do ? The idea is to create a hello world web server which will later be upgraded to serve new code. Modify you nine9s_app.erl so that the start/2 function looks as follows,
start(_StartType, _StartArgs) ->
Dispatch = cowboy_router:compile(
[{‘_’, [
{“/”, default_handler, []}
]}
]),
{ok, _} = cowboy:start_http(http, 10, [{port, 9090}],
[{env, [{dispatch, Dispatch}]}]), ‘nine9s_sup’:start_link().
Now lets create a module called default_handler.erl,
-module(default_handler).
-export([init/2]).init(Req, Opts) ->
Req2 = cowboy_req:reply(200, [ {<<”content-type">>,
<<”text/plain”>>} ],
<<”Hello world!”>>, Req),
{ok, Req2, Opts}.
Next, we compile and run the app,
$ rebar3 compile && rebar3 release
$ _build/default/rel/nine9s/bin/nine9s-0.1.0 consoleNow you have your app running, you can verify by browsing to http://localhost:9090. Keep this app running as we will create a new version of this app and then try to hot load that code in this shell.
Now will add some new features to our project which will form the version 0.2.0 for this project after which we will try to hot load this code into the shell which is running version 0.1.0 . The code for version 0.2.0 is located in branch 0.2.0 of the project.
Let’s say that we want to count the number of times our default_handler has served requests. This is simple to solve, we create a state_handler.erl gen_server which will store the number of times the default_handler.erl was invoked,
-module(state_handler).
-behaviour(gen_server).%% API functions
-export([hello_world/0,
get_hello_world_count/0,
start_link/0]).%% gen_server callbacks
-export([init/1,
handle_call/3,
handle_cast/2,
handle_info/2,
terminate/2,
code_change/3]).-record(state, {count = 0}).%%%==========================================================
%%% API functions
%%%==========================================================
hello_world() ->
gen_server:cast(?MODULE, hello_world).get_hello_world_count() ->
gen_server:call(?MODULE, hello_world_count).start_link() ->
gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).%%%===========================================================
%%% callback functions
%%%===========================================================
init([]) ->
{ok, #state{}}.handle_call(hello_world_count, _From, State) ->
{reply, State#state.count, State};
handle_call(_Request, _From, State) ->
Reply = ok,
{reply, Reply, State}.handle_cast(hello_world, State) ->
Count = State#state.count,
{noreply, State#state{count = Count + 1}};handle_cast(_Msg, State) ->
{noreply, State}.handle_info(_Info, State) ->
{noreply, State}.terminate(_Reason, _State) ->
ok.code_change(_OldVsn, State, _Extra) ->
{ok, State}.
We modify our default_handler.erl so that each time it receives a request it will notify state_handler,
-module(default_handler).
-export([init/2]).init(Req, Opts) ->
state_handler:hello_world(),
Req2 = cowboy_req:reply(200, [ {<<”content-type”>>,
<<”text/plain”>>} ],
<<”Hello world 2 !”>>, Req),
{ok, Req2, Opts}.
Our state_handler will be a worker process under nine9s_sup supervisor,
-module(‘nine9s_sup’).
-behaviour(supervisor).
-export([start_link/0]).
-export([init/1]).
-define(SERVER, ?MODULE).
-define(CHILD(Id, Mod, Args, Restart, Type), {Id, {Mod, start_link, Args}, Restart, 60000, Type, [Mod]}).start_link() ->
supervisor:start_link({local, ?SERVER}, ?MODULE, []).init([]) ->
State_Handler = ?CHILD(state_handler, state_handler, [], transient, worker),
{ok, { {one_for_all, 0, 1}, [State_Handler]} }.
Since we are recording the count for the default_handler we would like to have a cowboy route which would serve the current count. So we modify the nine9s_sup.erl,
-module(‘nine9s_app’).
-behaviour(application).%% Application callbacks
-export([start/2
,stop/1]).
-export([set_routes_new/0
,set_routes_old/0 ]).start(_StartType, _StartArgs) ->
Dispatch = cowboy_router:compile([{‘_’, get_new_routes()}]),
{ok, _} = cowboy:start_http(http, 10, [{port, 9090}],
[{env, [{dispatch, Dispatch}]}]),
‘nine9s_sup’:start_link().stop(_State) -> ok.get_new_routes() ->
[{“/count”, count_handler, []}] ++ get_old_routes().get_old_routes() ->
[{“/”, default_handler, []}].set_routes_new() ->
CompileRoutes = cowboy_router:compile([{‘_’, get_new_routes() }]),
cowboy:set_env(http, dispatch, CompileRoutes).set_routes_old() ->
CompileRoutes = cowboy_router:compile([{‘_’, get_old_routes() }]),
cowboy:set_env(http, dispatch, CompileRoutes).
Notice that I have divided the routes into two parts, the ones which were there in version 0.1.0 i.e. old routes and new routes. The functions set_routes_new/0 and set_routes_old/0 will be explained later.
Following is count_handler for the route “/count”,
-module(count_handler).
-export([init/2]).init(Req, Opts) ->
Count = state_handler:get_hello_world_count(),
BCount = integer_to_binary(Count),
Req2 = cowboy_req:reply(200, [ {<<”content-type”>>,
<<”text/plain”>>} ],
BCount, Req),
{ok, Req2, Opts}.
Lastly we will update the version number in nine9s.app.src and rebar.config. This completes the features for version 0.2.0 . Now we will try to upgrade the code running in the shell from 0.1.0 to 0.2.0
In order to upgrade to new version we need to create a appup file i.e. nine9s.appup.src which describes how to update from version number 0.1.0 to 0.2.0,
{"vsn in app.src",
[ {"upgrade from vsn", Instructions1}],
[ {"downgrade to vsn", Instructions2}]
}The appup file is a tuple of three elements where the first element is the version as defined in the .app.src (current version). The second argument is a list of tuples where the first element in each of these tuples is the version number from which to upgrade and the second argument is a list of instructions of how to upgrade from that version. These tuples direct how to upgrade from a given version to current version. The third argument in appup file is a list of tuples where first argument of each of these tuples is version number to downgrade to and second argument is a list of instructions directing how to downgrade to this version.
Following is the appup file for nine9s.
{“0.2.0”,
[{“0.1.0”, [
{add_module, state_handler}
,{update, nine9s_sup, supervisor}
,{apply, {supervisor, restart_child, [nine9s_sup, state_handler]}}
,{load_module, default_handler}
,{add_module, count_handler}
,{load_module, nine9s_app}
,{apply, {nine9s_app, set_routes_new, [] }} ] }],
[{“0.1.0”, [
{load_module, default_handler}
,{apply, {supervisor, terminate_child, [nine9s_sup, state_handler]}}
,{apply, {supervisor, delete_child, [nine9s_sup, state_handler]}}
,{update, nine9s_sup, supervisor}
,{delete_module, state_handler}
,{apply, {nine9s_app, set_routes_old, [] }}
,{delete_module, count_handler}
,{load_module, nine9s_app}
]
}]}.
Now we will discuss the upgrading instructions here. NOTE that the instructions are executed in the order they a specified,
- {add_module, state_handler} : directs to add the state_handler.erl file to the shell.
- {update, nine9s_sup, supervisor} : this will update the supervisor changing its internal i.e. changing the restart strategy and maximum restart frequency properties, as well as changing the existing child specifications. This will add our state_handler child spec to the supervisor.
- {apply, {supervisor, restart_child, [nine9s_sup, state_handler]}} : “apply” takes {M, F, A} as arguments and executes M:F(A1, … An). So we execute supervisor(nine9s_sup, state_handler) which will start the state_handler as a worker process under nine9s_sup supervisor. Notice the order of instructions here, we have added the state_handler first then changed the supervisor state and then spawned the state_handler process.
- {load_module, default_handler} : this will reload the default_handler module which replace the old code.
- {add_module, count_handler} : now we add the count_handler
- {load_module, nine9s_app} we reload the nine9s_app module so that new function which we added are loaded.
- {apply, {nine9s_app, set_routes_new, [] }} ] }] : since we have loaded the new functions we will now execute nine9s_app:set_routes_new() to add new routes to our server.
Next argument gives instruction on how to downgrade which works similarly as above but old modules are loaded in place of new modules.
- {load_module, default_handler} : will load the old default_handler
- {apply, {supervisor, terminate_child, [nine9s_sup, state_handler]}} : terminate state_handler process.
- {apply, {supervisor, delete_child, [nine9s_sup, state_handler]}} : delete the child spec for state_handler from the nine9s_sup
- {update, nine9s_sup, supervisor} : update internal state of the supervisor.
- {delete_module, state_handler} : remove the state_handler module
- {apply, {nine9s_app, set_routes_old, [] }} : set the routes to the older version.
- {delete_module, count_handler} : remove the count_handler module.
- {load_module, nine9s_app} : load the older nine9s_app module.
Since our appup file is complete, we are now ready to upgrade to latest code.
# Firstly we copy the appup file to nine9s src folder under lib
$ cp apps/nine9s/src/nine9s.appup.src _build/default/lib/nine9s/ebin/nine9s.appup# Next we compile and release the new code
$ rebar3 compile
$ rebar3 release# generate relup w.r.t to the previous release
$ rebar3 relup -n nine9s -v "0.2.0" -u "0.1.0"# genereate tar file of the new release
$ rebar3 tar -n nine9s -v "0.2.0"$ mv _build/default/rel/nine9s/nine9s-0.2.0.tar.gz _build/default/rel/nine9s/releases/0.2.0/nine9s.tar.gz# upgrade to the new release
$ _build/default/rel/nine9s/bin/nine9s-0.1.0 upgrade "0.2.0"
That's it ! If everything executed successfully then we have upgraded to version 0.2.0. You can check it by browsing to http://localhost:9090 and http://localhost:9090/count
The project source has two branches i.e. 0.1.0 and 0.2.0 . You can compile and run the code in 0.1.0 and then checkout 0.2.0 and use the python script upgrade.py to upgrade to version 0.2.0
