Functional testing of WebSockets. Erlang and more

© Caterina Carraro/Billie

Hi everyone and welcome to Billie Engineering Blog!

That is our first article but we want to start with a very serious topic here. Functional testing and how we use it to tests WebSockets.

General architecture

First of all, let’s have a look at what we use WebSockets for, using this simplified diagram.

A generic diagram of a process
  1. Client opens a WebSocket connection with a Push-server through Nginx. Nginx works as a proxy and it is mostly responsible for a load balancing.
  2. Once the connection is established, Client has to send the first message through a WebSocket with a data, required for an authentication of a user. It might be a token or an API key or any other key, which allows authenticating user uniquely.
  3. Push-server checks all the data and confirms an authentication. Now Client is ready to receive messages from the push server.
  4. Since many of our objects are stateful, one of the most common cases we use WebSockets for is to send status updates to the Client. For example, if our automatic risk checks approve an application of a user, we notify him immediately. Push-server has a special API endpoint and Backend uses it to send an update on this endpoint. The message itself also contains a data of a receiver.
  5. Push-server finds all active connections of a given user and sends a message to it.

Technologies and libraries

The process, described above, should be highly reliable, that is why we want to always be sure that it works. One of the methods we use to prove that everything works as expected is Functional testing.

Our Push-server is written in Erlang, that’s why we decided to use the same language for functional testing.

Here is a list of libraries, which help us to run functional tests

  • Etest is a convention over a test framework for Erlang.
  • Etest_HTTP is an extension over Etest, which allows testing HTTP responses.
  • Gun is a WebSocket client for Erlang
  • Jiffy is a JSON NIFs for Erlang

Code

Here is a full code of a functional test:

-module(push_server_SUITE).

-compile(export_all).

-include_lib("common_test/include/ct.hrl").
-include_lib ("etest/include/etest.hrl").
-include_lib ("etest_http/include/etest_http.hrl").

all() ->
[
websocket_handler_test_case
].

init_per_suite(Config) ->
application:ensure_all_started(push_server),
application:ensure_all_started(gun),
Config.

end_per_suite(Config) ->
application:stop(push_server),
application:stop(gun),
Config.

wsConnect() ->
{ok, Pid} = gun:open("localhost", 8080),
{ok, http} = gun:await_up(Pid),
Ref = monitor(process, Pid),
gun:ws_upgrade(Pid, "/websocket", [], #{compress => true}),
receive
{gun_ws_upgrade, Pid, ok, _} ->
ok;
Msg ->
ct:print("Unexpected message ~p", [Msg]),
error(failed)
end,
{Pid, Ref}.

wsClose(Pid, Ref) ->
demonitor(Ref),
gun:close(Pid),
gun:flush(Pid).

listener(Pid, ExpectedFrame) ->
receive
{gun_ws, Pid, ReceivedFrame} ->
ct:print("Received ~p", [ReceivedFrame]),
?assert_equal(ExpectedFrame, ReceivedFrame);
Msg ->
ct:print("Unexpected message ~p", [Msg])
end.

websocket_handler_test_case(_Config) ->
% open a connection to a push server
{Pid, Ref} = wsConnect(),
% send an authentication message to a websocket
gun:ws_send(Pid, {text, jiffy:encode({[{{<<"token">>, <<"token1">>}]})}),
% start listener to consume a message
listener(Pid, {text,<<"You're logged id! {\"token\":\"token1\"}">>}),
% send a message to an API endpoint to be transfered to a client through the push server
?assert_status(204, ?perform_post(
"http://localhost:8080/api/v1/message",
[{"Content-Type", "application/json"}],
jiffy:encode({[{{<<"token">>, <<"token1">>}, {<<"data">>, <<"data1">>}]})
)),
% start listener to consume a message
listener(Pid, {text,<<"\"data1\"">>}),
% close connection to a push server
wsClose(Pid, Ref),
ok.

Now let’s have a deep look at each part of it.

-module(push_server_SUITE).

-compile(export_all).

-include_lib("common_test/include/ct.hrl").
-include_lib ("etest/include/etest.hrl").
-include_lib ("etest_http/include/etest_http.hrl").

This part is quite common for all Erlang modules, but please pay an attention to the last three lines. There we include Etest, Etest_HTTP and Common Test, which is the main testing framework for Erlang.

all() ->
[
websocket_handler_test_case
].

init_per_suite(Config) ->
application:ensure_all_started(push_server),
application:ensure_all_started(gun),
Config.

end_per_suite(Config) ->
application:stop(push_server),
application:stop(gun),
Config.

All three functions above belong to Common Test Framework. all() is a mandatory function, which returns a list of test cases and groups in the module. init_per_suite() — this function gets executed before the first test in a suite. In our case, we need to start there our Push-server and Gun applications. And finally end_per_suite() — is a function, which runs at the end of a suite and there we need to stop both applications.

wsConnect() ->
{ok, Pid} = gun:open("localhost", 8080),
{ok, http} = gun:await_up(Pid),
Ref = monitor(process, Pid),
gun:ws_upgrade(Pid, "/websocket", [], #{compress => true}),
receive
{gun_ws_upgrade, Pid, ok, _} ->
ok;
Msg ->
ct:print("Unexpected message ~p", [Msg]),
error(failed)
end,
{Pid, Ref}.

This one is a first helper to be used in the test case later. This function is responsible for opening a WebSocket connection. Therefore let’s have a look at it in details.

{ok, Pid} = gun:open("localhost", 8080),

Here Gun opens a connection to a Push-server, but keep in mind — it’s not a WebSocket yet. Another important moment is that Gun runs a separate process for the connection and this is a key part of the whole test. Because it allows us to send messages from one process (functional test by itself) to another one (the one we have just started).

{ok, http} = gun:await_up(Pid),

This function waits for a process to be completely up. When it is ready, Gun sends its own message gun_up and gun:await_up handles it.

Ref = monitor(process, Pid),

This function does what it says. It monitors a new process.

gun:ws_upgrade(Pid, "/websocket", [], #{compress => true}),

And only now we tell the server to upgrade a connection to a WebSocket.

receive
{gun_ws_upgrade, Pid, ok, _} ->
ok;
Msg ->
ct:print("Unexpected message ~p", [Msg]),
error(failed)
end,
{Pid, Ref}.

This part simply awaits for a successful upgrade of a connection.

wsClose(Pid, Ref) ->
demonitor(Ref),
gun:close(Pid),
gun:flush(Pid).

This function simply stops monitoring, closes and releases the connection. As you can guess, it has to be called at the end of a test.

listener(Pid, ExpectedFrame) ->
receive
{gun_ws, Pid, ReceivedFrame} ->
ct:print("Received ~p", [ReceivedFrame]),
?assert_equal(ExpectedFrame, ReceivedFrame);
Msg ->
ct:print("Unexpected message ~p", [Msg])
end.

This function is a helper, which receives a Pid of a running process and a message, it expects to consume. When it gets a message either from the Client or from the Backend, it compares it with the expected one.

websocket_handler_test_case(_Config) ->
% open a connection to a push server
{Pid, Ref} = wsConnect(),
% send an authentication message to a websocket
gun:ws_send(Pid, {text, jiffy:encode({[{{<<"token">>, <<"token1">>}]})}),
% start listener to consume a message
listener(Pid, {text,<<"You're logged id! {\"token\":\"token1\"}">>}),
% send a message to an API endpoint to be transfered to a client through the push server
?assert_status(204, ?perform_post(
"http://localhost:8080/api/v1/message",
[{"Content-Type", "application/json"}],
jiffy:encode({[{{<<"token">>, <<"token1">>}, {<<"data">>, <<"data1">>}]})
)),
% start listener to consume a message
listener(Pid, {text,<<"\"data1\"">>}),
% close connection to a push server
wsClose(Pid, Ref),
ok.

And finally the test suite. Let’s look at each line separately again.

{Pid, Ref} = wsConnect(),

Here we open a WebSocket and save Pid and Ref of the process.

gun:ws_send(Pid, {text, jiffy:encode({[{{<<"token">>, <<"token1">>}]})}),

Here we send a pre-encoded JSON message with an authentication data of the user. Also important that we send it not from a current process, but from a process where a Websocket connection is open.

listener(Pid, {text,<<"You're logged id! {\"token\":\"token1\"}">>}),

Here we check that the authentication has been done successfully.

Now let’s send the first message in an opposite direction. And that’s how we do it.

?assert_status(204, ?perform_post(
"http://localhost:8080/api/v1/message",
[{"Content-Type", "application/json"}],
jiffy:encode({[{{<<"token">>, <<"token1">>}, {<<"data">>, <<"data1">>}]})

We send a message for a user authenticated by a token1 on an API endpoint. Afterwards, we prove that it has been received correctly, checking a status of a response. Our Push-server defines processes, which should receive this message (in our current case, just the one identified by Pid) and sends it.

listener(Pid, {text,<<"\"data1\"">>}),

And here we consume the message and prove that it is what we expect.

wsClose(Pid, Ref),

And finally, we close a connection.

Run, Forest, run!

Let’s run our test. Since we use Erlang.mk, nothing is simpler for us:

make tests

And here’s an output:

Testing billie.push-server.push_server_SUITE: Starting test, 1 test case
Starting Sync (Automatic Code Compiler / Reloader)
Scanning source files...
----------------------------------------------------
2017-09-11 10:58:37.665
Received {text,<<"You're logged id! {\"token\":\"token1\"}">>}
----------------------------------------------------
2017-09-11 10:58:37.666
Received {text,<<"\"data1\"">>}
=INFO REPORT==== 11-Sep-2017::10:58:37 ===
application: push_server
exited: stopped
type: temporary
=INFO REPORT==== 11-Sep-2017::10:58:37 ===
application: gun
exited: stopped
type: temporary
Testing billie.push-server.push_server_SUITE: TEST COMPLETE, 1 ok, 0 failed of 1 test case

As you can see, everything is successful and we can even prove output messages, which have been added in the code above.

Conclusion

I have just demonstrated you a basic implementation of how to do a functional testing of WebSockets. But of course, you can do much more with it. Negative cases, wrong responses, wrong status codes, broken connections, concurrent connections, load testing etc.

Also, everything we have just written in Erlang might be written on many other languages using the same approach.

Good luck and have a nice testing!

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.