Robust compute for RDF queries

Managing fault tolerance in Elixir with supervision trees

“flock of birds” by Barth Bailey on Unsplash

In this post I aim to provide some reasons for why Elixir (and under the hood Erlang) ushers in a whole new paradigm for distributed compute and how it makes this both simple and fun to master for semantic web applications.

(And just to anticipate any comment on the figure above signifying some kind of distributed compute pattern, check out the note at the end of this post.)

1. Review process model of Erlang (and Elixir)

The essence of distributed compute is embedded deep within the genes of the language. Erlang was by design developed by Ericsson to manage telephony applications and to provide for fault tolerant processing famously achieving nine nine’s of uptime. It’s native home is the network.

Erlang uses the actor model and implements actors as processes which are one of its main language constructs. These are very lightweight structures and are implemented at the language level – not the OS level. Communication between processes is strictly via message passing and state is private to the process.

Erlang runtime and the process pool, and process schematic.

Now processes can be managed with primitive operations but for reliability and robustness we need something more.

OTP (originally Open Telecom Platform) is a framework of middleware, libraries, and tools for Erlang and ships as an integral part of the Erlang distribution. Among the tools are the Observer which we’ve seen before in an earlier post and which we will shortly use again below. But especially of interest here are some of the abstraction patterns that OTP provides for processes: agents, servers, supervisors, and tasks.

1a. (Generic) Server
Perhaps the key process abstraction in OTP is the generic server, or gen_server (or, as it is known in Elixir, the GenServer). This process behaviour provides the server of a client–server relation and can be used to maintain state, execute code, etc. Any module implementing this behaviour only needs to provide the functionality for a small known set of callbacks. The server pattern is managed by the behaviour module itself and has been exhaustively tested over the years so makes for a reliable application component.

Erlang process-to-process communication by message passing.

Elixir provides further abstractions over the GenServer: we have Agent for state management, and Task for asynchronous calculations.

1b. Supervisor
The second key process abstraction in OTP is the supervisor, or supervisor (or, as it is known in Elixir, the Supervisor). This process behaviour provides a supervisor for other processes and can be used to build up a hierarchical process structure called a supervision tree. Supervision trees provide fault tolerance and encapsulate how our applications start up and shut down.

Erlang process supervision trees with supervisors and workers.

Again Elixir provides a couple further abstractions over the Supervisor: DynamicSupervisor for managing dynamic child processes, and TaskSupervior for supervising tasks.

1c. ‘Let it crash’
Indeed, so ingrained is the supervision tree model that Erlang (and Elixir too) has something of a ‘Let it crash’ mantra. Rather than defensively programming for all sorts of unknown conditions, some of which inevitably arise through dynamic combinations of circumstances which cannot be predicted nor (more worryingly) reproduced, Erlang instead puts worker processes under a supervisor which will restart any faulting process according to a predeclared supervision strategy. And it will do so transparently. (This is similar to how Erlang also allows for hot-code swapping, or software upgrades, to occur in situ on a running system – with no downtime.)

But this is not to say that Erlang skimps on checks and balances. It uses strong typing and compile time type checking, allows for guards on functions and for exceptions to be raised, and supports running of test suites. It is just more realistic about the context in which it will be executed.

This paradigm shift is an excellent fit to the networked environment where we must always expect the unexpected. Indeed, things happen. This is the true nature of an open world of computation.

1d. Examples in RDF.ex/SPARQl.ex
These process behaviours are so fundamental to applications that we also see examples of Agent, GenServer and Supervisor patterns used in the RDF.ex and SPARQL.ex packages for various purposes: blank node management, SPARQL extension functions, etc.

2. Plan the approach

I aim to demo here the following abstractions:

  • GenServer
  • Supervisor
  • DynamicSupervisor

Our rough goal is to create an Elixir Application that can be used to query DBpedia for random resources and save the links (together with labels and Wikipedia links) into GenServer processes which can be created dynamically. The processes will be created under OTP supervision trees and will be automatically recreated when killed (manually, or otherwise). In this simple demo though we won’t attempt to preserve existing state across process restarts.

(I had a further notion of establishing the GenServer processes as language-specific buckets, e.g. an English bucket, a French bucket, a German bucket, etc. But for now let’s just keep things simple.)

Let’s lay out an outline for the project:

  • We’ll initially create the project.
  • We’ll then implement a GenServer, show some callbacks and a public interface for those.
  • We’ll next demo the GenServer as a basic key/value store.
  • We’ll then add SPARQL queries into the mix using SPARQL.Client and save our results into a GenServer.
  • Next we’ll put a GenServer under a static supervision tree and show how it is automatically restarted when it errors.
  • And lastly we’ll create GenServer processes dynamically.

3. Create a ‘TestSuper’ project

As usual, let’s create a new project TestSuper with Mix but this time we’ll also use a --sup flag to set up an Application. (We’ll be using the Application to auto-start our supervision trees.)

% mix new test_super --sup

Note that this resulted in two changes. We now have an extra application.ex file under the lib/test_super/ directory which defines a TestSuper.Application module. Let’s just ignore that for now.

% tree lib/
lib/
├── test_super
│ └── application.ex
└── test_super.ex
1 directory, 2 files

We also have one extra line in our mix.exs file which will automatically invoke the TestSuper.Application.

def application do
[
extra_applications: [:logger],
mod: {TestSuper.Application, []}
]
end

Since we’re going to be later using the SPARQL.Client.ex package let’s now declare our dependency in the mix.exs file. We’ll also use the hackney HTTP client in Erlang. And we want to generate some documentation so will include the ex_doc dependency.

defp deps do
[
{:ex_doc, "~> 0.18", only: :dev, runtime: false},
{:sparql_client, "~> 0.2.1"},
{:hackney, "~> 1.6"},
]

And we add this line to the config.exs file:

# config :tesla, :adapter, Tesla.Adapter.Hackney

We then use Mix to add in the dependency:

% mix deps.get

Let’s also clear out the boilerplate in lib/test_super.ex and add in a @moduledoc annotation.

defmodule TestSuper do
@moduledoc """
Top-level module used in "Robust compute for RDF queries" post.
"""

end

And also to simplify naming in IEx we’ll add a .iex.exs configuration file.

% cat .iex.exs
import TestSuper
import TestSuper.Client

See here for the project TestSuper code. (And note that the project TestSuper code also includes documentation which can be browsed here.)

4. Add in a GenServer

We’re now ready to define a GenServer. Let’s create ourselves a new module TestSuper.Server and add that to a file server.ex under the directory lib/test_super/.

defmodule TestSuper.Server do
use GenServer
  ## Constructor
  # def start_link(_)
  ## Callbacks
  # def init(_), etc.
end

Recall that a GenServer is a process with an OTP behaviour and OTP defines the server functionality. All we need to do is to add in client-side functionality via the callbacks. And also, to start up the process.

The layout of this module has three main pieces. First there is a use GenServer directive which will import the GenServer behaviour into our module. We then have a process constructor start_link/1 which spawns a new GenServer process and links to it. And lastly we have various callbacks, the only mandatory callback being init/1.

So let’s now flesh this out.

defmodule TestSuper.Server do
@moduledoc """
Module providing server-side functions for `GenServer`.
"""
use GenServer
  ## Constructor
  def start_link(opts \\ []) do
case GenServer.start_link(__MODULE__, opts) do
{:ok, pid} ->
# register process name
num = String.replace("#{inspect pid}", "#PID", "")
Process.register(pid, Module.concat(__MODULE__, num))
{:ok, pid}
{:error, reason} -> {:error, reason}
end
end
  ## Callbacks
  def init(_) do
{:ok, Map.new}
end
  def handle_call({:get}, _from, state) do
{:reply, state, state}
end
  def handle_call({:get, key}, _from, state) do
{:reply, Map.fetch!(state, key), state}
end
  def handle_call({:keys}, _from, state) do
{:reply, Map.keys(state), state}
end
  def handle_cast({:put, key, value}, state) do
{:noreply, Map.put(state, key, value)}
end
end

The only thing needed for start_link/1 is the call through to the GenServer.start_link/2 function. So here we spawn a GenServer process and link to that process. But we’ll add in a more friendly entry point in a moment.

The other piece here is for registering a process name (which we’ll see later in the Observer). Basically we’re just going to name the GenServer process with our module name and a concatenated process ID.

Client processes can pass various messages , or requests, to a GenServer which needs corresponding callback functions to be implemented. Now, there are two types of requests that we can send to a GenServer: calls and casts. Calls are synchronous and the server must send a response back to such requests. Casts are asynchronous and the server won’t send a response back.

On the server side, we can implement a variety of callbacks to guarantee server initialization, termination, and handling of requests. We are just going to implement these three callbacks for initialization and request handling:

  • init/1 – intialize the server
  • handle_call/3 – service a call/2 client request (for reads)
  • handle_cast/2 – service a cast/2 client request (for writes)

Since we’re going to use this GenServer to manage a key/value store we’ll use a Map to implement this. This is set up by the init/1 callback.

Often the client functions (calls) and server functions (callbacks) will be placed together in the server module. Here, however, we have made a cleaner separation and put our client calls into a TestSuper.Client module and kept the server callbacks in the TestSuper.Server module.

Here’s our TestSuper.Client module calls.

defmodule TestSuper.Client do
@moduledoc """
Module providing client-side functions for `GenServer`.
"""
  ## Calls
  def get(pid) do
GenServer.call(pid, {:get})
end
  def get(pid, key) do
GenServer.call(pid, {:get, key})
end
  def keys(pid) do
GenServer.call(pid, {:keys})
end
  def put(pid, key, value) do
GenServer.cast(pid, {:put, key, value})
end
end

This is just for demo purposes so we only define some basic read/write functions (get/1, get/2 and put/3). We don’t have any update or delete functions but they are easy to add.

So far, so good. We just need a more friendly way of starting our GenServer. We’ll add this genserver/0 constructor to our main module TestSuper.

def genserver() do
opts = [
]
case TestSuper.Server.start_link(opts) do
{:ok, pid} ->
IO.puts "TestSuper.Server is starting ... #{inspect pid}"
pid
{:error, reason} ->
IO.puts "! Error: #{inspect reason}"
end
end

We’re now all set.

5. Try out the GenServer by storing some key/value pairs

So, let’s try this out first with our client API.

iex(1)> pid = genserver
TestSuper.Server is starting ... #PID<0.219.0>
#PID<0.219.0>
iex(2)> put(pid, :foo, "bar")
:ok
iex(3)> get(pid)
%{foo: "bar"}
iex(4)> put(pid, :baz, 123)
:ok
iex(5)> get(pid)
%{baz: 123, foo: "bar"}
iex(6)> get(pid, :foo)
"bar"

Now we can also inspect the guts of the GenServer process with the Observer tool we saw in an earlier post.

iex(7)> :observer.start

Now if we scoot over to the ‘Processes’ tab we can locate the GenServer process, by sorting on the ‘Pid’ or ‘Name’ column: Elixir.TestSuper.Server.<0.219.0>. This is the name that we registered for the process.

Also, under the ‘Current Function’ heading we see the current function identified as gen_server:loop/7. So, we are indeed running a GenServer process.

Now selecting the ‘State’ tab, we can see the actual state stored in the process and it’s an empty map.

If we now put a couple key/value pairs as before and again inspect the state we see now we have a map with entries.

So, that’s a GenServer process storing key/value state.

6. Try out the GenServer by storing SPARQL results

Let’s add some support now to our TestSuper.Client module for querying DBpedia. First, as in previous posts, we’ll store a SPARQL query dbpedia_query.rq in our /priv/queries/ directory.

@priv_dir "#{:code.priv_dir(:test_super)}"
@queries_dir @priv_dir <> "/queries/"
@query_file "dbpedia_query.rq"
@service "http://dbpedia.org/sparql"
## Queries
def query() do
File.read!(@queries_dir <> @query_file)
end

The query/0 function just reads the query from the priv/queries/ directory. We can try this out.

iex(2)> query |> IO.puts
prefix dbo: <http://dbpedia.org/ontology/>
prefix foaf: <http://xmlns.com/foaf/0.1/>
prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>
select *
where {
bind (12345 as ?id)
?s dbo:wikiPageID ?id .
optional { ?s foaf:isPrimaryTopicOf ?topic }
optional { ?s rdfs:label ?label }
filter (langMatches(lang(?label), "en"))
} limit 1
:ok

This query will look for a resource with a dbo:wikiPageID of 12345. It will add in foaf:isPrimaryTopicOf and rdfs:label properties if present, and filter for English-language labels.

def putq(pid) do
{key, value} = _do_query()
IO.puts "#{inspect key} => #{inspect value}"
GenServer.cast(pid, {:put, key, value})
end
defp _do_query() do
# rewrite query with new random wikiPageID
rand = Integer.to_string(Enum.random(1..50000))
q = String.replace(query(), "12345", rand)
  # send query
{:ok, result} = SPARQL.Client.query(q, @service)
  # parse query result set
if length(result.results) == 0 do
{rand, "(not found)"}
else
id = result |> SPARQL.Query.Result.get(:id) |> List.first
s = result |> SPARQL.Query.Result.get(:s) |> List.first
label = result |> SPARQL.Query.Result.get(:label) |> List.first
topic = result |> SPARQL.Query.Result.get(:topic) |> List.first
{id.value, {s.value, label.value, topic.value}}
end
end

So here we have the first function client call putq/0 which will send the results from a query to a GenServer for storing.

The second (private) function _do_query/0 does the actual querying and parses the result set. It first generates a new random dbo:wikiPageID and replaces that ID in the query string. The modified query is sent to the DBpedia SPARQL endpoint using SPARQL.Client.query/2. The result set is tested and if empty a tuple {rand, “(not found)”} is returned. Otherwise the ?id?s, ?label and ?topic variables are parsed out of the first result and returned in the tuple {id.value, {s.value, label.value, topic.value}}.

We can try this out in IEx.

iex(1)> pid = genserver
TestSuper.Server is starting ... #PID<0.227.0>
#PID<0.227.0>
iex(2)> putq(pid)
49631 => {"http://dbpedia.org/resource/Turner's_syndrome", "Turner's syndrome", "http://en.wikipedia.org/wiki/Turner's_syndrome"}
:ok
iex(3)> get(pid)
%{
49631 => {"http://dbpedia.org/resource/Turner's_syndrome", "Turner's syndrome", "http://en.wikipedia.org/wiki/Turner's_syndrome"}
}
iex(4)> get(pid, 49631)
{"http://dbpedia.org/resource/Turner's_syndrome", "Turner's syndrome", "http://en.wikipedia.org/wiki/Turner's_syndrome"}

Or for querying (and storing) in bulk:

iex(5)> for _ <- 1..5, do: putq(pid)
23367 => {"http://dbpedia.org/resource/Telecommunications_in_Pakistan", "Telecommunications in Pakistan", "http://en.wikipedia.org/wiki/Telecommunications_in_Pakistan"}
1130 => {"http://dbpedia.org/resource/Avicenna", "Avicenna", "http://en.wikipedia.org/wiki/Avicenna"}
10216 => {"http://dbpedia.org/resource/Eth", "Eth", "http://en.wikipedia.org/wiki/Eth"}
"37810" => "(not found)"
"29721" => "(not found)"
[:ok, :ok, :ok, :ok, :ok]

And we can get this back as:

iex(6)> get(pid)
%{
1130 => {"http://dbpedia.org/resource/Avicenna", "Avicenna", "http://en.wikipedia.org/wiki/Avicenna"},
10216 => {"http://dbpedia.org/resource/Eth", "Eth", "http://en.wikipedia.org/wiki/Eth"},
23367 => {"http://dbpedia.org/resource/Telecommunications_in_Pakistan", "Telecommunications in Pakistan", "http://en.wikipedia.org/wiki/Telecommunications_in_Pakistan"},
49631 => {"http://dbpedia.org/resource/Turner's_syndrome", "Turner's syndrome", "http://en.wikipedia.org/wiki/Turner's_syndrome"},
"29721" => "(not found)",
"37810" => "(not found)"
}

Note that this includes both the single result we stored earlier (49631), as well as the bulk query results (1130, 10216, 23367, 29721, 37810).

Now we can again inspect the state stored in the GenServer process with the Observer tool.

iex(7)> :observer.start

If we choose the ‘Processes’ tab, sort the processes by clicking on the ‘Pid’ heading , and select the GenServer process we just created (#PID<0.227.0>) we have the following screen:

Double clicking on this process and choosing the ‘State’ tab gives us this screen showing the state.

And if we click the link ‘Click to expand above term’ we get this:

So, that’s a GenServer process storing key/value state for the results from our SPARQL queries.

7. Add in a static supervision tree

At this point we want to add in a supervision tree. Let’s first create ourselves a TestSuper.Supervisor module.

defmodule TestSuper.Supervisor do
@moduledoc """
Module providing server-side functions for `Supervisor`.
"""
use Supervisor
  ## Constructor
  def start_link(opts \\ []) do
Supervisor.start_link(__MODULE__, [], opts)
end
  ## Callbacks
  def init([]) do
children = [
TestSuper.Server
]
opts = [
name: TestSuper.Supervisor,
strategy: :one_for_one
]
    Supervisor.init(children, opts)
end
end

So, we also want a more friendly means to start this up. Let’s add this to the main TestSuper module.

def supervisor() do
opts = [
name: TestSuper.Supervisor,
strategy: :one_for_one
]
case TestSuper.Supervisor.start_link(opts) do
{:ok, pid} ->
[{_, child_pid, _, _}] = Supervisor.which_children(pid)
IO.puts "TestSuper.Supervisor is starting ... #{inspect pid}"
IO.puts "TestSuper.Server is starting ... #{inspect child_pid}"
pid
{:error, reason} ->
IO.puts "! Error: #{inspect reason}"
end
end

All this does is to call through to our TestSuper.Supervisor.start_link/1 constructor function passing a couple options: a name and a supervisaion strategy. There are three defined supervision strategies (see the Supervisor documentation for more info) but we are going to choose the simplest: :one_for_one, i.e. one child process (the GenServer) for our Supervisor.

We also test the success of this operation and deal accordingly. And lastly we write out a report of the Supervisor and GenServer processes starting up together with their process IDs. (We use the Supervisor.which_children/1 function to get the child process ID.)

So, we can use the supervisor/0 constructor to create a new Supervisor with an attached GenServer. But we can simplify further and get the Application itself to start up the Supervisor automatically.

Now let’s have a quick peek at that TestSuper.Application module. The one public function start/2 calls a private function _start/3 with a boolean argument which selects for a dynamic supervision tree on true and a static supervision tree on false. The intial setting is false, i.e. selects for a static supervision tree.

defmodule TestSuper.Application do
@moduledoc """
Module providing the `Application` start function.
"""
use Application
  def start(type, args) do
_start(type, args, false)
end
  defp _start(type, args, flag) do
case flag do
false -> _static_start(type, args)
true -> _dynamic_start(type, args)
end
end
end

I won’t say any more about the Application behaviour here other than to note that they are process bundles having similar functionality. Our main reason for using the Application to start up the Supervisor is that we can then use the Observer to view our Supervisor and its GenServer.

% iex -S mix
Erlang/OTP 21 [erts-10.0.6] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]
Compiling 1 file (.ex)
Interactive Elixir (1.7.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :observer.start
:ok

Now if we choose the ‘Applications’ tab in the Observer we’ll see this and see at left the list of applications being used here. Our test_super application has a couple processes which are linked and link through to the Supervisor and its attached GenServer.

If we now double click the GenServer process button Elixir.TestSuper.Server.<0.234.0> we will open up in the ‘Process information’ tab.

And if we select the ‘State’ tab we’ll get this.

We can again try adding state as before and viewing here. (Note that the panel does not refresh and you’ll need to close and reopen to view changes.)

Now, since the GenServer process was auto-started by the Application, we’re going to have to create a process ID for our client functions which we can do with the pid/3 function:

iex(4)> pid = pid(0,234,0)
#PID<0.234.0>

And now we can send requests to the GenServer:

iex(5)> putq(pid)
"495" => "(not found)"
:ok
iex(6)> putq(pid)
"3894" => "(not found)"
:ok
iex(7)> putq(pid)
23535 => {"http://dbpedia.org/resource/Photon", "Photon", "http://en.wikipedia.org/wiki/Photon"}
:ok

And if we close and repoen the ‘State’ tab we’ll see the new state we just stored.

So, what’s the Supervisor doing for us? Let’s see.

If we go back to the ‘Applications’ tab on the main window and now right click on the GenServer process, we’ll get this popup.

So, choose ‘Kill process’ and this dialog box pops up.

Just click ‘OK’, and …

Look at that. It’s the same as before (almost), but now the GenServer process has been automatically restarted (albeit with a different process ID, #PID<0.338.0> instead of #PID<0.234.0>).

Now if there was any state stored in the process then that will have been erased. We’re back to a blank slate. We need other strategies to preserve the state. But the process itself survives, and any compute functions associated with the GenServer are again available. Good as new.

8. Add in a dynamic supervision tree

Now the last example was for a static supervision tree. We could have added more GenServer proceeses under a single Supervisor, or even built up a tree with many levels by adding Supervisor processes under a Supervisor.

But now let’s look at how to create a dynamic supervision tree. Let’s first create ourselves a TestSuper.DynamicSupervisor module.

defmodule TestSuper.DynamicSupervisor do
@moduledoc """
Module providing server-side functions for `DynamicSupervisor`.
"""
use DynamicSupervisor
  ## Constructor
  def start_link(opts \\ []) do
DynamicSupervisor.start_link(__MODULE__, [], opts)
end
  ## Callbacks
  def init([]) do
opts = [
name: TestSuper.DynamicSupervisor,
strategy: :one_for_one
]
    DynamicSupervisor.init(opts)
end
end

And again we also want a more friendly means to start this up. Let’s add this to the main TestSuper module.

def supervisor_d() do
opts = [
name: TestSuper.DynamicSupervisor,
strategy: :one_for_one
]
case TestSuper.DynamicSupervisor.start_link(opts) do
{:ok, pid} ->
IO.puts "TestSuper.DynamicSupervisor is starting ... #{inspect pid}"
pid
{:error, reason} ->
IO.puts "! Error: #{inspect reason}"
end
end

This supervisor_d/0 constructor function is pretty much the same as the supervisor/0 constructor.

But again we can just use the Application to start up our DynamicSupervisor process. For now let’s just flip the false arg in the TestSuper.Application module to true on the start/2 function to read as:

def start(type, args) do
_start(type, args, true)
end

And let’s restart IEx.

% iex -S mix
Erlang/OTP 21 [erts-10.0.6] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]
Compiling 1 file (.ex)
Interactive Elixir (1.7.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :observer.start
:ok

Now this time we only have the DynamicServer. There are no child GenServer processes attached.

Let’s add a GenServer.

iex(2)> genserver_d
#PID<0.266.0>

Now we can see this atached. And, in fact, this looks very similar to the previous static supervision tree example.

Let’s add another GenServer.

iex(3)> genserver_d
#PID<0.271.0>

And now we can see two child GenServer processes for the DynamicSupervisor.

Let’s go crazy and add ten GenServer child processes.

iex(4)> for _ <- 1..10, do: genserver_d
[#PID<0.277.0>, #PID<0.278.0>, #PID<0.279.0>, #PID<0.280.0>, #PID<0.281.0>,
#PID<0.282.0>, #PID<0.283.0>, #PID<0.284.0>, #PID<0.285.0>, #PID<0.286.0>]

Yup.

That’s right. The DynamicSupervisor here is managing a dozen GenServer processes, but could just as easily be twelve thousand, or more.


Summary

I’ve shown here in this post how to manage fault tolerance in Elixir with supervision trees. We first reviewed the process model for Erlang (Elixir) and then discussed some of the abstraction patterns that OTP provides for processes: agents, servers, supervisors, and tasks.

We then developed a demo to show how the GenServer process can be used to store state. In this example we used a map to maintain a key/value store. We also made use of the SPARQL.Client.ex package to query DBpedia and to save our results into the GenServer.

We then showed how both static and dynamic supervision trees may be created for managing child processes using the Supervisor and DynamicSupervisor patterns. Using the Observer we showed how child processes can be killed and get restarted automatically by the Supervisor.

It is this support for fault tolerance and distributed compute that makes Elxir such a fascinating candidate for semantic web applications.


Endnote

Earlier I had in mind a kind of picture of distributed compute and thought that the banner image to this post was an excellent metaphor. And then I came across this image below by Martin Stannard using the Scenic UI framework for Elixir where each triangle is a separate GenServer process. (See his Twitter post which shows an animated version, and GitLab repo for the project.)

“Each triangle is a GenServer!” by Martin Stannard on Twitter


See here for the project TestSuper code. (And note that the project TestSuper code also includes documentation which can be browsed here.)

This is the fourth in a series of posts. See my previous post ‘Working with SHACL and Elixir’.

You can also follow me on Twitter as @tonyhammond.