Robust compute for RDF queries
Managing fault tolerance in Elixir with supervision trees
In this post I aim to provide some reasons for why Elixir (and under the hood Erlang) ushers in a whole new paradigm for distributed compute and how it makes this both simple and fun to master for semantic web applications.
(And just to anticipate any comment on the figure above signifying some kind of distributed compute pattern, check out the note at the end of this post.)
1. Review process model of Erlang (and Elixir)
The essence of distributed compute is embedded deep within the genes of the language. Erlang was by design developed by Ericsson to manage telephony applications and to provide for fault tolerant processing famously achieving nine nine’s of uptime. It’s native home is the network.
Erlang uses the actor model and implements actors as processes which are one of its main language constructs. These are very lightweight structures and are implemented at the language level – not the OS level. Communication between processes is strictly via message passing and state is private to the process.
Now processes can be managed with primitive operations but for reliability and robustness we need something more.
OTP (originally Open Telecom Platform) is a framework of middleware, libraries, and tools for Erlang and ships as an integral part of the Erlang distribution. Among the tools are the Observer
which we’ve seen before in an earlier post and which we will shortly use again below. But especially of interest here are some of the abstraction patterns that OTP provides for processes: agents, servers, supervisors, and tasks.
1a. (Generic) Server
Perhaps the key process abstraction in OTP is the generic server, or gen_server
(or, as it is known in Elixir, the GenServer
). This process behaviour provides the server of a client–server relation and can be used to maintain state, execute code, etc. Any module implementing this behaviour only needs to provide the functionality for a small known set of callbacks. The server pattern is managed by the behaviour module itself and has been exhaustively tested over the years so makes for a reliable application component.
Elixir provides further abstractions over the GenServer
: we have Agent
for state management, and Task
for asynchronous calculations.
1b. Supervisor
The second key process abstraction in OTP is the supervisor, or supervisor
(or, as it is known in Elixir, the Supervisor
). This process behaviour provides a supervisor for other processes and can be used to build up a hierarchical process structure called a supervision tree. Supervision trees provide fault tolerance and encapsulate how our applications start up and shut down.
Again Elixir provides a couple further abstractions over the Supervisor
: DynamicSupervisor
for managing dynamic child processes, and TaskSupervior
for supervising tasks.
1c. ‘Let it crash’
Indeed, so ingrained is the supervision tree model that Erlang (and Elixir too) has something of a ‘Let it crash’ mantra. Rather than defensively programming for all sorts of unknown conditions, some of which inevitably arise through dynamic combinations of circumstances which cannot be predicted nor (more worryingly) reproduced, Erlang instead puts worker processes under a supervisor which will restart any faulting process according to a predeclared supervision strategy. And it will do so transparently. (This is similar to how Erlang also allows for hot-code swapping, or software upgrades, to occur in situ on a running system – with no downtime.)
But this is not to say that Erlang skimps on checks and balances. It uses strong typing and compile time type checking, allows for guards on functions and for exceptions to be raised, and supports running of test suites. It is just more realistic about the context in which it will be executed.
This paradigm shift is an excellent fit to the networked environment where we must always expect the unexpected. Indeed, things happen. This is the true nature of an open world of computation.
1d. Examples in RDF.ex/SPARQl.ex
These process behaviours are so fundamental to applications that we also see examples of Agent
, GenServer
and Supervisor
patterns used in the RDF.ex
and SPARQL.ex
packages for various purposes: blank node management, SPARQL extension functions, etc.
2. Plan the approach
I aim to demo here the following abstractions:
GenServer
Supervisor
DynamicSupervisor
Our rough goal is to create an Elixir Application
that can be used to query DBpedia for random resources and save the links (together with labels and Wikipedia links) into GenServer
processes which can be created dynamically. The processes will be created under OTP supervision trees and will be automatically recreated when killed (manually, or otherwise). In this simple demo though we won’t attempt to preserve existing state across process restarts.
(I had a further notion of establishing the GenServer
processes as language-specific buckets, e.g. an English bucket, a French bucket, a German bucket, etc. But for now let’s just keep things simple.)
Let’s lay out an outline for the project:
- We’ll initially create the project.
- We’ll then implement a
GenServer
, show some callbacks and a public interface for those. - We’ll next demo the
GenServer
as a basic key/value store. - We’ll then add SPARQL queries into the mix using
SPARQL.Client
and save our results into aGenServer
. - Next we’ll put a
GenServer
under a static supervision tree and show how it is automatically restarted when it errors. - And lastly we’ll create
GenServer
processes dynamically.
3. Create a ‘TestSuper’ project
As usual, let’s create a new project TestSuper
with Mix but this time we’ll also use a --sup
flag to set up an Application
. (We’ll be using the Application
to auto-start our supervision trees.)
% mix new test_super --sup
Note that this resulted in two changes. We now have an extra application.ex
file under the lib/test_super/
directory which defines a TestSuper.Application
module. Let’s just ignore that for now.
% tree lib/
lib/
├── test_super
│ └── application.ex
└── test_super.ex1 directory, 2 files
We also have one extra line in our mix.exs
file which will automatically invoke the TestSuper.Application
.
def application do
[
extra_applications: [:logger],
mod: {TestSuper.Application, []}
]
end
Since we’re going to be later using the SPARQL.Client.ex
package let’s now declare our dependency in the mix.exs
file. We’ll also use the hackney
HTTP client in Erlang. And we want to generate some documentation so will include the ex_doc
dependency.
defp deps do
[
{:ex_doc, "~> 0.18", only: :dev, runtime: false},
{:sparql_client, "~> 0.2.1"},
{:hackney, "~> 1.6"},
]
And we add this line to the config.exs
file:
# config :tesla, :adapter, Tesla.Adapter.Hackney
We then use Mix to add in the dependency:
% mix deps.get
Let’s also clear out the boilerplate in lib/test_super.ex
and add in a @moduledoc
annotation.
defmodule TestSuper do
@moduledoc """
Top-level module used in "Robust compute for RDF queries" post.
"""
end
And also to simplify naming in IEx we’ll add a .iex.exs
configuration file.
% cat .iex.exs
import TestSuper
import TestSuper.Client
See here for the project TestSuper
code. (And note that the project TestSuper
code also includes documentation which can be browsed here.)
4. Add in a GenServer
We’re now ready to define a GenServer
. Let’s create ourselves a new module TestSuper.Server
and add that to a file server.ex
under the directory lib/test_super/
.
defmodule TestSuper.Server do
use GenServer ## Constructor # def start_link(_) ## Callbacks # def init(_), etc.end
Recall that a GenServer
is a process with an OTP behaviour and OTP defines the server functionality. All we need to do is to add in client-side functionality via the callbacks. And also, to start up the process.
The layout of this module has three main pieces. First there is a use GenServer
directive which will import the GenServer
behaviour into our module. We then have a process constructor start_link/1
which spawns a new GenServer
process and links to it. And lastly we have various callbacks, the only mandatory callback being init/1
.
So let’s now flesh this out.
defmodule TestSuper.Server do
@moduledoc """
Module providing server-side functions for `GenServer`.
"""
use GenServer ## Constructor def start_link(opts \\ []) do
case GenServer.start_link(__MODULE__, opts) do
{:ok, pid} ->
# register process name
num = String.replace("#{inspect pid}", "#PID", "")
Process.register(pid, Module.concat(__MODULE__, num))
{:ok, pid}
{:error, reason} -> {:error, reason}
end
end ## Callbacks def init(_) do
{:ok, Map.new}
end def handle_call({:get}, _from, state) do
{:reply, state, state}
end def handle_call({:get, key}, _from, state) do
{:reply, Map.fetch!(state, key), state}
end def handle_call({:keys}, _from, state) do
{:reply, Map.keys(state), state}
end def handle_cast({:put, key, value}, state) do
{:noreply, Map.put(state, key, value)}
endend
The only thing needed for start_link/1
is the call through to the GenServer.start_link/2
function. So here we spawn a GenServer
process and link to that process. But we’ll add in a more friendly entry point in a moment.
The other piece here is for registering a process name (which we’ll see later in the Observer
). Basically we’re just going to name the GenServer
process with our module name and a concatenated process ID.
Client processes can pass various messages , or requests, to a GenServer
which needs corresponding callback functions to be implemented. Now, there are two types of requests that we can send to a GenServer
: calls and casts. Calls are synchronous and the server must send a response back to such requests. Casts are asynchronous and the server won’t send a response back.
On the server side, we can implement a variety of callbacks to guarantee server initialization, termination, and handling of requests. We are just going to implement these three callbacks for initialization and request handling:
init/1
– intialize the serverhandle_call/3
– service acall/2
client request (for reads)handle_cast/2
– service acast/2
client request (for writes)
Since we’re going to use this GenServer
to manage a key/value store we’ll use a Map
to implement this. This is set up by the init/1
callback.
Often the client functions (calls) and server functions (callbacks) will be placed together in the server module. Here, however, we have made a cleaner separation and put our client calls into a TestSuper.Client
module and kept the server callbacks in the TestSuper.Server
module.
Here’s our TestSuper.Client
module calls.
defmodule TestSuper.Client do
@moduledoc """
Module providing client-side functions for `GenServer`.
""" ## Calls def get(pid) do
GenServer.call(pid, {:get})
end def get(pid, key) do
GenServer.call(pid, {:get, key})
end def keys(pid) do
GenServer.call(pid, {:keys})
end def put(pid, key, value) do
GenServer.cast(pid, {:put, key, value})
endend
This is just for demo purposes so we only define some basic read/write functions (get/1
, get/2
and put/3
). We don’t have any update or delete functions but they are easy to add.
So far, so good. We just need a more friendly way of starting our GenServer
. We’ll add this genserver/0
constructor to our main module TestSuper
.
def genserver() do
opts = [
]
case TestSuper.Server.start_link(opts) do
{:ok, pid} ->
IO.puts "TestSuper.Server is starting ... #{inspect pid}"
pid
{:error, reason} ->
IO.puts "! Error: #{inspect reason}"
end
end
We’re now all set.
5. Try out the GenServer by storing some key/value pairs
So, let’s try this out first with our client API.
iex(1)> pid = genserver
TestSuper.Server is starting ... #PID<0.219.0>
#PID<0.219.0>iex(2)> put(pid, :foo, "bar")
:okiex(3)> get(pid)
%{foo: "bar"}iex(4)> put(pid, :baz, 123)
:okiex(5)> get(pid)
%{baz: 123, foo: "bar"}iex(6)> get(pid, :foo)
"bar"
Now we can also inspect the guts of the GenServer
process with the Observer
tool we saw in an earlier post.
iex(7)> :observer.start
Now if we scoot over to the ‘Processes’ tab we can locate the GenServer
process, by sorting on the ‘Pid’ or ‘Name’ column: Elixir.TestSuper.Server.<0.219.0>
. This is the name that we registered for the process.
Also, under the ‘Current Function’ heading we see the current function identified as gen_server:loop/7
. So, we are indeed running a GenServer
process.
Now selecting the ‘State’ tab, we can see the actual state stored in the process and it’s an empty map.
If we now put a couple key/value pairs as before and again inspect the state we see now we have a map with entries.
So, that’s a GenServer
process storing key/value state.
6. Try out the GenServer by storing SPARQL results
Let’s add some support now to our TestSuper.Client
module for querying DBpedia. First, as in previous posts, we’ll store a SPARQL query dbpedia_query.rq
in our /priv/queries/
directory.
@priv_dir "#{:code.priv_dir(:test_super)}"@queries_dir @priv_dir <> "/queries/"
@query_file "dbpedia_query.rq"@service "http://dbpedia.org/sparql"## Queriesdef query() do
File.read!(@queries_dir <> @query_file)
end
The query/0
function just reads the query from the priv/queries/
directory. We can try this out.
iex(2)> query |> IO.puts
prefix dbo: <http://dbpedia.org/ontology/>
prefix foaf: <http://xmlns.com/foaf/0.1/>
prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>select *
where {
bind (12345 as ?id)
?s dbo:wikiPageID ?id .
optional { ?s foaf:isPrimaryTopicOf ?topic }
optional { ?s rdfs:label ?label }
filter (langMatches(lang(?label), "en"))
} limit 1:ok
This query will look for a resource with a dbo:wikiPageID
of 12345
. It will add in foaf:isPrimaryTopicOf
and rdfs:label
properties if present, and filter for English-language labels.
def putq(pid) do
{key, value} = _do_query()
IO.puts "#{inspect key} => #{inspect value}"
GenServer.cast(pid, {:put, key, value})
enddefp _do_query() do
# rewrite query with new random wikiPageID
rand = Integer.to_string(Enum.random(1..50000))
q = String.replace(query(), "12345", rand) # send query
{:ok, result} = SPARQL.Client.query(q, @service) # parse query result set
if length(result.results) == 0 do
{rand, "(not found)"}
else
id = result |> SPARQL.Query.Result.get(:id) |> List.first
s = result |> SPARQL.Query.Result.get(:s) |> List.first
label = result |> SPARQL.Query.Result.get(:label) |> List.first
topic = result |> SPARQL.Query.Result.get(:topic) |> List.first
{id.value, {s.value, label.value, topic.value}}
end
end
So here we have the first function client call putq/0
which will send the results from a query to a GenServer
for storing.
The second (private) function _do_query/0
does the actual querying and parses the result set. It first generates a new random dbo:wikiPageID
and replaces that ID in the query string. The modified query is sent to the DBpedia SPARQL endpoint using SPARQL.Client.query/2
. The result set is tested and if empty a tuple {rand, “(not found)”}
is returned. Otherwise the ?id
, ?s
, ?label
and ?topic
variables are parsed out of the first result and returned in the tuple {id.value, {s.value, label.value, topic.value}}
.
We can try this out in IEx.
iex(1)> pid = genserver
TestSuper.Server is starting ... #PID<0.227.0>
#PID<0.227.0>iex(2)> putq(pid)
49631 => {"http://dbpedia.org/resource/Turner's_syndrome", "Turner's syndrome", "http://en.wikipedia.org/wiki/Turner's_syndrome"}
:okiex(3)> get(pid)
%{
49631 => {"http://dbpedia.org/resource/Turner's_syndrome", "Turner's syndrome", "http://en.wikipedia.org/wiki/Turner's_syndrome"}
}iex(4)> get(pid, 49631)
{"http://dbpedia.org/resource/Turner's_syndrome", "Turner's syndrome", "http://en.wikipedia.org/wiki/Turner's_syndrome"}
Or for querying (and storing) in bulk:
iex(5)> for _ <- 1..5, do: putq(pid)
23367 => {"http://dbpedia.org/resource/Telecommunications_in_Pakistan", "Telecommunications in Pakistan", "http://en.wikipedia.org/wiki/Telecommunications_in_Pakistan"}
1130 => {"http://dbpedia.org/resource/Avicenna", "Avicenna", "http://en.wikipedia.org/wiki/Avicenna"}
10216 => {"http://dbpedia.org/resource/Eth", "Eth", "http://en.wikipedia.org/wiki/Eth"}
"37810" => "(not found)"
"29721" => "(not found)"
[:ok, :ok, :ok, :ok, :ok]
And we can get this back as:
iex(6)> get(pid)
%{
1130 => {"http://dbpedia.org/resource/Avicenna", "Avicenna", "http://en.wikipedia.org/wiki/Avicenna"},
10216 => {"http://dbpedia.org/resource/Eth", "Eth", "http://en.wikipedia.org/wiki/Eth"},
23367 => {"http://dbpedia.org/resource/Telecommunications_in_Pakistan", "Telecommunications in Pakistan", "http://en.wikipedia.org/wiki/Telecommunications_in_Pakistan"},
49631 => {"http://dbpedia.org/resource/Turner's_syndrome", "Turner's syndrome", "http://en.wikipedia.org/wiki/Turner's_syndrome"},
"29721" => "(not found)",
"37810" => "(not found)"
}
Note that this includes both the single result we stored earlier (49631
), as well as the bulk query results (1130
, 10216
, 23367
, 29721
, 37810
).
Now we can again inspect the state stored in the GenServer
process with the Observer
tool.
iex(7)> :observer.start
If we choose the ‘Processes’ tab, sort the processes by clicking on the ‘Pid’ heading , and select the GenServer
process we just created (#PID<0.227.0>
) we have the following screen:
Double clicking on this process and choosing the ‘State’ tab gives us this screen showing the state.
And if we click the link ‘Click to expand above term’ we get this:
So, that’s a GenServer
process storing key/value state for the results from our SPARQL queries.
7. Add in a static supervision tree
At this point we want to add in a supervision tree. Let’s first create ourselves a TestSuper.Supervisor
module.
defmodule TestSuper.Supervisor do
@moduledoc """
Module providing server-side functions for `Supervisor`.
"""
use Supervisor ## Constructor def start_link(opts \\ []) do
Supervisor.start_link(__MODULE__, [], opts)
end ## Callbacks def init([]) do
children = [
TestSuper.Server
]
opts = [
name: TestSuper.Supervisor,
strategy: :one_for_one
] Supervisor.init(children, opts)
endend
So, we also want a more friendly means to start this up. Let’s add this to the main TestSuper
module.
def supervisor() do
opts = [
name: TestSuper.Supervisor,
strategy: :one_for_one
]
case TestSuper.Supervisor.start_link(opts) do
{:ok, pid} ->
[{_, child_pid, _, _}] = Supervisor.which_children(pid)
IO.puts "TestSuper.Supervisor is starting ... #{inspect pid}"
IO.puts "TestSuper.Server is starting ... #{inspect child_pid}"
pid
{:error, reason} ->
IO.puts "! Error: #{inspect reason}"
end
end
All this does is to call through to our TestSuper.Supervisor.start_link/1
constructor function passing a couple options: a name and a supervisaion strategy. There are three defined supervision strategies (see the Supervisor
documentation for more info) but we are going to choose the simplest: :one_for_one
, i.e. one child process (the GenServer
) for our Supervisor
.
We also test the success of this operation and deal accordingly. And lastly we write out a report of the Supervisor
and GenServer
processes starting up together with their process IDs. (We use the Supervisor.which_children/1
function to get the child process ID.)
So, we can use the supervisor/0
constructor to create a new Supervisor
with an attached GenServer
. But we can simplify further and get the Application
itself to start up the Supervisor
automatically.
Now let’s have a quick peek at that TestSuper.Application
module. The one public function start/2
calls a private function _start/3
with a boolean argument which selects for a dynamic supervision tree on true
and a static supervision tree on false
. The intial setting is false
, i.e. selects for a static supervision tree.
defmodule TestSuper.Application do
@moduledoc """
Module providing the `Application` start function.
"""
use Application def start(type, args) do
_start(type, args, false)
end defp _start(type, args, flag) do
case flag do
false -> _static_start(type, args)
true -> _dynamic_start(type, args)
end
endend
I won’t say any more about the Application
behaviour here other than to note that they are process bundles having similar functionality. Our main reason for using the Application
to start up the Supervisor
is that we can then use the Observer
to view our Supervisor
and its GenServer
.
% iex -S mix
Erlang/OTP 21 [erts-10.0.6] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]Compiling 1 file (.ex)
Interactive Elixir (1.7.3) - press Ctrl+C to exit (type h() ENTER for help)iex(1)> :observer.start
:ok
Now if we choose the ‘Applications’ tab in the Observer
we’ll see this and see at left the list of applications being used here. Our test_super
application has a couple processes which are linked and link through to the Supervisor
and its attached GenServer
.
If we now double click the GenServer
process button Elixir.TestSuper.Server.<0.234.0>
we will open up in the ‘Process information’ tab.
And if we select the ‘State’ tab we’ll get this.
We can again try adding state as before and viewing here. (Note that the panel does not refresh and you’ll need to close and reopen to view changes.)
Now, since the GenServer
process was auto-started by the Application
, we’re going to have to create a process ID for our client functions which we can do with the pid/3
function:
iex(4)> pid = pid(0,234,0)
#PID<0.234.0>
And now we can send requests to the GenServer
:
iex(5)> putq(pid)
"495" => "(not found)"
:okiex(6)> putq(pid)
"3894" => "(not found)"
:okiex(7)> putq(pid)
23535 => {"http://dbpedia.org/resource/Photon", "Photon", "http://en.wikipedia.org/wiki/Photon"}
:ok
And if we close and repoen the ‘State’ tab we’ll see the new state we just stored.
So, what’s the Supervisor
doing for us? Let’s see.
If we go back to the ‘Applications’ tab on the main window and now right click on the GenServer
process, we’ll get this popup.
So, choose ‘Kill process’ and this dialog box pops up.
Just click ‘OK’, and …
Look at that. It’s the same as before (almost), but now the GenServer
process has been automatically restarted (albeit with a different process ID, #PID<0.338.0>
instead of #PID<0.234.0>
).
Now if there was any state stored in the process then that will have been erased. We’re back to a blank slate. We need other strategies to preserve the state. But the process itself survives, and any compute functions associated with the GenServer
are again available. Good as new.
8. Add in a dynamic supervision tree
Now the last example was for a static supervision tree. We could have added more GenServer
proceeses under a single Supervisor
, or even built up a tree with many levels by adding Supervisor
processes under a Supervisor
.
But now let’s look at how to create a dynamic supervision tree. Let’s first create ourselves a TestSuper.DynamicSupervisor
module.
defmodule TestSuper.DynamicSupervisor do
@moduledoc """
Module providing server-side functions for `DynamicSupervisor`.
"""
use DynamicSupervisor ## Constructor def start_link(opts \\ []) do
DynamicSupervisor.start_link(__MODULE__, [], opts)
end ## Callbacks def init([]) do
opts = [
name: TestSuper.DynamicSupervisor,
strategy: :one_for_one
] DynamicSupervisor.init(opts)
endend
And again we also want a more friendly means to start this up. Let’s add this to the main TestSuper
module.
def supervisor_d() do
opts = [
name: TestSuper.DynamicSupervisor,
strategy: :one_for_one
]
case TestSuper.DynamicSupervisor.start_link(opts) do
{:ok, pid} ->
IO.puts "TestSuper.DynamicSupervisor is starting ... #{inspect pid}"
pid
{:error, reason} ->
IO.puts "! Error: #{inspect reason}"
end
end
This supervisor_d/0
constructor function is pretty much the same as the supervisor/0
constructor.
But again we can just use the Application
to start up our DynamicSupervisor
process. For now let’s just flip the false
arg in the TestSuper.Application
module to true
on the start/2
function to read as:
def start(type, args) do
_start(type, args, true)
end
And let’s restart IEx.
% iex -S mix
Erlang/OTP 21 [erts-10.0.6] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]Compiling 1 file (.ex)
Interactive Elixir (1.7.3) - press Ctrl+C to exit (type h() ENTER for help)iex(1)> :observer.start
:ok
Now this time we only have the DynamicServer
. There are no child GenServer
processes attached.
Let’s add a GenServer
.
iex(2)> genserver_d
#PID<0.266.0>
Now we can see this atached. And, in fact, this looks very similar to the previous static supervision tree example.
Let’s add another GenServer
.
iex(3)> genserver_d
#PID<0.271.0>
And now we can see two child GenServer
processes for the DynamicSupervisor
.
Let’s go crazy and add ten GenServer
child processes.
iex(4)> for _ <- 1..10, do: genserver_d
[#PID<0.277.0>, #PID<0.278.0>, #PID<0.279.0>, #PID<0.280.0>, #PID<0.281.0>,
#PID<0.282.0>, #PID<0.283.0>, #PID<0.284.0>, #PID<0.285.0>, #PID<0.286.0>]
Yup.
That’s right. The DynamicSupervisor
here is managing a dozen GenServer
processes, but could just as easily be twelve thousand, or more.
Summary
I’ve shown here in this post how to manage fault tolerance in Elixir with supervision trees. We first reviewed the process model for Erlang (Elixir) and then discussed some of the abstraction patterns that OTP provides for processes: agents, servers, supervisors, and tasks.
We then developed a demo to show how the GenServer
process can be used to store state. In this example we used a map to maintain a key/value store. We also made use of the SPARQL.Client.ex
package to query DBpedia and to save our results into the GenServer
.
We then showed how both static and dynamic supervision trees may be created for managing child processes using the Supervisor
and DynamicSupervisor
patterns. Using the Observer
we showed how child processes can be killed and get restarted automatically by the Supervisor
.
It is this support for fault tolerance and distributed compute that makes Elxir such a fascinating candidate for semantic web applications.
Endnote
Earlier I had in mind a kind of picture of distributed compute and thought that the banner image to this post was an excellent metaphor. And then I came across this image below by Martin Stannard using the Scenic UI framework for Elixir where each triangle is a separate GenServer
process. (See his Twitter post which shows an animated version, and GitLab repo for the project.)
See here for the project TestSuper
code. (And note that the project TestSuper
code also includes documentation which can be browsed here.)
This is the fourth in a series of posts. See my previous post ‘Working with SHACL and Elixir’.
You can also follow me on Twitter as @tonyhammond.