Comparing Linked Data Triplestores
Virtuoso, GraphDB, Blazegraph, Stardog, AnzoGraph and more…
There are many Triplestores available and it is difficult to decide which is best for each use-case. In this article I tie all of my previous posts together and explore the pros and cons of some of the most popular triplestores.
I will be using RDF and SPARQL queries that I have created and discussed previously. If interested, I have covered:
Why Use Linked Data to Tackle Big Data Problems
Understanding Linked Data Formats
Creating Linked Data with OpenRefine
Constructing SPARQL Queries
Edit (22/01/2019): AnzoGraph completes C++ compilations on the first run of a query so was unfairly represented originally. I have therefore evaluated AnzoGraph once again and updated the results throughout.
The Triplestores:
I have tried to cover a wide array of Triplestores and filter down as we explore deeper. These were all deployed on the same server, one at a time, with default settings where possible. To begin in no particular order, I cover:
Blazegraph
Version: 2.1.4
Deployed using Docker from this repository.
Predicate used in full text index query: bds:search
Stardog
Version: 5.3.5
Installed directly following the official instructions.
Predicate used in full text index query: <tag:stardog:api:property:textMatch>
Virtuoso
Version: 07.20.3217
Deployed using Docker from this repository.
Predicate used in full text index query: bif:contains
GraphDB
Version: graphdb-free-8–3–1
Deployed using Docker from this repository.
Predicate used in full text index query: luc:myIndex
AnzoGraph
Version : r201901202057.beta
Deployed using Docker following official instructions.
AllegroGraph
Version: 6.4.3
Deployed using Docker from this repository.
MarkLogic
Version: 9.0–7
Deployed using Docker from this repository.
Apache Rya
Rya is still incubating so not quite mature enough to compare fairly. In the future, when suitable, I will do a further comparison including Rya.
RDF Loading Speed:
It may not be the most important factor, but loading RDF into the triplestore is one of the first things we will need to do. As mentioned above, the data we are loading was created in one of my previous posts and can be downloaded here.
Right from the start, AllegroGraph takes an extraordinarily long time to populate. Marklogic is overshadowed by AllegroGraph but still takes 2 minutes to load the data in RDF/XML format! Due to these extreme loading times and other factors (discussed throughout this article) I eliminated these two from deeper exploration.
Inspecting the chart, without AllegroGraph and MarkLogic, we can see that loading RDF in RDF/XML format is generally faster than in N-Triples. AnzoGraph does not support RDF/XML as it is a much newer triplestore and RDF/XML is an older format that is dropping in popularity.
Outstandingly, AnzoGraph loads the data in N-Triples format in just 18753ms, significantly beating all others. It appears that by excluding older RDF formats, AnzoGraph can optimise loading speeds for the newer formats. GraphDB is the only other triplestore that is faster to load N-Triples than RDF/XML. Blazegraph, Stardog and Virtuoso are all much faster at loading RDF/XML (over twice as fast in Stardog and Virtuoso). Why this is the case I do not know, but this may be important to consider if you will be loading lots of data regularly. Remember from one of my previous articles, if you are loading data using data streams you cannot use RDF/XML so don’t fall into that trap.
SPARQL Query Speeds:
In an article about constructing SPARQL queries, I created eight queries of varying complexity to run on the data I just loaded above. These queries can be found here. In summary, the queries are:
Q1: List the names of every gold medallist (excluding duplicates).
Q2: List the names of every athlete, with at least one medal, alongside their total number of medals (sorted by the number of medals).
Q3: List each country alongside the average height and weight of its athletes (sorted by the average height).
Q4: List each year alongside the oldest Judo competitor in that year
Q5: List every athlete with “louis” in their name alongside the city and season that they competed in. This query uses regex.
Q6: Get a list of every city that an athlete from Serbia and Montenegro competed in. Count the number of males and females that ever competed in one of those cities and return these counts.
The next two are a little more complex:
Q7: List every sport’s name that has a stored team size in DBpedia and return these sizes next to the corresponding sport.
QFTI: List every athlete with “louis” in their name alongside the city and season that they competed in. This query uses each triplestores full text index feature unlike Q5.
I ran each of these queries three times and chart the mean of these times.
The first six queries are very standard and should run with no problems on every triplestore. In the following chart I have summed the result times to get an overall idea of each triplestore’s performance:
This aggregation suggests that Virtuoso and AnzoGraph are the ones to watch as they are significantly faster (almost 3 times faster than the next best; Stardog) than all of the other triplestores over six diverse queries. Stardog and GraphDB are similar to each other and Blazegraph is a lot slower overall. AnzoGraph was by far the fastest at loading data so this result is promising.
It is possible that AllegroGraph and MarkLogic take additional loading time to optimise the graph for queries so in a future article (once problems I discuss later are resolved) I will test this hypothesis.
This summary is nice to get an idea but let’s explore each triplestore’s performance per query. The fastest queries to complete were queries 1 and 4 so we will investigate these first.
Interestingly, GraphDB is the fastest to complete these simple queries with Virtuoso being the slowest to finish query 4 (still only 90ms). Stardog and AnzoGraph take a particularly long time to complete query 1. Stardog for example took 412ms to complete compared to GraphDB which took just 16.7ms. This disparity to answer the exact same query could be significant but, as we saw above, their total time to complete the first six queries was similar.
Next, queries 2, 3 and 5 are a little more complex:
With the exception of query 2, AnzoGraph is faster than the others and Blazegraph is the slowest. Queries 2 and 3 both require more operations (count, mean, group, order, etc..) than queries 1 and 4. This suggests that GraphDB is the fastest at retrieving entities but Virtuoso is far better at running operations efficiently.
Both Blazegraph and GraphDB are much slower to complete query 3 than query 2 which suggests that they are not optimised for mathematical queries. In contrast, AnzoGraph seems to be the opposite, completing query 3 much faster than query 2.
Query 5 is a text query using regex which AnzoGraph performs the fastest, followed by Stardog. Blazegraph, which is noted to be very good at text queries, performs the slowest. This is because this query uses standard regex and not its full text indexing feature. This is shown later in the full text index query. It is also important to note that AnzoGraph does not have have a built-in full text index which is likely why it optimises regex queries so much more than the others.
If you want to use multiple triplestores, these regex results are useful to ensure your query will run on all of them without any query tailoring.
Query 6 is a nested query that traverses almost the entire knowledge graph twice.
These results really show the power of Linked Data. If you work with relational databases you will know that a query with this many joins would take minutes if not hours.
Virtuoso and AnzoGraph complete this query in an exceptional time. Virtuoso in 128ms and AnzoGraph in just 105.7ms! Stardog and GraphDB follow at around 600ms and Blazegraph trails behind, taking over 3 seconds to complete. If you plan to regularly send deep queries, Blazegraph may be a poor choice.
Query 7 is a federated query, meaning it links to an external knowledge graph which in this case is DBpedia (running on Virtuoso). This means that the time does depend on the response of a public (and popular) triplestore. As mentioned, I did repeat the queries multiple times so the effects of this variation should be minimised.
AnzoGraph was the fastest and surprisingly, Virtuoso the slowest to complete query 7. The poor performance from Virtuoso results from how it stores the data. The linked data we loaded contained duplicates due to the method I used to create it. Most triplestores never store duplicates whereas Virtuoso stores duplicate literals. This means that the sport entities have duplicate labels which are sent to the DBpedia endpoint. DBpedia then has to find the team size of each sport multiple times, hence the increased response time.
Edit (17/12/2018): I cleared my Virtuoso instance and loaded the data without duplicates and re-tested this query. Here is the updated chart:
I left both charts to highlight that this can happen in Virtuoso.
It may seem like Stardog responded incredibly fast but it has no bar because it timed out every time. This is due to the fact that Stardog does not optimise federated queries adequately.
Finally, in the full text query I want the same results as in query 5 but instead of using regex, the query is customised for each triplestore. This is done by using the predicates listed above in the Triplestores section.
AnzoGraph by itself does not implement it’s own full text index queries so does not appear in this chart. If you want this feature you have to integrate AnzoGraph with its counterpart, Anzo.
Virtuoso once again outperforms the other triplestores followed by Blazegraph and GraphDB. Stardog took significantly longer but still only took 127.7ms.
If you scroll up and inspect the query 5 results, you will notice that AnzoGraph was the fastest and Blazegraph was the slowest. This is because Blazegraph focuses on full text index queries so has not optimised for regex queries. Blazegraph took 2586.7ms to complete query 5 but only 42.7ms to return the exact same results using its full text index. To point out, Stardog is not slower to complete this query (127.7ms) than query 5 (509.3ms), this is just not their focus.
AnzoGraph and Virtuoso were not the fastest to complete every query but overall they come out on top if speed is your priority. Just note that literals can be duplicated in Virtuoso which can significantly slow down federated queries.
Deployment:
Blazegraph
I deployed Blazegraph using Docker from this repository that has very clear documentation. You can load data into Blazegraph in multiple ways but with large RDF files, data must be accessible from within the container. You can docker cp your data into the container but we just set up a volume and had no problems.
Overall: Very easy to deploy and load data.
Stardog
Stardog was also very easy to deploy. You have to download a zip and licence key, which you have to make sure is in the right place, but that is as complex as it gets. One command to start Stardog and one more to load your data.
Overall: Very easy to deploy and load data.
Virtuoso
Similar to Blazegraph, Virtuoso is very easy to deploy following these instructions. These also detail volumes for loading data. There are some quirks with isql-v to load your data but following the documentation, it is relatively simple.
Overall: Very easy to deploy and fairly easy to load data.
GraphDB
Again, using this repository, GraphDB deployed smoothly. The instructions do not detail volumes so adding -v /path/to/data:/root/graphdb-import to the docker run command is what you need to ensure your data shows up on the interface. Again, this is only required if you are loading very large data files as you can select a local file through the user-interface.
Overall: Very easy to deploy and load data.
AnzoGraph
Using the official Docker image of AnzoGraph, everything starts fine. There are some bugs however and if you hit one and want to restart, everything breaks. This is true even if you restart a completely functioning container. As you can see from our results, you can deploy, load and query AnzoGraph as long as you never hit a bug or restart the container.
Edit (22/01/2019): These problems are now fixed in the latest Docker Image.
Overall: Loading data is simple enough but everyone needs to restart at some point so the official image needs fixed.
AllegroGraph
Using the official Docker repository, AllegroGraph was easy to deploy but unlike the others, is not the smartest when loading data. The free version of AllegroGraph can store up to 5,000,000 triples. The data I was loading contained more than this if you count the duplicate triples (which are not stored). If I removed the duplicates, I could not fairly compare the loading and query times against the other triplestores which is another reason I decided not to include AllegroGraph in this round of speed tests.
Overall: Easy to deploy but a pain to load without cleaning RDF first.
MarkLogic
Using these instructions, MarkLogic is fairly easy to deploy. You need to download an installer and edit a few lines of code but it is well documented. Loading data was relatively easy but it is clear that MarkLogic is made for consultants to use. I could not find an easy way to query the data. Maybe I missed something obvious but neither the user interface or documentation provided me with an answer so this was an additional reason to exclude MarkLogic from this round of speed tests.
Overall: Fairly easy to deploy and load data but querying that data is a mystery.
User Interface:
Obviously this section is more subjective so for clarity I consider these factors as important:
- Ease to load data
- Ease to query data
- Ease to explore data
- Ease to monitor system
- Ease to manage data
I have excluded AllegroGraph and MarkLogic from my detailed discussion here as poor performance in previous tests can not be outweighed by a nice interface. In short, Allegrograph has a fairly nice interface whereas MarkLogic’s UI is very un-intuitive.
Edit (22/01/2019): I also don’t cover AnzoGraph’s UI as it originally appeared to perform poorly. Its interface is very easy to use! Data loading, data querying and system monitoring are all very intuitive. Data is managed using graphs and data exploration could be improved if you compare it to GraphDB. I will discuss it more in depth in the future.
Blazegraph
Blazegraph has a relatively nice user-interface and I consider it easy to find whatever I need. There is a tab called update with multiple options to load your data. If a method is not suitable (data file too large for example) the error message clearly explains alternative methods.
The query tab is also simple and intuitive. Results are shown in simple tables and queries are stored in a history which is always useful when building applications. Query errors generally explain the problem which helps when debugging queries (that have syntax errors). For other errors (such as a spelling error in a predicate), Blazegraph has an explore tab that is opened whenever you click on a URI in query results.
You can also manually type a URI in the explore tab which will show every link to and from that given entity. This explorer is a bit clunky but really helps with query creation and debugging as you can quickly see what predicates are linking entities.
Monitoring Blazegraph is a simple couple of tabs (status and performance) but it does have useful information such as status of running queries, system up-time, etc… Extensive logs can also be accessed using the docker logs command.
You can split your RDF by creating ‘Namespaces’ in Blazegraph which are extremely useful to keep applications separate and queries fast. You can enable geospatial mode or full text mode for example over a portion of your data instead of the whole lot.
Stardog
Compared to Blazegraph, Stardog has an overall prettier interface. Loading data can not be done through the UI however but is a simple command. Once you have a database (equivalent of namespace), you can update it with an update SPARQL query.
The query panel is very similar to the others, queries can be typed into a text box and results display in a table. Errors are again fairly understandable to sort syntax errors and queries can be saved for later use.
Exploring data (called browsing in Stardog) is very easy and has a much nicer design. To show the full extent of this, here is a screenshot from the official docs (displaying all the niceties):
Like Blazegraph, monitoring Stardog can be done through the UI with more detailed information stored in the logs.
As mentioned, you can load data into separate databases to keep your data organised. Similar to Blazegraph, features like full text indexing can be turned on or off per database. Stardog also allows the activation and deactivation of databases individually if that is something you need.
Virtuoso
Compared to the others, Virtuoso has the worst interface. This has the benefit that you can be sure it will work in any browser. You cannot load data using the UI, this must be done using isql through your terminal. This is easy enough but removes the options to upload data held on your local machine.
Querying is again standard, a little hard to find when you first use Virtuoso but once there it is a text box like the others. Results are displayed in a table like the others and errors are relatively clear.
Annoyingly (really annoyingly), Virtuoso does not have a nice implementation of data exploration. You have to write a query every time you want to remember what predicate is used to link anything. For development, this is very time consuming! If you are constructing a large query for example, you have to keep copying it somewhere so that you can run tiny queries to explore your knowledge graph. In Blazegraph these are two separate tabs which keeps exploration and construction isolated from one and other. For this reason alone, I avoid developing with Virtuoso.
Monitoring the system is quite extensive in Virtuoso and again, there are detailed logs if you need.
Data is organised using graphs, I do not find this as simple to manage my data as I can in Blazegraph or Stardog. They work similarly but in practice it is not as intuitive through the UI. One benefit is that you can use graphs as a variable in your SPARQL queries. In the others, you can use SERVICE queries to query multiple namespaces but in Virtuoso you can construct queries that for example: “Get all graphs that contain people entities and return the number of people in each of these graphs”. If you have many graphs, this can really simplify cross-graph queries.
GraphDB
Finally, GraphDB has a very pretty interface. Loading data is very intuitive and there are many options so you can do this however you want.
Querying is similar but again, very nice to look at. The text box implements syntax highlighting, auto-prefixing and can be saved easily. Results can be displayed in a table but, through integration with Google charts, can be visually output also.
As you may have noticed, I really appreciate the ability to explore my data during development and GraphDB is by far the winner for this. Data can be visually explored which speeds up query construction significantly. You can very quickly look at the section of the graph you are working with. If your query is not working you can instantly see if there is a spelling error in a predicate for example. This is an example of the visual explorer:
Monitoring the system in GraphDB is once again similar to the others, logs can be easily accessed using the docker logs command if you ever need more information than is provided through the interface.
Similar to Virtuoso, GraphDB uses graphs to organise data. Though the interface is much easier to manage than in Virtuoso. Every dataset is given a new graph by default unless you actively tell it to combine with an existing graph. Graphs can be managed and deleted from a table which ensures clarity to the user.
Authentication:
All of the triplestores apart from Blazegraph have an authentication layer built in. GraphDB is off by default but you can turn security on and set up users and roles. Stardog does have its authentication on by default but you should change the default username and password to make use of it. Virtuoso, AnzoGraph, AllegroGraph and MarkLogic have security on by default. If security is of high importance, it is probably best to avoid Blazegraph.
Conclusions:
Depending on your priorities, there are a few viable options but in my opinion I recommend the following:
- GraphDB for development while you are constructing your application.
- AnzoGraph or Virtuoso when you move into production if speed is important.
The reason for this is, the extra time your queries take to run does not come close to the amount of additional time you have to spend exploring and debugging your data in Virtuoso. Once your queries are built, then you can move to Virtuoso or AnzoGraph to take advantage of their speed.
This is very simple to do as SPARQL queries are standard so you just have to change endpoint. If you are using a feature like full text index search you may have to switch a few predicates but this is very quick and easy to do with a find and replace.
Notes:
- Needing to use an old browser? Use Virtuoso.
- If regularly loading large batches of data, use RDF/XML if running Virtuoso, Blazegraph or Stardog.
- Remember however, RDF/XML is not suitable for streaming.
- Literals can be duplicated in Virtuoso which can significantly slow down federated queries.
In the future, when Apache Rya becomes more stable, I will compare triplestores with the no-duplicate data so that AllegroGraph and Rya can be added.
The results in this article can be found here.
Next I will increase the number of triples to see how these triplestores perform on a larger scale.
