MongoDB Driver performance in several languages

At Clarity AI, we like to make decisions based on data, whenever that is possible. When that information is not available then we have to gather it ourselves.

That is exactly what we do in our company when we miss some sustainability data and that’s also what we like to do in order to have better context in our technical discussions within the Tech team.

And what discussion is more exciting for any team of programmers than the typical “my language is better than yours”? ;)

Internet forums, chats and social networks are flooded with flame wars about which is the best programming language, and it’s quite usual to find messages like “programming language X is much faster than Y, that’s a fact!…”, “language Y consumes much less memory than Z”, etc…

Those assertions are just too generic and most of the time, lack enough context to be taken seriously. Usually we just repeat something we have heard from someone we know or maybe we have read somewhere else, some other times it’s just a widely accepted notion associated with this or that language.

My personal opinion is that there isn’t a language (or tool) that’s good for everything, and that there are very few programmers out there with enough experience in all of them to be able to make a valuable assertion based on his/her own developments.

The fact that there’s no Swiss Army Knife is the reason for having so many languages available out there, which is actually quite good for programmers in terms of options to choose from.

That said, I wanted to check if I could perform some kind of simple benchmark which I could use to confirm or deny (or at least to have some hints) some assertions I heard or read about. Things like “Python is slow”, “Java eats all of your RAM”, “C++ is the fastest thing you can use”, “Rust has the lowest memory footprint” …. and so on.

Let’s get some numbers

I thought that something like storing and reading Data from a DB could be a good test, as it was simple enough but at the same time it had a bit of everything: Data transformation, buffering, creation and destruction of a lot of data structures in memory, networking, use of an external lib (the driver) …

I decided to try MongoDB, as it’s one of the data stores that we use in our company, so maybe we can somehow use the results of the test in order to make future technical decisions.

The idea is to do a huge amount of insertions in a collection and then fetch all inserted data. To make things a bit more interesting we’ll do the insertions using many threads at the same time. Fetching the data after the insertion will be single threaded, though.

Please note that this is not a complete test of each of the languages, but more like “a sample” of how they behave while performing something demanding in terms of memory and CPU like creating and fetching a good amount of data.

Most likely the only real conclusion that we will get after the test is which MongoDB driver flavor is in better shape, but even finding only that is something that will be worth investigating.

So let’s start, and remember that you can find a link to the code used to write this article below in “The Code” section.

Benchmark Description

• We have a collection of “books”. Each book having an “author” and a “title” field. There are only 4 different books: “Dune” (1%), “I, Robot” (33%), “Foundation” (33%), “Brave New World” (33%)
• We’ll insert 5 Million book documents in Mongo, using 8 threads (One Mongo Client and 625K documents per thread).
• Then we’ll do a search looking for all {“author”: “Isaac Asimov”} books (3.3 Million documents) and will fetch them sequentially in a single thread, storing the “title” and “author” in strings and then summing up the length of those strings for all docs.

Rules

• Make the code as simple and standard as possible. Just as anyone would do while following the tutorial of the driver of each language. Don’t take advantage of any personal experience to improve times, memory consumption …
• Try to repeat the steps exactly, create the same number of objects and in general reproduce the same behavior for each language, so the comparison of all different language implementations is meaningful. (Sometimes this isn’t possible, e.g: There is no real multi-threading in Node nor Python)
• We’ll configure the MongoDB client with a Batch size of 1000 documents so all tests are requesting documents to the DB at the same pace.

Chosen languages

The test will be implemented in 7 different programming languages:

C++, Go, Java, .NET (C#), Node, Python and Rust

The reasons for choosing these languages are:

• official support of the driver
• current or potential use inside the company
• acceptance by the developer community
• my own familiarity with some of them.

We’ll use the latest stable version of the driver for each language. At the time of writing these lines those versions are:

• C++: mongo-cxx-driver-r3.4.2
• Go: mongo-driver v1.5.1
• Java: mongo-java-driver:3.12.8
• .NET: MongoDB.Driver 2.12.3
• Node: mongodb 3.6.6
• Python: pymongo 3.11.3
• Rust: mongodb 1.2.0 (sync)

Some languages require different approaches to achieve the same thing

First difficulty we found: Not all languages manage the same concepts, so to make things as similar as possible, we applied the following “tweaks” in code:

• Go: Use goroutines instead of Threads (Goroutines are usually faster)
• Node: Use async calls and await instead of Threads. So all inserts are actually done using the same client and same thread

The hardware

The benchmark was ran in my personal laptop (i7–6th generation, 24 GB RAM), against a dockerized MongoDB 4.2 also running locally, so network times are negligible but on the other hand we have to take in account that the same machine has to cope with running the DB server and the client concurrently.

The code

You can check out the source of all tests as well as the full results and numbers on this Github repo.

The Results

And finally, the interesting part, let’s get some numbers that we can compare with each other in order to get some conclusions.

We are going to compare, for the two main stages of the tests (insert data and fetch data) these three metrics:

• Max CPU%
• Max RAM
• Execution time in seconds

Max CPU percentage used in each step

Max RAM memory used in each step

Total time in seconds spent in each step

Conclusions

Immediate conclusions from the numbers

There are some immediate conclusions that we can reach after a quick review of the previous graphs:

• C++ is, by far, the absolute king of performance and resource optimization.
• Java is the only real competitor of C++ when talking about performance, with almost the same figures. But if we take in account the memory footprint, it consumes 10 times more memory.
• Go is the second best all-around after C++, with very good memory consumption taking into account it’s using a garbage collector.
• Node and Python are not multi-threaded and that affects their comparison with the rest on the Insert stage dramatically. Python is actually supposed to be multi-threaded but in practice it turns out that it isn’t.
• Rust performance is really disappointing compared with the expectations before the test, especially for a language that is focused precisely on being fast. The language is still very young, so it looks like the official driver still has some room to improve.
• .NET performance is comparable to Java but a bit worse. However its memory footprint is much better (less than half).
• Almost all languages are better at fetching than inserting data, but the particular case of Python is very surprising, as it’s 6 times slower inserting than getting data. Results are even worse than those of Node, that it’s the other single-threaded test, but probably the difference is caused by to the non-blocking async calls of Node.

Other interesting conclusions

What if we take into account how was the experience of creating the code for each of the benchmark tests?

Here is where personal opinions come to place. It’s absolutely not the same to be used to the language idioms, tools, libraries than being a newbie and just follow the tutorial from the driver web page and complement that with some Stack Overflow examples (exactly what I did for most of the languages).

In saying this, and taking in account that I’m mostly a Java developer and these are my personal impressions, these are the implementations I found harder to create and test:

• C++: I needed to compile the driver myself, including compiling some dependencies like libmongoc, install it locally… It’s not really straightforward, especially compared with the rest of languages that have automatic dependency management. It felt like the Old Times and that isn’t necessarily a good thing.
• Go: Even when multi-threading is easier to do than for the rest of languages, due to using goroutines instead of Threads, I found some things a bit awkward. Concepts like “defer” or inserting objects of type “interface” instead of maybe directly “Books”... Not a big deal, but also not really intuitive from my point of view and sometimes I was just copy pasting code without really understanding why it was needed there.

On the other hand, these are the implementations I found easier:

• Python: The code is probably the smallest one. Directly to what you need, no verbosity, integrates perfectly with JSON… Also easy to test
• Java: No surprises here, as I am a Java developer. However, I found some points harder, like testing, when compared with most of the other languages (especially those that generate binaries), as you need to use the JVM, take care of the classpath (or create an executable JAR file)…

Final thoughts

So, well, after checking the results, are we going to change all of our code base to C++ now that we know that is faster and more efficient? No, we are not.

Then, are we going to migrate from Java to C++ at least the section of our code base that access MongoDB so we save most of the memory consumed by those processes? Probably neither.

Even if we changed only a part of it, it would have a cost in development and future maintenance, we would need to maintain 2 different development environments, and we don’t have enough people with experience (or even interest) in C++.

We won’t really notice a big difference in performance and we don’t currently have any problem with memory in our servers, so even when we always look to improve everything, having the minimal possible footprint isn’t our maximum priority right now, while there are so many new features yet to implement.

However, we now have some hints about alternatives we could use in case we need to dramatically decrease the demand for memory of some specific and critical parts of our services, so who knows…