Android DBMS Performance Comparison Using the Jetpack Benchmark Library

There are plenty of Android database management systems you can choose from and you will probably pick one based on multiple factors: intuitiveness/ease of use, flexibility, documentation, performance.. However, if exactly the performance is the major factor, this article might have some answers for you.

TESTING SETUP

In this research, I have benchmarked the speed of executing the CRUD operations on various datasets for six highly popular Android DBM systems:

• SQLite,
• Room Persistence Library, v 2.2.5,
• DBFlow, v 4.2.8,
• GreenDao, v 3.2.2,
• Realm, v 6.0.2,
• ObjectBox, v 2.5.1.

The benchmarks were done under the standardized conditions. Each DBMS was tested:

• using the same datasets,
• using the same code-base,
• multiple and the same number of times (49 measures per benchmarked operation per dataset size, with the total of three runs for each test class),
• on the same device (Google Pixel 2, Android Q) via the same host,
• while the device was in the air-plane mode,
• while there were no other applications running in the background,
• in the sustained performance mode,
• using the Android Test Orchestrator to ensure the minimum shared state between the tests.

The datasets used for testing contain 10k, 20k, 30k, 40k and 50k cities. At the moment of conducting the research, I have experienced some issues while running the performance tests with the current version of Android Studio (3.6.3); therefore, I used the Android Debug Bridge (adb) for both running the tests and generating the detailed benchmark reports. However, I gave the tests a run with the newly released version of Android Studio (4.0) and everything seemed to be working fine, at least for the base test runs.

As mentioned before, the measurements were made using the Jetpack Benchmark library. Even though I am leaving the library as a topic for another blog, I will quickly brief you about some advantages you can benefit from by using it:

• it runs each measurement several times,
• in addition to the multiple measurement runs, you can generate the detailed report for your benchmark results containing different statistical parameters related to the measurements, like the minimum, maximum and median time,
• if using with adb, you will get some extra statistical data in the terminal output, like the mean value and the standard deviation,
• while using with devices which support the sustained performance mode, the library will use a combination of this API and launching its own activity to both prevent thermal throttling and stabilize results.

In conclusion, there was 147 measurements made per operation (per dataset size), resulting in total of 2940 measurements made per DBMS and 17 640 measurements for the entire research, which should make the results reliable.

At the end, I ended up with plenty of tables following the later pattern.

The mean value of all the runs in milliseconds is used for the further comparison. If you want to fully preview the detailed results, check this sheet.

INDIVIDUAL RESULTS

Each DBMS will be presented with its own, individual results first, and the mutual comparison will be done afterwards.

SQLite:

Comparing to the other relational DBM systems, the times for executing the create and update operations are almost identical.

Room:

There is a clear separation between the times necessary to execute each of the CRUD operations. The speed of deleting the rows is blazing fast.

DBFlow:

Has the slowest reading speed of all relational DBM systems. For 50 000 rows, the times necessary for reading and creating the rows are almost the same.

GreenDao:

The slowest relational DBMS in terms of updating the data, but the fastest in terms of reading the data from the database.

Realm:

No, the Read line is not missing here. Moreover, the speed for reading the data is so fast that its representation on the chart is being covered with the Delete line. Contrary, requires the most time to perform the Create operation, compared to all of the benchmarked DBM systems.

ObjectBox:

A quite nice surprise in terms of performance. This was by far the fastest DBMS taking the overall performance into account.

MUTUAL COMPARISON

In this section, the individual results will be mutually compared. For each operation a rank will be formed where database management systems will be positioned from the fastest to the slowest. To form the rank, a coefficient is formed by ranking the DBM systems from the fastest to the slowest for the executed operation per dataset size. For instance, the systems were ranked for reading the 10k rows from the database. They were ranked again for reading 20k rows. The procedure was repeated for all dataset sizes. The final ranking for the operation was created by calculating the average of the previously formed ranks. If the average rank values for two or more DBM systems were the same, a tie was called.

Rank: ObjectBox, SQLite, DBFlow, GreenDao, Room, Realm.

Rank: Realm, ObjectBox, GreenDao, Room, SQLite, DBFlow.

Rank: ObjectBox, Realm, SQLite, DBFlow, Room, GreenDao.

Rank: Realm, Room, ObjectBox, DBFlow/GreenDao (tie), SQLite.

At the end, the average rank was calculated, giving the final, general ordering of the tested database management systems (from the fastest to the slowest):

ObjectBox, Realm, Room/SQLite (tie), DBFlow/GreenDao (tie).

CONCLUSION

The scope of the research includes benchmarking of the CRUD operations. The benchmark was made using the Jetpack Benchmark library with the total of 17 640 measurements made. Currently, the object-oriented DBM systems show better results for performing the most of the tested operations. However, that does not have to be the case if more complex operations were introduced.

Once again, the presented rankings are related to performance only and do not cover various other factors which may determine which DBMS would suit your project the best.

You can check the entire benchmark code on my GitHub.

Hope this was helpful! Let me know what is your DBMS of choice in the comments. :-)