ScalarDB 3.11 has been released!

Recently, we released ScalarDB 3.11, the latest minor version of ScalarDB. This new version introduces several features aimed at making ScalarDB easier to adopt.

New features include:

• A table import function to enable the management of existing relational database tables within ScalarDB.
• Cross-partition scan, allowing the execution of arbitrary WHERE clauses in ScalarDB SQL.
• Support for join operations (JOIN) in ScalarDB SQL.
• Other enhancements to ScalarDB SQL.

These functions make it easier to migrate applications that use relational databases to ScalarDB-based applications. Additionally, ScalarDB Cluster introduces new authentication and authorization mechanisms.

In this blog post, we will share these new features in detail.

Importing existing tables to ScalarDB

ScalarDB manages transactions by itself, without relying on the transaction management mechanisms of individual databases under its control by adding metadata columns to the records of the database. In previous versions of ScalarDB, it was not possible to properly handle records with null values in the metadata columns, so even if metadata columns were added to existing database records afterwards, they could not be managed by ScalarDB. Therefore, if you wanted to manage a database with ScalarDB, you would need to create a new table that included metadata columns by using a tool such as ScalarDB Schema Loader.

In ScalarDB 3.11, if the value of a metadata column is null, the record is treated as committed. In addition, the Schema Loader tool has been updated to add metadata columns with null values to existing tables if the target database is a relational database. These changes allow existing relational databases to be managed under ScalarDB without the need for reconstruction.

Since this import function only adds metadata columns to existing tables and has a short processing time, it has become easier to migrate existing relational databases to be managed under ScalarDB.

For more details on the table import function, please refer to Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader.

Cross-partition scan (execution of arbitrary WHERE clauses)

In previous versions of ScalarDB SQL, there were limitations in writing WHERE clauses from the perspective of processing efficiency, such as the requirement to specify the Partition key or the Secondary index. In addition, you could only specify range conditions by using the Clustering key in search conditions, which prevented flexible searches based on various conditions in the SELECT statement.

In ScalarDB 3.11, when a database under ScalarDB management is a relational database, it is now possible to execute (push down) the WHERE clause within that database. As a result, specifying the Partition key or the Secondary index is no longer mandatory, and you can combine AND, OR, and LIKE operators freely in the WHERE clause, enabling more flexible data searches in ScalarDB SQL.

Please note that inefficient processing may occur if appropriate indexes do not exist in the database under ScalarDB management or if the table size is enormous, for example. Therefore, please use this feature with appropriate indexes set up, or with small-sized tables.

For more details on cross-partition scan, please refer to the ScalarDB Java API Guide and ScalarDB SQL Grammar.

Enhancements to ScalarDB SQL*

In ScalarDB 3.11, we have made enhancements to ScalarDB SQL, including adding support for JOIN operations.

Support for the JOIN operation

In previous versions of ScalarDB, if your application wanted to join multiple tables, you needed to implement processing equivalent to JOIN on the application side by selecting each table and extracting records matching the join conditions from each table.

In ScalarDB 3.11, JOIN is supported in ScalarDB SQL, so joining tables only requires writing JOIN clauses in ScalarDB SQL. This reduces the implementation burden on the application side, leading to improved development efficiency.

For more details on JOIN, please refer to the ScalarDB SQL Grammar.

Other enhancements

Additionally, various other features have been expanded, including:

• Support for table aliases (SELECT, UPDATE, and DELETE)
• Support for specifying multiple records in a single INSERT statement
• Support for arbitrary WHERE clauses in UPDATE and DELETE statements
• Addition of START TRANSACTION and ABORT commands
• Addition of the UPSERT statement

For details on ScalarDB SQL, including these enhancements, please refer to the ScalarDB SQL Grammar.

Authentication and authorization mechanisms in ScalarDB Cluster*

Previous versions of ScalarDB lacked their own authentication and authorization mechanisms. Therefore, essentially, any user could access tables under ScalarDB management and perform any operation (SELECT, INSERT, UPDATE, DELETE, etc.). As a result, it was not possible to control which user could perform which operations on which tables.

In ScalarDB 3.11, authentication and authorization are supported in ScalarDB Cluster. This enables management of access control and operation permissions for namespaces and tables for users accessing ScalarDB Cluster. Consequently, authentication and authorization can now be performed for each user in a unified manner across multiple different databases.

For details on authentication and authorization in ScalarDB Cluster, please refer to ScalarDB Auth with ScalarDB SQL.

Wrap-up

We released ScalarDB 3.11, which contains great enhancements and improvements to help our customers get the most out of ScalarDB. Please refer to the ScalarDB 3.11 release notes for details, which include the new features introduced here. To learn more about ScalarDB, please take a look at other posts on the Scalar Engineering Blog and at the ScalarDB documentation.

If you are considering using ScalarDB, please contact us.

* ScalarDB SQL and ScalarDB Cluster are features of the Enterprise Edition, so you need to purchase a commercial license if you want to use these features.