Scalar: A Reliable Data Management Company
In the first Scalar Engineering Blog post, I will briefly introduce Scalar, Inc. and our core products, Scalar DB and Scalar DL. (Japanese version is here.)
Scalar, Inc. is a Tokyo-based startup that aims to accelerate customers’ digital transformation (DX) with our reliable data management technologies. We provide two core products, Scalar DB and Scalar DL, that work on existing databases to address various reliability challenges that the databases do not handle well.
Scalar DB: Universal transaction manager
The first challenge we tackle is the transaction consistency challenge that arises when multiple separated databases need to interact with each other. We see such a case in a single organization where multiple engineering divisions manage different services (e.g., credit card service and point service) that interact with each other. Each service manages a different database separately to achieve higher service independence (i.e., microservice architecture). Likewise, we also see a similar case in a system that spans multiple organizations that separately manage their services (e.g., travel booking and insurance services) and databases. Their services interact with each other to exchange information and make certain respective operations (e.g., booking travel and buying insurance for the travel) occur at the same time.
It is challenging to guarantee transaction consistency between multiple databases in the above cases, especially when the databases are different or don’t even have transaction capability. A two-phase commit based on XA transactions could address the issue when the databases are XA-compliant, but it is not the case when the databases are not XA-compliant or are NoSQLs like Amazon DynamoDB and Azure Cosmos DB.
Scalar DB provides a universal solution to the issue. Scalar DB provides a database abstraction and a database-agnostic transaction manager on top of the abstraction. Therefore, the transaction manager does not even know what database systems it interacts with. With this mechanism, the transaction manager can naturally do a transaction that spans multiple databases.
This mechanism provides additional benefits. Since it is a database-agnostic transaction manager, it can provide transaction capability to non-transactional databases such as Cassandra, HBase, Amazon DynamoDB, and Azure Cosmos DB. Moreover, since it is a database abstraction, it decouples an application from the underlining database, i.e., Scalar DB makes applications portable, so an application can be migrated from one database to another without changing the application code.
For more details, please look at the slide or the source code licensed under Apache 2. We also explain more details in upcoming posts.
Scalar DL: Byzantine-fault detection middleware for transactional database systems
The second challenge we tackle is the accountability challenge that arises when data is digitized. Although accountable digital data is one of the keys to achieving DX, digital data can be easily copied, altered, and removed maliciously; thus, it is not accountable by itself. Such a malicious attack is called a Byzantine fault, and there have been lots of work exploring a way to deal with the fault in systems that manage digital data (i.e., data management systems).
Byzantine-fault tolerance (BFT) technologies such as BFT SMR and Blockchains could be one of the promising solutions to address the issue. However, they have a scalability limitation due to the sequential processing of totally-ordered data. Specifically, they have difficulties scaling their performance in proportion to computing resources; thus, it is hard to increase the total processing capability even if computing resources increase. Some research projects have addressed the challenge; however, BFT technologies have difficulty in their practicality because they require at least 4 (i.e., N > 3f) administrative domains to guarantee correctness properly. We have seen a lot of cases where those BFT technologies are used in a single administrative domain, e.g., deploying a BFT system to 4 node-cluster in a single administrative domain, but it defeats the original purpose because a fully-privileged administrator can tamper with all the nodes’ data in the same way.
Scalar DL deals with Byzantine faults differently. Scalar DL provides a Byzantine-fault detection middleware for transactional database systems and makes the databases tamper-evident while preserving the scalability of the database to address both the accountability and scalability challenges. It also requires only two administrative domains to achieve correctness; thus, it also addresses the practicality challenge as much as possible. Moreover, Scalar DL is database-agnostic middleware so that it can run on various database implementations.
Please look at the slide or wait for upcoming posts and a paper we are working on for more details.
Finally, we also work on middleware and applications using the core products to solve customers’ problems directly. We will talk more about them in future posts.
