Empowering blockchain to build complex fully decentralized real-world applications
A very brief history of software architecture
Software programming architecture started with monolithic applications, where UI, business logic and data access were all in the same code. In fact, at the beginning, computers programs were built into the operating system and software programmers were computer operators. Only in the 1950s did programmers start to write programs, using punch cards, while operators were taking care of the physical machine.
Until then, most computer operators were women, as men did not trust these machines and thought they were unreliable compared to the human brain. Computer engineering was at that time considered a quasi-secretarial job and some adverts for computer-related jobs were specifying “no degree required”, meaning “women can apply”. In the picture below, which shows the ENIAC computer, two women are seen working in the background. The contribution of women in computer engineering is also depicted in the ‘Hidden Figures’ Hollywood movie.
However, computers improved quickly and in the 1960s, it was possible to run multiple processes on the same computer. This architecture, which is called host-based, evolved into client/server architecture thanks to RPC (remote procedure calls) which allowed processes to engage into a live discussion. Finally, in the 1970s, Arpanet laid the foundations of Internet which was standardized in the 1980s with the TCP/IP protocol, breaking the inter-computers barrier.
As you can see, client/server architecture and networking capabilities are well established for a long time and software architecture closely followed hardware evolution since the beginning.
Architecture then continued into the layered trend which brings many benefits in term of testing, maintenance and evolution capabilities. At first it was just logical layers created within a monolithic application, thanks to computer programming languages features such as OOP. Then, with the use of public APIs, external applications could also be layered as services of a bigger application. Finally, today a lot of applications are built using micro-services, the ultimate evolution of layered applications, eventually deployed over distributed networks and communicating through message queuing component.
To make a long story short, regular application nowadays can generally be organized around the following parts:
- Web and Mobile application
- Web Server
- Back-end
- Database
- Message Queuing
Decentralized Vs Distributed
Distributed applications are mostly built to address performance issues and consist of aggregating several computers in order to share more processor power, network bandwidth and data storage. In a way, they implement a classical application over a distributed network, usually in a master/slave setting.
However since the early 2000s, a new kind of applications appeared, based on a protocol that can be run by heterogeneous computers, called nodes, together forming a new type of network. A famous example of such a decentralized application is Napster, which introduced peer-to-peer sharing in 1999. A few other protocols were inspired by its success, which led to BitTorrent protocol. However these protocols usually all rely on some central server and are regularly closed by the authorities due to illegal content being shared.
Around 10 years ago, the Bitcoin revolution showed that an alternative way to design applications was possible. A way that would involve a community of peers collaborating in building a fully decentralized ledger without any central authority. While Bitcoin has obviously become hugely successful, being worth over $300 billion today, it is still merely a shared accounting ledger. So in 2014, Ethereum came up with smart contracts, allowing anyone to build a fully decentralized application on top of the Ethereum blockchain, using a Turing complete programming language.
To make a long story short, a classical decentralized application nowadays is:
- a smart contract.
Back to the 60s
It turns out, by a strange coincidence, that software programmers on Ethereum are now back to the 60s, forced to build a decentralized application using a low-level language, with very limited memory size, no access to external data sources and having to pay a non-predictable fee for every invocation of their application…
At Sikoba we do believe in decentralized protocols, aka the blockchain, but our purpose is not to build a blockchain for the sake of it, or for introducing a new cryptocurrency or yet another Ponzi scheme. We need a blockchain because our application is inherently decentralized. However, due to the complex business logic of Sikoba, there is no way it can fit into a “smart contract”. Instead, we decided to start with a classical centralized application, which, thanks to the layered approach of modern software architecture, we were able to change into a powerful decentralized application.
We expose here our experience in refactoring our application from a standard application into a fully decentralized application.
Blockchain as a middleware
If one cannot use a smart contract, that means one must implement one’s own decentralized application. Obviously implementing a complex application in a decentralized way from scratch is very ambitious and implies a number of pitfall to avoids, and a number of subjects to master, including distributed computing and cryptography.
However there is hope, the approach we took at Sikoba is a modular approach, looking at each feature independently and trying to see how they can be combined.
Indeed, blockchains are often seen as distributed database so one may ask, what if I simply replace the database component of my application with a blockchain? Of course the reality is more complex than that but nevertheless, such idea led to middlewares for building blockchains.
What is a blockchain?
In order to continue, we need to understand what a blockchain really is, and why we do not simply use a distributed database as our blockchain layer.
After all, distributed databases are vastly more powerful than a blockchain in terms of performance. The reason is immutability.
Immutability
A blockchain is a trusted source of data. It organizes data sequentially into blocks that are cryptographically linked; a block contains the hash of its previous block. Data cannot be changed inside a block, as this would change its hash, and blocks cannot be re-ordered as the hashes would not match.
This structure is very convenient to share and verify. Just by matching hashes of the last block, two different entities can see that they are using the same set of data. If they wanted to do the same with a regular database, they would need to hash the entire database.
Synchronization
Yet, in order to be fully decentralized, the data needs to be shared among all nodes that run the app. Blockchains are able to synchronize data through the use of consensus mechanisms, which result in all the nodes agreeing on the data forming the next block. For instance one simple way to achieve this would be to have the nodes to vote for the next block. In practice voting is not used because it involves too many constraints but the underlying principles of elections are often used in consensus.
Transactions
Because of the immutability nature of the blockchain, the data can be seen as a transaction log, called the ledger. Now, every system does have, at least virtually, a transaction log of all the events happening in the system, and conversely, any system status can be seen as the result of a transaction log.
In traditional blockchains, transactions are only user transactions, signed by a private key. The corresponding public key is the account name, and this makes the system fully decentralized as anyone can submit transactions but only you are in control of your private/public key pair.
Buildind a decentralised app
Now we start to see what do we need in order to build a truly decentralized application deployed on a set of nodes:
- Give control to your users with asymmetric cryptography: users must sign all their actions on the system so that nobody else can impact on their state;
- The system can only evolve through a set of ordered transactions applied to the current state;
- The nodes must agree on the set of transactions to be applied to the current state;
- The log of the applied transactions is commited to an immutable blockchain.
While each step involves a non-trivial task, each can be done independently of the others so it is an effective way to split the work into smaller do-able tasks. Let’s review them one-by-one.
I. User Keys
User key management is a big issue that is often not considered in blockchain projects. At the minimum you must allow the users to generate and backup their keys on the client side. This is often achieved with the help of a seed phrase that can be stored on paper. Although this works well for users familiar with cryptocurrencies, it won’t work for users of a classical applications.
Our solution is to offer users the choice to either back up the private key on the server side, or to use the seed phrase and manage the backup process themselves himself. We also have investigated other recovery process, such as knowledge or community based authentication but they all come with some security concerns so they may not be relevant, depending on the desired security level.
Because the private key is supposed to stay on the client side, this cause issues when users have multiple ways to access the application. For instance we support mobile and web front-ends and so we needed to implement client-side key synchronization and secure storage of keys.
Finally, users in a traditional blockchains are just a public key represented by long hexadecimal string. In Sikoba, we managed to pack this info into a 20 characters long account number using state-of-the-art memory-hard password hashing and checksum validation.
2. System State
The evolution of the state is seen as the result of a transaction log applied to it. These changes are originating from inputs external to the system: the users. This step consist into writing any user action that changes the system state as a set of elementary transactions. In other words, every database write operation needs to be encapsulated into a transaction, signed by the person at the origin of the write. We process the transactions from the queue one by one, and commit them to the database in that same order.
Another important point is to decide who has the right to modify which data?
For instance it would make sense for your application that only the user can modify his/her data. These access-right rules must be respected when applying the transaction.
Sometimes, the system has to modify the state by itself. There are two cases:
- The modification is deterministic, it happens as a result of following business rules. In that case every nodes will run this process, after applying a block.
- The modification is not deterministic, and the originating node must create a transaction, signed by the node key, and send it to the transaction queue.
3. Consensus
The consensus layer is a crucial component of a blockchain system. There are many consensus models, with different properties. The main points to consider are usually
- resource consumption
- communication complexity
- fault-tolerance
Although many models exist, they ultimately all do the same one thing: make a set of nodes agreeing on some values. This means that in theory it is easy to isolate this feature into a single component. In Sikoba, we use a modern and powerful pluggable consensus layer called Babble.
4. The Blocks
Finally, once a sequence of transactions is determined, we just have to process them and add them to a new block to perpetuate the blockchain. A block has some cryptographic elements defined in its header, such as the Merkle root and the hash of the previous block. As a hashing function, we do not use the usual Sha256 hash but instead use a zero-knowledge proof friendly hash called Poseidon. This is quite important for permissioned blockchains.
An important design decision is to determine what data should actually be stored in the blocks of the blockchain. Indeed, you should try to keep only critical data for the business logic into the blocks and this will of course depend on your application. You must group your transactions into the following three categories:
- Data specific to a node: Node specific data should not be synchronized and can be written directly to the database, in a specific table for instance.
- Non critical data that must be kept in sync: Non critical data must be written through transactions that go into the consensus layer but are not written into the block.
- Critical business data: only the transactions containing critical data are added to the block.
For example, at Sikoba we record the phone numbers of our users but this data does not go into the blockchain because of privacy and GDPR, also it does not impact on the business logic. However, it is important that the phone number gets synchronized through all the nodes, so changing a phone number is a transaction that goes into the second category. For recovery purposes, the database part that is not persisted into blocks must be well separated.
One should note that transactions are added to the block after they have been processed so that we can decide, depending on the result of their process, if we add them to the block or not. Indeed, since the process of the transaction is part of the business logic, it is important to remove it from the consensus layer in order to follow separation of concerns principle.
If a transaction is failing, we do not add the transaction into the block, instead we add it to a temporary list of failed transactions so that users can be notified.
Implementation
We decided to move from a centralized application into a decentralized application in two steps, as illustrated in the following schemas.
Baseline architecture
This schema represents different components that a classical 3-tier application may have. In our case the first tier, the ‘client’, consists of a web application and a mobile app. The second tier, the ‘server’, has three components, none of which are mandatory for all applications. So we have:
- Web-server
- Message Queuing
- Back-end
The web server is serving web pages for the web front-end. For very simple apps it is possible to pack everything into the web server, but best practices do not favor monolithic applications that ultimately are not easy to maintain. Rather, it is recommended to move out the business logic into another component that we call here the back-end. The arrows from the mobile app and web server to the back-end should be viewed as API calls.
Web servers can nowadays be available for many development frameworks in a very lightweight form, so it is not uncommon to not have a standalone web server but having it integrated into the ‘back-end’. The ‘back-end’ component should not be viewed as a monolithic component, it would usually follow micro-services pattern that separate features into independent processes.
The message queuing component is also not mandatory but it is often present in modern applications as it helps them to be responsive. It serves as a queue for messages that can be either numerous or slow to process, so that they can be processed asynchronously, or as a way to decouple interactions between components of the applications. Indeed as the number of components grows, their interactions quickly become unmanageable.
Most classical application can be properly described in this way, depending on what exactly we put under the boxes. Note that even more complex applications that need to interact with other servers or external services can still follow this template. One would just need to add an interaction with other servers through an arrow from the back-end to the outside.
Transition architecture
Because of the immutability of the blockchain, it is important to completely separate the read-only operations from the write operations. We achieve this by modifying the architecture, without yet adding the blockchain capability. In the diagram, black arrows are strictly read-only and red arrows represent database write operations.
One can see that the transition architecture is very similar to the baseline architecture so that it should not be a very difficult step to implement. However, it is important that all the red arrows consist only of transactions, as explained in the System State section. Another remark is that the message queuing component makes the task easier by grouping all the database write operations.
Target architecture
From the transition architecture, we now just need to remove the processing of the transactions in the message queue, and instead to send them to the consensus component. This component will discuss with the other nodes and they will all come up with the same ordered set of transactions that should form a new block.
Conclusion
We have shown how a complex application can be changed into a fully decentralized application and still benefits from the latest software engineering improvements in term of tools, flows or design patterns. At Sikoba we can provide support and building blocks for teams that want to make their application decentralized.
