Best Practices for Running a Permissioned Blockchain Network in a Regulated Production Environment

Adhara’s 11 requirements for a blockchain client and why they chose Hyperledger Besu for their latest settlement solution

Adhara, a real-time solution for global liquidity management, recently developed a blockchain-based system for a South African exchange ZAR X in conjunction with Nominee Administrator Computershare South Africa.

The solution is the first distributed settlement system in South Africa for unit trusts. Using the Ethereum blockchain, it enables the public to purchase and settle unit trusts directly from fund managers on a regulated, secure exchange called ZAR X.

The Adhara team chose Hyperledger Besu as the Ethereum blockchain client for this production system. This post will dive into the team’s 11 requirements for a blockchain client and ultimately why they chose Hyperledger Besu.

Adhara Requirements

The Adhara team have worked on regulated permissioned blockchain solutions for several years. They were part of the design team for Project Ubin Phase 2 with the Monetary Authority of Singapore (MAS) and were the technical team behind Project Khokha with the South African Reserve Bank (SARB). They have also completed a set of pilot cross border payments that terminate onto the i2i network in the Philippines.

As a result of the work the team has done, they have put together a set of requirements for a blockchain client (with related services) that is suitable for use in a regulated permissioned environment as follows:

1. A distributed general ledger with ledger-wide transactions
2. Private data channels with routing capabilities
3. Well-tested ledger software
4. Well-established smart contract technology
5. Excellent testing tools
6. Permission layer to ensure counterparties can be authenticated and authorised on the network
7. Ability to whitelist accounts
8. Secure interoperability with well-established key management systems
9. Event streams
10. Enterprise support
11. Strong technical support

Each of these will be unpacked in more detail below.

1. A distributed general ledger with ledger-wide transactions

An essential principle in a distributed ledger system is that stores of value are accessible across the entire ledger. It is important that if a credit is made into any account on the ledger, that credit can be used on any other account on the ledger with a simple debit/credit transaction.

In the case of liquidity management, this prevents fragmentation of liquidity, which is currently a major and costly problem in capital markets with treasurers having liquidity pools on many centralised ledgers. A distributed ledger can solve this problem, but only if the liquidity on that ledger is easily available across the whole ledger.

When dealing with distributed ledger systems, however, ensuring that balances and transaction amounts are private to a select group of counterparties is complicated and raises many interesting questions surrounding privacy in general in distributed systems.

Many ledgers have the concept of “private confidential transactions” where privacy is ensured by putting the transaction onto a private channel that is not visible or accessible by other counterparties on the network. This is problematic for a number of reasons including liquidity fragmentation and scaling out to new institutions joining the network.

There are ledger-wide techniques for shielding balances and transaction amounts (using Pedersen commitments for example) and these techniques do not run into the same problems as private channels.

Maintaining ledger-wide consensus on balances and transaction amounts is a requirement for the Adhara team.

2. Private data channels with routing capability

While consensus on the value of transactions on the ledger needs to be ledger-wide, there are many specific details linked to individual transactions that do not require ledger-wide consensus. Names of counterparties doing the transaction and the reason for the transaction are examples of data that can be communicated via a private channel between the specific nodes involved in the transaction.

In this case, a secure, point-to-point private channel is required, and the private data on this channel needs to be easily linked to its value transaction.

In addition, due to IT compliance rules in most enterprise organisations, the private channels need to have the ability to be routed across the network through relay nodes. This is because most enterprises will not allow multiple point-to-point connections from their data centres. Private channels that require direct connections between the endpoints are therefore not useful. The Adhara solution requires private channels that allow routing through relay nodes to enable scaling of the network and to ensure enterprise compliance.

3. Well-tested ledger software

In our experience, the best testing ground for distributed ledger software are public blockchain networks. Those networks are publicly accessible, and there is a strong incentive for being able to create fraudulent transactions on them, or even to cripple or destroy them. They are permanently under attack and as such have developed robust mechanisms for ensuring the integrity of the networks.

A good example of this were the spam attacks on the public Ethereum network in September/October 2016 which particularly affected the Go Ethereum (Geth) client.

Erik Voorhees said at the time, “Every time someone attacks a blockchain platform and fails to kill it, the platform gets stronger, and more worthy of building upon.”

There are a number of distributed ledger platforms that have no public expression. There are also a number of distributed ledger platforms that are based on public network software but are not consistently updated with the public software.

From Adhara’s perspective, we were looking for a software stack that was very close to the public chain software stack.

4. Well-established contract technology

Everything in the Adhara world is based on contracts. The way that money is tokenised onto a distributed ledger is through a contract. The way that liquidity is locked for a transaction is through a contract. The Adhara stack is heavily reliant on a good contracting environment.

5. Contract layer with well-established test tools

The contracts go hand in hand with good test tools. Building out large interconnected enterprise applications required test tools for both development and continuous integration. Having the ability to test contracts and include those tests in a CI pipeline is critical for any enterprise development team.

The Adhara team have used a number of test tools, but the suite that is head and shoulders above the rest is the Truffle test suite. Truffle supports a number of different blockchain clients and the blockchain clients that can integrate into Truffle test tools have an advantage over others.

6. Permissioning layer to ensure counterparties can be permissioned onto the network

In a permissioned distributed ledger, it’s important to have a clear, simple process of ensuring that counterparties can only connect to the ledger if they have the required permissions. In public blockchain environments, there are no requirements for connecting to the network. In traditional centralised enterprise environments, permissions are controlled in a centralised fashion and authorization on any system is generally well-understood. Permissioning across a distributed ledger that typically spans multiple organisations is challenging.

7. Ability to whitelist accounts

While access to the distributed ledger provides one layer of protection, being able to whitelist accounts provides another layer of protection. This whitelisting of accounts is typically done at the contract level. There may be different contracts on the same ledger with different requirements for whitelisting.

Whitelisting of accounts determines which accounts are able to transact with a specific contracts. There are a number of regulatory requirements around proving that the owner of the account has been through a Know Your Customer (KYC) process. The KYC process is an off-chain process, but the flagging of the account as whitelisted by a specific contract is an on-chain process.

8. Secure interoperability with well-established key management systems

Key management systems are very important in the distributed ledger ecosystem because this is where the private keys are stored which allow users (or their custody providers) to sign transactions.

Blockchains typically use three different types of private key management.

1. Software-based key storage. This is generally used on mobile wallets and software wallets by individual users.
2. Individual hardware keystores. These are designed for individuals to store some keys on them. Good examples of these are the Ledger Nano S, the Trezor or the KeepKey. These are typically USB devices designed to work with a personal computer.
3. Hardware keystores designed for custody of large numbers of keys. These range from keystores as a service to custom hardware/software stores designed for data centres. These keystores are typically used by exchanges or custody banks.

While this is not a blockchain client specific requirement, some of the keystores designed in option 3 above do work better with some blockchain or distributed ledger platforms.

The Adhara team have evaluated various keystore solutions to determine the security of storing keys and signing transactions, specifically where large numbers of keys need to be held by a custodian.

9. Event streams

In a distributed ledger environment, there is a requirement to be able to respond to events that occur on the ledger. These events are generally initiated by different entities on the ledger and so a reliable, persistent event stream is critical to ensure that events that will trigger off-chain business processes are not missed. For example, a regulator may need to approve a cross border payment. The payment request will be put onto the distributed ledger, and the regulator will need to either approve or deny that request thereby releasing the payment. This is typically an automated process, but relies heavily on the ability of the regulator to view and respond to the events on the ledger.

Having a reliable, persistent event stream is critical to the full Adhara stack.

10. Enterprise support

While distributed ledgers are inherently more resilient that centralised systems, there is still a requirement to be able to get business critical support. Some distributed ledger platforms provide no support at all, while some provide really good enterprise grade support. Adhara needed to choose a platform that provided good support both to the Adhara technical team (for dev and pilot work) and to Adhara’s clients for production systems support.

11. Strong technical team backing the distributed ledger development

For any business critical infrastructure software, there is a requirement for a capable, well established and accessible technical team. Software changes all the time because the requirements are changing all the time. Having consistency and longevity in the team that is supporting software is critical to the ongoing quality of that software. If that software is fully open sourced, that is an added benefit, because even if the company supporting the software can no longer support it for whatever reason, the industry can continue to support it, and other teams can continue to maintain it.

One of the biggest and most significant efforts in the space is the Hyperledger community. It is supported by the Linux Foundation and all software within the Hyperledger community is open sourced. There are a number of well supported distributed ledger clients within Hyperledger, including Fabric and Sawtooth. Besu, however, is the only one that has both a permissioned enterprise application as well as being an active client on a public blockchain (public Ethereum).

Decision time

Given the requirements that Adhara has, we needed to look at the options out there. There are basically three groups of blockchain clients:

• Enterprise-focused clients (with no public network). Examples of this are Hyperledger Fabric, Clearmatics, Digital Asset Holdings, Quorum and R3 Corda
• Public-focused clients. Examples of these are Bitcoin Core, Geth, ZCash, Monero and many others
• Public-focused clients with strong enterprise support and tools. Examples of these are Hyperledger Besu, Parity, Ripple(x-Connect), and Interstellar

There were a number of reasons why the Adhara team did not look at purely private or purely public blockchain clients and focused rather on public clients with strong enterprise support. The more obvious reasons are:

• No public chain testing for purely enterprise-focused distributed ledgers
• Some enterprise-focused chains have limited ledger wide consensus mechanisms
• Some public blockchains have limited (programmable) contracting capabilities

The focus of the Adhara team has been on public-focused clients with strong enterprise support and tools

• The Adhara team has worked extensively on Quorum. Project Khokha for example which was a testnet run by the South African Reserve Bank was run on Quorum.
• The Adhara team have tested their components on Parity, but have not done extensive work on it.
• Ripple, while it has a public expression (Ripple) and an enterprise expression X-connect has a limited contract language and support compared to some of the competitors.

Hyperledger Besu ticked all the boxes for the Adhara team:

Strong team. PegaSys brings a 65-person team including PhDs in cryptography and distributed systems, as well as enterprise software experience. They have a customer-driven roadmap and work closely with users and customers that are aiming for production.

Support. PegaSys and ConsenSys partner to offer the production support which is a critical piece for blockchain networks.

Whitelisting. Hyperledger Besu 1.2 was released in July 2019 with the ability to define a whitelist of Ethereum accounts permissioned to transact on the network.

Close to public. Hyperledger Besu works with standard Ethereum smart contracts in solidity and popular tools like Truffle, and incorporates key features like fast sync.

The Hyperledger Besu enterprise client is the same as the public chain client. There is no difference. Changes that are made in the public client are also made in the enterprise client because it is the same software.

Permissioning. Hyperledger Besu offers a robust set of permissioning tools, with a complete set of permissioning features to be used out of the box for most enterprise needs including:

• On-chain smart contract Node and Account permissioning to ensure all nodes are in sync and using the same data
• Set of smart contracts that include the whitelist of nodes and accounts and administrator access rights for easy deployment of permissions
• An easy to use dapp that interacts with a Web3 provider (e.g. MetaMask) to read/write to the contracts with a simple graphical UI

In future releases of Hyperledger Besu, the team plans on extending the permissioning feature even further by enabling support for Role-Based Access Controls and more granular levels of permissioning, enabling permissions to be set across groups of nodes and accounts, and defining what types of transactions each user is able to make, ensuring the security of the chain is maintained.

Solid consensus algorithm. IBFT 2.0 is a consensus algorithm developed by PegaSys using the work done by AMIS on the original IBFT consensus algorithm which was used in JP Morgan’s Quorum client. Using a validator-based BFT consensus, IBFT 2.0 allows for faster transaction throughput with rapid settlement and finality for any digital asset transfer (such as exchanging currency or trading equities) or smart contract operations (like issuance of an asset). This allows Hyperledger Besu to produce a more stable and reliable Enterprise Ethereum client, balancing the increased performance of Proof of Authority consensus algorithms without being too centralized, and still providing the protection against bad actors as seen in Proof of Work.

Conclusion

The Adhara team have chosen to work with Hyperledger Besu and the PegaSys team that supports it for all of the reasons listed above. We are, however, aware that this is still a very new space and we expect blockchain clients and distributed ledger providers to evolve significantly over the next few years. While our requirements listed above will stay reasonably constant in the near term, we will continue to evaluate the available software on which to run Adhara. Through the PegaSys Partner Program, we now have a primary relationship with the PegaSys team, and while we prefer to use our stack with Hyperledger Besu clients, we do support other enterprise Ethereum clients.

We also see a scenario in the near future where permissioned networks will be able to run with different clients in the same way that the public Ethereum ledger has a number of different clients connected to it (Besu, Geth, Parity etc). Once the permissioning and the consensus mechanisms become standardised, it will be possible to run a permissioned distributed ledger consisting of different enterprise blockchain clients. While this may have an impact on how the network is operated (enterprise support contracts for specific parts of the network for example), we as Adhara would welcome this evolution.

Want the latest product updates, enterprise blockchain news, and more straight to your inbox? Subscribe to ConsenSys newsletters.

ConsenSys is the seed funder and an existing shareholder of Adhara.

Disclaimer: The views expressed by the author above do not necessarily represent the views of Consensys AG. ConsenSys is a decentralized community with ConsenSys Media being a platform for members to freely express their diverse ideas and perspectives. To learn more about ConsenSys and Ethereum, please visit our website.