A Study on Blockchain Key Management Systems — Part 2

This article is a study on the different Key Management Systems that exist today and how they can be used as an Ethereum Wallet for key management.

This is Part 2 in a three part series, Part 1 can be found here. Part 2 covers Integration of EthSigner, Hashicorp Vault, and Infura.

Please refer to Part 1 for a background of Key Management Systems and how you can configure EthSigner (Consensys) with Microsoft Azure Key Vault and Infura.

Our end architecture would look something like this:

Blockchain KMS using different solutions.

The (Decentralised Identity) DID Service Endpoints will require the following functionality against an Ethereum public network:

  1. Deploy a Smart Contract on to the Rinkeby Testnet through Infura (and paying gas costs)

The challenge is to achieve the above without revealing your private key / mnemonic key by delegating all signing and transactions to the key management system. At no point is your key ever revealed to the outside world.

What is Hashicorp Vault?

From the Vault documents: Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.

Source: https://dev.to/xfrarod/dynamically-securing-databases-using-hashicorp-vault-2kf

Key Concepts:

  • Secure Secret Storage: Arbitrary key/value secrets can be stored in Vault encrypted.
High Level overview (source: https://www.vaultproject.io/docs/internals/architecture)

All keys are stored in a directory-like structure, allowing security teams to partition keys and manage them individually or as an entire group by applying policies to each key, or an entire directory. They have abstracted the above key concepts into easy to extend modules, allowing you to swap out the authentication/authorization modules (LDAP, IdP, etc) secret engines, and secret storage engines.

Hashicorp does an amazing job of documenting all their products in a clear and concise manner (to the point even I could understand it!). They have figured out a way to introduce key concepts and ideas in easy to understand way while still having sufficient depth and breadth in details for more experienced users. I suggest going to the website to learn more (or take their development track to become an expert).

How do we integrate EthSigner with Hashicorp Vault and Infura?

Note: as of this document, there is a PR that I’ve submitted to EthSigner to support Infura. As per my last article there are some issues with configuring Infura and EthSigner based on how EthSigner works with URL parameters and Infura project ids

Launching Hashicorp Vault

There are many approaches you can take to start the Vault

Vault has an Open-Source version, and an Enterprise version. For the sake of simplicity I’ll be using the Development mode version of vault on my local machine. Note this is not a production ready deployment.

Install Hashicorp vault, and then start it by running:

vault server -dev

To validate that vault is running:

export VAULT_ADDR='http://127.0.0.1:8200'
vault status
You should get this message if you run the status command on vault

That’s it, that’s all you need to do to have a vault running on your machine. Vault runs on port 8200, and uses http by default. You’ll need to look through the logs to find the VAULT_TOKEN for the next step:

On startup in DEV mode, the terminal will show you a bunch of info. Make note of the root token

Configuring EthSigner

There is a lot of well written documents, examples and guides at the Ethsigner website on how to configure Ethsigner with Hashicorp Vault. The trick is to follow the guides carefully or you make the silly mistakes I made (Keys and Gotchas).

  1. Ethsigner with Hashicorp vault uses a secrets file. Copy the Root Token in the previous section into a file and pass the file path as part of — auth-file argument.

But once I figured this all out it was surprisingly easy to get the integration of Ethsigner and Hashicorp vault working. (Kudos to the team).

Keys and Gotchas

When I first started configuring EthSigner to Hashicorp vault I ran into a number of issues, all of which were not very clear from the error logs themselves. Regardless of what configuration error I had made, the log messages would always show:

Not a specifically helpful error message, even with — logging=ALL

This was the default error message that would appear and the application would end.

The only way I could figure out what was going wrong was to download the Ethsigner source code, set up the environment locally and start it in debug mode with a breakpoint where the exception occurred..

In all cases it was a silly configuration error, but it would be nice to know from the logs exactly what went wrong.

Here are some problems and gotcha’s that you need to be careful of:

  • TLS Enabled: If you run a local dev server of vault, it will default to port 8200, and use http. If you start Ethsigner connected to the dev vault, you’ll get the above error “Failed to extract secret from Hashicorp vault”. The error (in debug) is a 400 error (and something about https). To disable TLS, use the flag — tls-enabled=false in the Hashicorp vault configuration section (TLS flag is enabled by default, which is a good thing)

ethsignerSigningKey != ethSignerSigningKey

The breakpoint trick revealed a 404 error and after digging around it was my fault for typing ethSignerSigningKey. A simple copy/paste of the tutorial instructions would’ve saved me a lot of headache, but a clearer error message here would’ve helped a lot in debugging.

How do we validate that it worked?

As part of my learning exercise on verifiable credentials, I’ve been experimenting with opencerts.io, something created by GovTech (Singapore Government) for issuing verifiable graduation certificates on the Public Ethereum Mainnet. Check out their github, its a very interesting project that utilises Merkle Trees to issues hundreds of thousands of credentials. This is live for all Universities in Singapore.

To validate that the EthSigner, Infura and Hashicorp Vault integration works as intended, we will:

  1. Connect to EthSigner through localhost:8545 (default) using ethers, JsonRpcProvider and JsonRpcSigner.
Gist for the deployment of Smart Contracts using EthSigner

I ran this with EthSigner integrated with Hashicorp, Infura (Rinkeby) with results on etherscan.

What are the potential drawbacks to Hashicorp Vault + EthSigner?

One of Hashicorp Vault’s key selling points is being able to reduce the blast radius of cybersecurity leaks to a single secret so if one of your machines are compromised and the key is lost then you can isolate the damage and revoke that secret (e.g. a username/password or OAuth token, etc).

This is great for username/passwords which can be regenerated, but a loss of a private key in a cryptographic system is a disaster; because it cannot be easily regenerated or replaced.

Imagine a case where I have my private key which holds all the Ethereum I need to run my Enterprise blockchain solution. If I lose that key, I lose all the Ether, and that comes with it real world financial consequences and no recourse for recovery of lost funds.

One thing that I’m not too thrilled about from Hashicorp Vault is the need for me to generate the public/private key pair outside of the vault and then save it into the vault. EthSigner integrates with Vault by first extracting the private key into memory and then using that constructed wallet internally to sign transactions.

This is a major drawback because ideally the private key should never leave the vault. In the case of Azure Key Vault, the signing happened internal to Key Vault and we never expose or extract the stored key.

How do we address this drawback?

From this article, there is something called Eth Secrets Engine which is an extension of Hashicorp Secrets Engine for interacting with Ethereum. We cover this in Part 3 of this series.

Some Useful Scripts for interacting with Hashicorp Vault directly

Here are some scripts I found useful while learning about Hashicorp Vault, specifically around adding and reading the secret and using it to sign stuff. One thing to note is that all Vault APIs are RESTful JSON. Even the commandline tools interact with the vault through the http/https APIs. This makes testing really easy (once you figure out what APIs and arguments to use).

Curl Scripts

A list of useful curl scripts can be found here. Below are a few that I used frequently for my testing:

Gist for convenient curl scripts

Javascript

The code below shows you how you would extract a secret key from Hashicorp (private key for Ethereum wallet) and use it in Etherjs

Gist for interacting directly with Hashicorp vault

What’s Next?

This is the second of a 3-part series.

I wrote this because there was a lack of quality materials describing how to integrate these different wonderful technologies. As with most open-source tools the moment you stray from the path it was design for you run into a host of weird and wonderful problems. I hope this helps someone, and drop me a comment if you have some comments, suggestions, requests, etc.

Stay tuned, keep safe, and sign in next time! (pun totally intended).

Interested in the application of Blockchain technology in the betterment of civilisation

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store