This article is a study on the different Key Management Systems that exist today and how they can be used as an Ethereum Wallet for key management.
Our end architecture would look something like this:
The (Decentralised Identity) DID Service Endpoints will require the following functionality against an Ethereum public network:
- Deploy a Smart Contract on to the Rinkeby Testnet through Infura (and paying gas costs)
- Calling Smart Contract functions on the Rinkeby Testnet through Infura (an paying any gas costs)
- Signing and verifying digital signatures
The challenge is to achieve the above without revealing your private key / mnemonic key by delegating all signing and transactions to the key management system. At no point is your key ever revealed to the outside world.
What is Hashicorp Vault?
From the Vault documents: Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.
- Secure Secret Storage: Arbitrary key/value secrets can be stored in Vault encrypted.
- Dynamic Secrets: Grant temporary access credentials that are automatically revoked after lease expires
- Data Encryption: Vault can encrypt and decrypt data without storing it.
- Leasing and Renewal: All secrets in Vault have a lease associated with them. At the end of the lease, Vault will automatically revoke that secret.
- Revocation: Vault has built-in support for secret revocation. Revocation assists in key rolling as well as locking down systems in the case of an intrusion.
All keys are stored in a directory-like structure, allowing security teams to partition keys and manage them individually or as an entire group by applying policies to each key, or an entire directory. They have abstracted the above key concepts into easy to extend modules, allowing you to swap out the authentication/authorization modules (LDAP, IdP, etc) secret engines, and secret storage engines.
Hashicorp does an amazing job of documenting all their products in a clear and concise manner (to the point even I could understand it!). They have figured out a way to introduce key concepts and ideas in easy to understand way while still having sufficient depth and breadth in details for more experienced users. I suggest going to the website to learn more (or take their development track to become an expert).
How do we integrate EthSigner with Hashicorp Vault and Infura?
Note: as of this document, there is a PR that I’ve submitted to EthSigner to support Infura. As per my last article there are some issues with configuring Infura and EthSigner based on how EthSigner works with URL parameters and Infura project ids
Launching Hashicorp Vault
There are many approaches you can take to start the Vault
- Development mode on your local machine
- As part of a kubernetes cluster
- As part of an enterprise solution on a cloud service provider
Vault has an Open-Source version, and an Enterprise version. For the sake of simplicity I’ll be using the Development mode version of vault on my local machine. Note this is not a production ready deployment.
vault server -dev
To validate that vault is running:
That’s it, that’s all you need to do to have a vault running on your machine. Vault runs on port 8200, and uses http by default. You’ll need to look through the logs to find the VAULT_TOKEN for the next step:
There is a lot of well written documents, examples and guides at the Ethsigner website on how to configure Ethsigner with Hashicorp Vault. The trick is to follow the guides carefully or you make the silly mistakes I made (Keys and Gotchas).
- Ethsigner with Hashicorp vault uses a secrets file. Copy the Root Token in the previous section into a file and pass the file path as part of — auth-file argument.
- Ethsigner looks for secret private key in Vault using the secret/ethsignerSigningKey path(and this path is case sensitive). This is configurable.
- By default Ethsigner connects to Hashicorp vault with tls enabled which is a good thing! (flag — tls-enabled is true by default). Hashicorp Vault in dev mode has tls-enabled set to false
But once I figured this all out it was surprisingly easy to get the integration of Ethsigner and Hashicorp vault working. (Kudos to the team).
Keys and Gotchas
When I first started configuring EthSigner to Hashicorp vault I ran into a number of issues, all of which were not very clear from the error logs themselves. Regardless of what configuration error I had made, the log messages would always show:
This was the default error message that would appear and the application would end.
The only way I could figure out what was going wrong was to download the Ethsigner source code, set up the environment locally and start it in debug mode with a breakpoint where the exception occurred..
In all cases it was a silly configuration error, but it would be nice to know from the logs exactly what went wrong.
Here are some problems and gotcha’s that you need to be careful of:
- TLS Enabled: If you run a local dev server of vault, it will default to port 8200, and use http. If you start Ethsigner connected to the dev vault, you’ll get the above error “Failed to extract secret from Hashicorp vault”. The error (in debug) is a 400 error (and something about https). To disable TLS, use the flag — tls-enabled=false in the Hashicorp vault configuration section (TLS flag is enabled by default, which is a good thing)
- Case sensitive keys: Key vault keys and secret names are case sensitive. This tripped me up for a while because:
ethsignerSigningKey != ethSignerSigningKey
The breakpoint trick revealed a 404 error and after digging around it was my fault for typing ethSignerSigningKey. A simple copy/paste of the tutorial instructions would’ve saved me a lot of headache, but a clearer error message here would’ve helped a lot in debugging.
How do we validate that it worked?
As part of my learning exercise on verifiable credentials, I’ve been experimenting with opencerts.io, something created by GovTech (Singapore Government) for issuing verifiable graduation certificates on the Public Ethereum Mainnet. Check out their github, its a very interesting project that utilises Merkle Trees to issues hundreds of thousands of credentials. This is live for all Universities in Singapore.
To validate that the EthSigner, Infura and Hashicorp Vault integration works as intended, we will:
- Connect to EthSigner through localhost:8545 (default) using ethers, JsonRpcProvider and JsonRpcSigner.
- Deploy a smart contract (DocumentStoreCreator)
- Use the store creator to deploy a DocumentStore
- Issue a verifiable credential on the blockchain
- Verify the credential by calling the Smart Contract function
I ran this with EthSigner integrated with Hashicorp, Infura (Rinkeby) with results on etherscan.
What are the potential drawbacks to Hashicorp Vault + EthSigner?
One of Hashicorp Vault’s key selling points is being able to reduce the blast radius of cybersecurity leaks to a single secret so if one of your machines are compromised and the key is lost then you can isolate the damage and revoke that secret (e.g. a username/password or OAuth token, etc).
This is great for username/passwords which can be regenerated, but a loss of a private key in a cryptographic system is a disaster; because it cannot be easily regenerated or replaced.
Imagine a case where I have my private key which holds all the Ethereum I need to run my Enterprise blockchain solution. If I lose that key, I lose all the Ether, and that comes with it real world financial consequences and no recourse for recovery of lost funds.
One thing that I’m not too thrilled about from Hashicorp Vault is the need for me to generate the public/private key pair outside of the vault and then save it into the vault. EthSigner integrates with Vault by first extracting the private key into memory and then using that constructed wallet internally to sign transactions.
This is a major drawback because ideally the private key should never leave the vault. In the case of Azure Key Vault, the signing happened internal to Key Vault and we never expose or extract the stored key.
How do we address this drawback?
From this article, there is something called Eth Secrets Engine which is an extension of Hashicorp Secrets Engine for interacting with Ethereum. We cover this in Part 3 of this series.
Some Useful Scripts for interacting with Hashicorp Vault directly
Here are some scripts I found useful while learning about Hashicorp Vault, specifically around adding and reading the secret and using it to sign stuff. One thing to note is that all Vault APIs are RESTful JSON. Even the commandline tools interact with the vault through the http/https APIs. This makes testing really easy (once you figure out what APIs and arguments to use).
A list of useful curl scripts can be found here. Below are a few that I used frequently for my testing:
The code below shows you how you would extract a secret key from Hashicorp (private key for Ethereum wallet) and use it in Etherjs
This is the second of a 3-part series.
- EthSigner, Microsoft Azure Key Vault, and Ethereum Rinkeby through Infura
- EthSigner, Hashicorp Vault, and Ethereum Rinkeby through Infura
- Hashicorp Vault, Eth Secrets Engine, and Ethereum Rinkeby through Infura
I wrote this because there was a lack of quality materials describing how to integrate these different wonderful technologies. As with most open-source tools the moment you stray from the path it was design for you run into a host of weird and wonderful problems. I hope this helps someone, and drop me a comment if you have some comments, suggestions, requests, etc.
Stay tuned, keep safe, and sign in next time! (pun totally intended).