An Anatomy of the BlockVigil Transaction Manager

This article discusses the anatomy of the management layer that empowers the EthVigil backend to take API write calls and convert them to valid Ethereum transactions to be executed on the blockchain.

Part 1 of the BlockVigil transaction manager series provides a detailed overview of the various stages an Ethereum transaction goes through while keeping in mind that we are dealing with smart contracts and not just value transfers. I’d highly recommend going through that article before continuing further with this one.

This is a specific dive into our transaction manager for EthVigil, the stateful API gateway for Ethereum networks.

• Sign up for a beta access here: https://beta.ethvigil.com/
• Documentation and detailed tutorials: https://ethvigil.com/docs
• Holler at us on twitter: https://twitter.com/blockvigil

The responsibilities of a transaction manager

This state-flow diagram provides a concise overview of the different stages a contract deployment, or its corresponding write calls, go through.

To handle this flow an interface must at the least be able to —

1. Create raw transactions from API calls (contract deployment calls or write calls against deployed contracts).
2. Submit raw transactions to a node running an Ethereum client.
3. Handle errors.

But first, let’s take a look at how API calls are made to EthVigil. The following is an example of an API call that calls the method changeOwnerName with argument keltu on a smart contract that is deployed at 0x78c4588c5c116779b8b3bb6d3fcd3c5c342cce8c.

curl -X POST "https://beta-api.ethvigil.com/v0.1/contract/0x78c4588c5c116779b8b3bb6d3fcd3c5c342cce8c/changeOwnerName" -H "accept: application/json" -H "X-API-KEY: b7036bd7-0a4b-44f6-ae74-2d8057cf95d7" -H "Content-Type: application/x-www-form-urlencoded" -d "_ownerName=keltu"

To translate this API call to an Ethereum transaction that can execute the smart contract method, we need to first have an externally owned account that can initiate this transaction. We can recall that all Ethereum transactions will have to be signed and initiated by externally owned accounts.

The API calls are translated to native Ethereum transactions. Visit the code in our docs to see in action the abstraction of choosing Ethereum account addresses/private keys to sign transactions with.

1.1 Assigning signers to API calls

• Deployed contract addresses are mapped to signer accounts (160-bit Ethereum account addresses), that are used to sign the contract deployment transactions. Further transactions to these contracts are signed by these accounts.
• Every time a write call is made against a contract deployed via EthVigil APIs, we retrieve the correct signer against it and use it to sign the corresponding transaction.
• “Signers” for fresh contract deployments can be chosen from a pool of signers we maintain.

Transaction shooter in action

• Each signer is mapped to a singleton instance, referred as transaction shooter for the rest of the article. These instances process queued requests against a signer account and fires transaction submissions (or-resubmissions) synchronously.

Assigning signers to 2 types of API calls: Fresh contract deployment and method call to deployed contract

1.2. Assigning `nonce`

In Ethereum, transactions are processed or executed in a certain order maintained by a count variable termed as nonce. The implications of transactions being assigned incorrect nonces have been discussed in greater detail in the previous article from this series.

1.2.1. Who should assign nonce anyway?

When we submit transactions to local Ethereum nodes/clients with hosted keys, the nonce assignment along with gas price and gas limit assignments are handled by the Ethereum client running on the local nodes. Signing of transactions are also done by the client.

However to have more control over how transactions will be executed on the Ethereum blockchain, we submit raw transactions to Ethereum nodes. To create raw transactions, nonce along with gas price and gas limit assignment have to happen outside an Ethereum client — in the transaction manager.

1.2.2 Maintaining nonce ordering for transactions per signer account

Persist incremented nonces, under valid locks, only when a transaction has been accepted by a client or when a transaction has been rejected due to “nonce too low” errors.

Overview of how requests queued against a signer are assigned nonces

The above diagram shows how a transaction shooter corresponding to 0xSigner1 processes requests synchronously while persisting updated nonce values depending on transaction submission outcomes. This helps us deal with two of the most common nonce related issues —

1.2.2.1. The “nonce too low” complaint

This happens when transactions from same signer account with higher nonces are already present in the pending queue of the client’s transaction pool. We handle this by doing exactly what error messages by Parity/Geth recommend — "Try incrementing the nonce” and retry

1.2.2.2 Nonce gaps → Transactions stuck in `queued` state

If a transaction submission fails for whatever reason, the transaction shooter retries the next queued request with the same nonce. This makes sure that the transaction shooter won’t submit transactions with higher nonces.

1.3. Creating the final raw transaction

Once the nonce assignment is done, the transaction shooter fills up the remaining required fields for an Ethereum transaction — gas price, gas limit, input data (encoded ABI for contract method call invocation) and value.

To create the final raw transaction, the signer account initiating the transaction must sign it. This can happen entirely off-chain using Ethereum client libraries. A valid signature is of 65 bytes length and is in R||S||V format, with R and S components having 32 bytes length each.

Overview of the entire transaction creation and signing flow

2. Transaction submission

Submission of raw transactions created by the transaction manager to Ethereum nodes results in either of the two following outcomes —

1. Transaction is accepted → Persist incremented nonce value against signer → Update persisted request information with transaction information → Respond to API call with transaction hash.
2. Transaction is rejected by client (nonce too low or other errors)→ handle error.

This following sequence diagram depicts how the transaction manager handles the entire API call → Transaction creation and submission → API response lifecycle.

Transaction manager sequence diagram

3. Transaction accepted but stuck in pending state?

Transactions find themselves stuck in a pending state for a long time generally when the gas price is not high enough to attract miners. There are 2 ways out of this sticky situation —

1. Replacing the transaction with a higher gas price
2. Cancelling the transaction

Once a transaction is submitted successfully we persist the transaction details against the request-id associated with the API call. To replace a transaction corresponding to a request-id and optionally cancel it —

1. Retrieve the transaction details.
2. Create transaction with an increased gas price that is meets the minimum price bump with all other fields staying the same.
3. To “cancel” the transaction, with the value set to 0, also set the to field, such that fromand to addresses are the same.
4. Sign and submit transaction

This will replace the transaction in the pending queue with a transaction that’s more attractive to miners due to the higher incentives associated.

A dedicated transaction manager enables more control over the entire transaction management life cycle, while also opening up possibilities around other derivative services based on analytics, dashboarding and further integrations.

BlockVigil exists to offload protocol and infrastructural kinks away from the application/business logic development lifecycle. The transaction manager is one of the most important moving parts that power our stateful API gateways.

Have you worked on anything similar? Got any wisdom to share with us? Let us know in the comments below!