Interacting with Ethereum Smart Contract Events in Go with the Civil Events Crawler

Photo courtesy of Aidan Granberry

At Civil we’re creating a platform to support sustainable journalism by using the Ethereum blockchain. In a previous blog post, we introduced the Civil Events Crawler which speeds up data access to the Civil Decentralized application (Dapp), enables developers to build apps using Civil data and eventually will be generic enough to capture any smart contract data.

In this post, I’ll be diving into the code of the Civil Events Crawler. I’ll explain how we interact with smart contracts, why we use Golang and the go-Ethereum library for server-side Ethereum development and how we use code generation with Go. We’ve open sourced our crawler and invite collaborations!

Under The Hood Prerequisites: How a Dapp Interacts With a Smart Contract

Ethereum smart contracts are defined in human-readable code such as Solidity and are run in the Ethereum Virtual Machine (EVM). In order for the EVM to understand the Solidity code, we need to compile it into machine-readable bytecode that the EVM understands by using a compiler called solc.

ABI

Once a smart contract is deployed, a Dapp can interact with it by sending transactions to the blockchain through the contract’s functions. It does that by sending a function call with its input data to the EVM. As mentioned earlier, the EVM only understands bytecode, so the application needs to know how this data must be encoded. This is where the Application Binary Interface (ABI) comes in. The ABI specifies a common encoding scheme between the EVM and the application. Deploying your contract with solc generates this set of rules in bytecode and in a human readable JSON format.

The ABI also defines the encoding scheme for events emitted by the smart contract. Events are defined within the smart contract’s function, and can be triggered upon an Ethereum transaction. They are primarily used in Dapp development so that the front-end can react to data from the function, but also as a cheaper form of storage. More specifically, when a function is called and an event is triggered, the event is stored as a Bloom filter in the transaction’s block header, referred to as the transaction log. You can read further about this in the Ethereum Yellow Paper, where events are only referred to as logs and the different logging operations for the EVM are defined.

Here are 2 screenshots describing the following scenario that could happen on Civil. These screenshots are from a test transaction to Civil’s Token Curated Registry(TCR) contract on the Rinkeby test net.

When a newsroom completes an application for a newsroom, the Dapp sends an Ethereum transaction with input data (red marked “Input Data” below), describing a call to a function called `apply` in Civil’s TCR. When this happens, the smart contract emits an `Application` event containing data like the newsroom address, amount of tokens this newsroom spent, and some other fields (red marked “Topics” in the next screenshot) .

You could read the Solidity Docs for ABI to understand what the different bytes mean in all these encodings, for example, the first 4 bytes of `InputData` is defined as “the first 4 bytes of the Keccak hash of the ASCII form of the [method] signature”. There are many blog posts that go through encoding the bytes step by step. It’s pretty painful to do that though, so just read on!

How To Decode the ABI — Why we’re using Go

We don’t need to understand the ABI encoding schema in order to start interacting with smart contracts. There are many libraries in different languages that do this for us.

Which library should we use? The most common work happening in Ethereum is around developing Dapps, so there is naturally a large developer community around Javascript and the web3js ethereum API. Since our need for the crawler was server-side, Javascript wasn’t a natural choice. Go has some great features to support server-side development: it’s fast, it supports concurrency through light weight goroutines, and it’s a compiled language which makes it less error prone compared to interpreted languages like Python. Furthermore, Geth, one of the official Ethereum clients, is written in Go, which makes Go a native language to interact with the blockchain. All of these features made Go and using the go-ethereum library a natural choice for us.

Go and Ethereum

Go-ethereum makes it super easy to develop a Dapp that uses the ABI to interact with smart contracts. It has a Go binding generator that converts ABIs to type-safe Go packages. Here is a great tutorial from the go-ethereum wiki, so I won’t go into details, but all you have to do is run the following abigen command on the JSON ABI that’s produced when you compile your smart contract using solc.

$ godep go install path-to-goethereum/cmd/abigen
$ abigen --abi PathToCivilTCRContract.abi --pkg main --type CivilTCR --out CivilTCRContract.go

abigen uses code generation in Go, which produces a lot of code that is simple to maintain. Every time your ABI changes, you run the abigen command again to regenerate the bindings. Your ABI should not be changing very often so this shouldn’t be a problem.

Our Code Generation around Events

The abigen command also generates code that wraps around events in your ABI, contrary to what the go-ethereum wiki states. You can check out this PR to see how go-ethereum added Go wrappers around events. This is extremely useful because all the data emitted in the event, seen as the Topics field in the screenshot of encoded log data above, is represented in its own type-safe struct and all the Solidity variable types are already converted into Go variable types.

The following is the JSON ABI definition of an Application event described earlier in this post:

{
"anonymous": false,
"inputs": [
{
"indexed": true,
"name": "listingAddress",
"type": "address"
},
{
"indexed": false,
"name": "deposit",
"type": "uint256"
},
{
"indexed": false,
"name": "appEndDate",
"type": "uint256"
},
{
"indexed": false,
"name": "data",
"type": "string"
},
{
"indexed": true,
"name": "applicant",
"type": "address"
}
],
"name": "_Application",
"type": "event"
}

Compare this to the next code block, an Application event in Go struct form with all fields defined. As you can see, abigen does all the type conversions (i.e. a Solidity uint256 gets converted to a big.Int pointer).

// CivilTCRContractApplication represents a Application event raised by the CivilTCRContract contract.
type CivilTCRContractApplication struct {
ListingAddress common.Address
Deposit        *big.Int
AppEndDate     *big.Int
Data           string
Applicant      common.Address
Raw            types.Log // Blockchain specific contextual infos
}

NOTE: this is an older version of the Application event so the fields are a bit different than what you might currently see in Civil’s Dapp code.

As the previous blog post mentioned, the crawler both subscribes to new events and retrieves past events. When we run the abigen command, it generates a Filter<event name> method that retrieves and iterates through past logs that hold all the events emitted by Civil’s smart contracts and generates a Watch<event name> method that subscribes to future logs.

Let’s continue with the application scenario described earlier and walk through some more code! This is the function definition for a wrapper generated by abigen around the Civil TCR Application event to help us filter for past events:

// Solidity: event _Application(listingAddress indexed address, deposit uint256, appEndDate uint256, data string, applicant indexed address)
func (_CivilTCRContract *CivilTCRContractFilterer) FilterApplication(opts *bind.FilterOpts, listingAddress []common.Address, applicant []common.Address) (*CivilTCRContractApplicationIterator, error)

opts is a series of options for the filterer like the block number to start filtering from, and the rest of the arguments are addresses used to refer to the Application event. FilterApplication is a function of the struct CivilTCRContractFilterer, which contains a boundContract struct responsible for low-level interfaces that need to be called to read the Ethereum logs. FilterApplication returns an iterator which you can use to iterate through the retrieved events.

Next, is a WatchApplication that helps us watch for new Application events. It takes the same function arguments, with an additional channel of Application structs through which events come in. The function returns a subscription which can be used to unsubscribe from the stream of events being sent to the provided channel.

// Solidity: event _Application(listingAddress indexed address, deposit uint256, appEndDate uint256, data string, applicant indexed address)
func (_CivilTCRContract *CivilTCRContractFilterer) WatchApplication(opts *bind.WatchOpts, sink chan<- *CivilTCRContractApplication, listingAddress []common.Address, applicant []common.Address) (event.Subscription, error)

For the Civil Crawler, we’ve written a code generator that loops through the events defined in the supplied smart contract ABI, and generates both watchers and filterers around each event using those 2 functions generated by abigen above.

Here are some of the function headers of our template code.

func (f *{{$.ContractTypeName}}Filterers) startFilter{{.EventMethod}}(startBlock uint64, pastEvents []*model.Event) (error, []*model.Event)

When we start a filterer for a smart contract in gen/filterergen_template.go, we supply a starting block, and append to pastEvents that we’ve seen already.

And here’s a function header from our watcher code generator template.

func (w *{{$.ContractTypeName}}Watchers) startWatch{{.EventMethod}}(eventRecvChan chan *model.Event) (event.Subscription, error)

For watchers in gen/watchergen_template.go, we use go channels to receive events from the stream of events described earlier, for each event name.

To write the code generation, we use this general function header:

func generate(writer io.Writer, tmplName string, tmpl string, tmplData interface{}, gofmt bool)

where tmplName refers to the template name, tmpl refers to the variable the template data is stored in, tmplData is a struct of data (like ContractTypeName) containing variables that need to be passed to templates, gofmt, a boolean to gofmt the generated code, and writer, an io.Writer to write the generated code to.

Next Steps: A Generic Events Crawler

These templates are generic enough that theoretically for any smart contract, given its ABI and abigen generated code, they can generate watchers and filterers around the smart contract events.

If this interests you, check out our repo and run make generate-watchers and make generate-filterers to see the code generator for yourself. As we’ve mentioned, there’s still some work to be done. There’s a main file here which performs the code generation, but it’s currently relying on a list of contracts to track here. One starting point to help us out would be to figure out a way we can automate the generation of a file like contracttypes.go for any smart contract ABI file.

Hopefully, this post has helped you understand how applications use the ABI to interact with events under the hood and how our Civil Events Crawler is being built. This is the first of a series of posts we plan to write about how we wrote, how you can contribute, and how you can use the Crawler.

We are actively working on the code, but feel free to read, use, or add to it. Please submit any feedback, issues, or contributions to our Github repository.

If you are interested in our smart contracts, Dapp, and other tools we have built, check out our main monorepo.

You can also reach out to us on Telegram, or send an email to developers@civil.co.