Take a look at Geometry Labs’ newest collaboration with the ICON Foundation

Getting started with the ICON Streaming Data Platform

Published in

Geometry Labs

7 min readMar 30, 2021

Hello ICONists. Today we’re announcing the general availability of our first contribution as Geometry Labs: the ICON Streaming Data Platform. The platform was supported by a grant from the ICON Foundation.

With this new set of tools, developers are able to work with live copies of the blockchain and perform different types of queries and calculations that would normally be slow / difficult if run directly against a blockchain node. We are currently contributing its use for Balanced, and several other high profile ICON projects have expressed that the tool’s capabilities can unlock a whole slew of opportunities where responsive APIs are needed. Currently, to get data from the blockchain, one would use the standard JSON-RPC interface. This project now extends this API to include multiple different REST, websockets, and GraphQL interfaces.

The stack is packaged as a collection of microservices in a single docker-compose that you can find at the main project repo, icon-api. Building on the past work we did with icon-etl, we made modifications to the tool to support streaming data into Apache Kafka, an enterprise grade scalable message queue, which powers the rest of stack. Once in Kafka, data flows through various microservices that enable all the API endpoints and persistence in MongoDB.

There are several ways developers can use the stack depending on their specific use case. For querying historical data, we recommend using the REST or GraphQL interfaces that are built with Python’s FastAPI framework and golang’s graphql-go. These endpoints were built initially with the most basic functionality for querying blocks, transactions, and event logs though they can easily be extended to include more enriched data such as balances or transaction histories along various dimensions (PRs welcome!). Check out our docs for more information on these endpoints.

While querying historical data is useful for a lot of applications, we foresee the most useful endpoints being the websockets connections that are able to subscribe to contract events on the blockchain. Before getting into how to do that, let’s take a closer look at how the platform works in practice and then we’ll dive into how to subscribe to events on the blockchain.

Deploying your own setup

To get started using the platform, you can either use our existing beta deployment, run it locally with docker, or use our terraform module to setup a node on AWS with Ansible.

For use with testnet, you can use http://yeouido.geometry-dev.net with mainnet coming soon at http://icon.geometry-dev.net/. Please note this is an early beta. We will be putting in a CPS proposal in the near future to run the stack on our production systems at geometry.io for access to the community.

If you want to run the stack yourself, you will need to have docker/docker-compose installed and clone the icon-api repo to your local machine.

git clone https://github.com/geometry-labs/icon-api
cd icon-api 
docker-compose up -d 
docker-compose ps  
# To explore additional features, checkout the README and other 
# docker-compose.<override>.yml files "{{user `vpc_id`}}"

You’ll need to wait a few minutes while Docker downloads all of the containers (there are quite a few, so make sure you’re on a fast internet connection). You’ll then see something similar to this while Docker creates all of the containers for the platform:

Once Docker has completed the creation of all of the containers, you will need to wait a few more minutes before everything is ready to go. You can run docker-compose ps to see the status of all of the containers and should see something like this:

You’ll know you’re ready to start using the platform once all of the containers have an up state, except for the two *-init containers (which should say Exit 0).

For automated deployments using Terraform and Ansible, check out our docs for information on how to do that.

How to use websockets

If you would like to experiment with sending requests to the platform over websockets, we recommend using the Simple Websocket Client for Chrome.

Once you’ve installed and opened the client, you’ll see a window like this:

For the remaining part of the tutorial, we’ll be using testnet with a url of ws://icon.geometry-dev.net/ws/.

Streaming block and transaction data

Before we take a look at how to filter data from transaction logs, let’s first use the platform to see every block and transaction as they come in.

The endpoint for this is slightly different than the one that you’ll use to register and listen to transaction log events, but is equally easy to use. The base URL is the same for both blocks and transactions, ws://icon.geometry-dev.net/ws.

To begin streaming in blocks as they are made, just open a connection to ws://icon.geometry-dev.net/ws/blocks and you will see an output like this

And to stream in transactions, you’ll open a connection to ws://icon.geometry-dev.net/ws/transactions and see an output like this

For both data streams, you’ll find the data type as part of the response object so you will know exactly what you’re receiving.

But perhaps you’d like to see information relating to the output of certain transactions. The next section will go into more detail about that.

Subscribing to log outputs

To subscribe to filtered ouputs, we will be using the ws://icon.geometry-dev.net/ws/admin endpoint.

Let’s say you wanted to know about every time a TapToken changed hands. If we take a look at the TapToken SCORE code (cxc0b5b52c9f8b4251a47e91dda3bd61e5512cd782), we can see the Transfer() method.

{
  "name": "Transfer",
  "type": "eventlog",
  "inputs": [{
    "name": "_from",
    "type": "Address",
    "indexed": "0x1"
   },
   {
    "name": "_to",
    "type": "Address",
    "indexed": "0x1"
   },
   {
    "name": "_value",
    "type": "int",
    "indexed": "0x1"
   },
   {
    "name": "_data",
    "type": "bytes"
   }
  ]
 }

This method has three inputs _from, _to, and _value, which all have the indexed flag set (meaning they’ll show up in the blockchain log outputs).

Now that we know what we want, let’s subscribe to the data feeds for these events. To do that, we’ll need to build our object to send to the platform.

Generally, when you subscribe to events, you will be creating a new broadcaster. This allows the platform to bundle up related events and pass them back along the same connection. When we create a new broadcaster, we need to submit data similar to the following.

{
 "connection_type": "ws",
 "endpoint": "wss://demo",
 "log_events": [{
   "address": "cxc0b5b52c9f8b4251a47e91dda3bd61e5512cd782",
   "keyword": "Transfer",
   "position": 1
  },
  {
   "address": "cxc0b5b52c9f8b4251a47e91dda3bd61e5512cd782",
   "keyword": "Transfer",
   "position": 2
  },
  {
   "address": "cxc0b5b52c9f8b4251a47e91dda3bd61e5512cd782",
   "keyword": "Transfer",
   "position": 3
  }
 ]
}

The important part to note is the log_events list, where we have put in the three objects we want to receive events on. You will need to specify the contract address, the method keyword (i.e. the name of the method), and the position of the argument where the data you want is.

In our case, since we want three parts from the same method, we need to pass in three separate objects.

When you’ve successfully POSTed this object, you’ll receive a confirmation that your events have been registered, along with the unique ID for your registration. Now, whenever an event matching one of ours, we’ll receive a message like the following:

{
    "address": "cxc0b5b52c9f8b4251a47e91dda3bd61e5512cd782",
    "keyword": "Transfer",
    "position": 1,
    "value": "hx1f7f21cb97c5766abed06992fe9ed451ccfb983a",
}

Once you’re finished receiving your events, just send another request with the broadcaster ID, like the following:

{
    "broadcaster_id": "4027f4a8-f399-4626-afff-68de53460a60"
}

Tracking transactions

Now that we’ve seen how to receive events from contract log outputs, you might also want to receive notifications based on transactions.

For instance, perhaps you are interested in tracking changes in staking, so you’d like to receive transactions addressed to cx0000000000000000000000000000000000000000. Let’s take our example broadcaster that we registered earlier and update it to also receive these events.

When registering transaction events, you will need to prepare the transaction_events object to include in the broadcaster registration object. These objects look like

"transaction_events": [
  {
    "to_address": "cx0000000000000000000000000000000000000000",
    "from_address": "cx0000000000000000000000000000000000000001"
  }
]

If an address is not specified, it is assumed to be a wildcard search.

Modifying the broadcaster

Using our example from before, we can modify our existing registration by submitting the desired state of the broadcaster and the ID.

{
  "broadcaster_id": "4027f4a8-f399-4626-afff-68de53460a60",
  "connection_type": "ws",
  "endpoint": "wss://demo",
  "log_events":
    [
      {
        "address": "cxc0b5b52c9f8b4251a47e91dda3bd61e5512cd782",
        "keyword": "Transfer",
        "position":  1
      },
      {
        "address": "cxc0b5b52c9f8b4251a47e91dda3bd61e5512cd782",
        "keyword": "Transfer",
        "position":  2
      },
      {
        "address": "cxc0b5b52c9f8b4251a47e91dda3bd61e5512cd782",
        "keyword": "Transfer",
        "position":  3
      }
    ],
  "transaction_events": [
      {
        "to_address": "cx0000000000000000000000000000000000000000",
      }
    ]
}

If we wanted to not receive the log events, we would just omit that object and our broadcaster would be modified appropriately.

Recap

With the new ICON Streaming Data Platform, developers and ICON users alike are now able to easily access the blockchain data they need to build responsive applications on the ICON blockchain.

You can quickly try out the platform by using our beta endpoints for mainnet at icon.geometry-dev.net and testnet at yeouido.geometry-dev.net.

As always, we welcome any feedback on our tools and look forward to hearing from you.