Off-chain events and Tezos tokens indexing

In this article an approach to the token balance accounting is presented. This is a development of the ideas presented earlier taking into account the upcoming contract metadata standard TZIP-16.

In a few words, there is a problem when indexing operations that alter token balances: if the invoked method is not standardized (currently we have only FA1.2/FA2 transfer), or if there is an initial token distribution at the origination - it's not possible for the indexer to determine which particular balances have changed and how.

ULTIMATE GOAL
Get the list of changed token balances from the operation content and result.

The current approach is using custom handlers for known contracts. Obviously, it is tied to a specific indexer implementation and is not scalable, so we need a better alternative that is:

1. Flexible enough to cover the majority of cases;
2. Simple enough to implement/integrate with existing codebase;
3. Not tied to any specific entity nor implementation.

Off-chain events

The solution lies on the surface: we need to take all the custom logic out of the indexer, as well as to give the developers of the contracts the opportunity to edit it themselves.
This is effectively the concept of external (off-chain) views: a piece of Michelson code that is applied to the contract storage. Actually, one can write those external scripts in LIGO, SmartPy, Lorentz, or other high-level language and then compile down to Michelson.
The upcoming TZIP-16 standardizes off-chain views and defines two kinds that can be used in the contract metadata.

NOTE
The task of indexing metadata is beyond the scope of this article, we simply assume that for a particular contract we have a TZIP-16 compliant data file.

The existing view kinds are not enough for our needs, so we suggest adding some new ones. Below we will list cases that require the use of off-chain views, and the according implementation examples.

DEFINITION
In this article we use the term off-chain event (suggested by Seb Mondet) implying that this is a workaround until a native event logging system is implemented (suggested by Gabriel Alfour).

Initial storage

Let us consider a case when tokens are pre-minted and distributed when deploying a contract.

Except for the case when the entire ledger is copied from another contract ( Big_map copy, we’ll deal with it later), everything we need is in the resulting storage ( Big_map items are already included).

Let’s take the storage type of the sample contract above:

pair
    (big_map %accounts address 
                       (pair (map %allowances address nat) 
                             (nat %balance)))
    (nat %s)

The according event script would have:

• unit parameter type;
• Storage type similar to one in the target contract, except all big_map occurrences are replaced by map;
• Code must end with a FAILWITH instruction and there must be a value of type map address balance on top of the stack.

This value is actually all the balances changed during a contract call, and for each change indexer needs to know:

• Holder address;
• Token ID (in case there are more than one token within the contract);
• Resulting balance.

Note that we can omit token ID e.g. for FA1.2 or other contracts with a single token. Otherwise we have to put map (pair address nat) nat instead.

Here is a script that derives token balances from the given contract storage:

parameter unit;
storage (pair
            # we changed `big_map` to `map` to be able to iterate
            (map %accounts address (pair (map %allowances address nat) (nat)))
            (nat %s));
code {
    CDAR ; 
    MAP { CDDR } ;
    FAILWITH
}

Invoking the script with Unit parameter and origination storage we get an expected runtime error with:

{ Elt "tz1daj8qHeMtJ1XPCLGYRguH8BeNJc7YQ4ym" 30000000000000000000 ; 
  Elt "tz1euqMMX8dhf21M921UEq3f1EKy98FSYqTX" 30000000000000000000 ;
  Elt "tz1fWUkzMvnz4dmRn4kQMytahG6R4MMoobwp" 30000000000000000000 }

Finally, the metadata file would look like:

{
    "version": "1.0.0",
    "license": "MIT",
    "authors": ["Unknown"],
    "interfaces": ["TZIP-7"],
    "views": [{
        "name": "get-changed-token-balances",
        "description": "Get changed token balances from the operation receipt.",
        "pure": "true",
        "implementations": [{ 
            "michelson-initial-storage-token-event": {
                "storage": {"prim": "pair", "args": [...]},  // modified storage type
                "return-type": {"prim": "map", "args": [{"prim": "address"}, {"prim": "nat"}]},
                "code": [{"prim": "CDR"}, {"prim": "CAR"}, ... , {"prim": "FAILWITH"}]
            }
        }]
    }]
}

Note that we have a single view item that is responsible for deriving token balance updates and multiple implementations which are used depending on the situation.

Originally published at https://baking-bad.org on August 28, 2020, where you can find full version of the article.

Also, Read

• The Best Crypto Trading Bot
• Crypto Copy Trading Platforms
• The Best Crypto Tax Software
• Best Crypto Trading Platforms
• Best Crypto Lending Platforms
• Best Blockchain Analysis Tools
• Crypto arbitrage guide: How to make money as a beginner
• Best Crypto Charting Tool
• Ledger vs Trezor
• What are the best books to learn about Bitcoin?
• 3Commas Review
• AAX Exchange Review | Referral Code, Trading Fee, Pros and Cons
• Deribit Review | Options, Fees, APIs and Testnet
• FTX Crypto Exchange Review
• NGRAVE ZERO review
• Bybit Exchange Review
• 3Commas vs Cryptohopper
• The Best Bitcoin Hardware wallet
• Best monero wallet
• ledger nano s vs x
• Bitsgap vs 3Commas vs Quadency
• Ledger Nano S vs Trezor one vs Trezor T vs Ledger Nano X
• BlockFi vs Celsius vs Hodlnaut
• Bitsgap review — A Crypto Trading Bot That Makes Easy Money
• Quadency Review- A Crypto Trading Bot Made For Professionals
• PrimeXBT Review | Leverage Trading, Fee and Covesting
• Ellipal Titan Review
• SecuX Stone Review
• BlockFi Review | Earn up to 8.6% interests on your Crypto