Overview of the latest TzKT Indexer and API updates

Published in

Coinmonks

6 min readMay 27, 2020

TL;DR; The new version of TzKT Indexer and API brings very detailed and accurate Tezos baking data and is already available for testing in staging environment. The new version will be deployed to production next week. If your services or applications use TzKT API, please pay attention to the breaking changes section.

Several months ago, we released the “v1-preview” version of the TzKT indexer with its own API, which demonstrated high stability and fairly good performance with low resource consumption. Today TzKT API successfully processes over 200,000 requests per day from all over the world.

The only thing that made it less than perfect was the lack of baking-related data such as baking rights and staking rewards. And now, after months of hard work, we are proud to announce a big update which makes all baking stuff available via our API.

In this post we will skim through the changes and focus on key features.

Baking rights

First of all, we’ve added indexing of historical and future baking and endorsing rights, so that at the beginning of every cycle TzKT indexer fetches available rights from the node and then tracks their status, which rights were realized, which were missed, etc. Note: the indexer stores future baking rights up to priority 8 in order to optimize storage usage, not 64 as other indexers do.

Interesting fact: many people mistakenly believe that there are only 64 block priorities… That’s not how it actually works. Block priority doesn’t have a maximum value so you can get priorities up to 1000 or up to 10,000 or even more.

There are default indexes: by level, by cycle, by cycle and baker, which cover most common use cases. Thus, by specifying these parameters, you can significantly reduce the response time:

Uncovered rights

One of the unique features of TzKT indexer is baking rights status processing with additional analysis of the reasons a particular baking or endorsing right has been missed.

TzKT indexer splits missed rights into two categories: missed and uncovered. If a baker misses blocks or endorsements due to the lack of bonds (for example, when he is overdelegated) those rights are marked as uncovered, otherwise if baker misses blocks or endorsements for no apparent reason (for example, due to network issues) those rights are marked as actually missed.

Fair efficiency

Splitting baking rights into uncovered and missed allows to do a more accurate baker analysis and build more fair efficiency-based indicators.

The problem of all existing efficiency metrics is that they are completely unfair to overdelegated bakers. For example, if a baker has twice as much staking balance as staking capacity, he will receive twice as many baking rights. But his bond can cover only half of that, so the other half will be skipped, and baker’s efficiency will be 50% despite the fact that everything is fine with baker’s node. TzKT indexer solves this problem.

Staking rewards

We’ve added indexing of cycle statistics for all active bakers and delegators including summary of baking rights, staking rewards, bond usage, etc. All in one place. This statistic is also very detailed (see complete field descriptions in the API documentation). The more details, the more use cases, and the more various services and apps can be built on top of it, and the more complex payment schemes can be used by Tezos bakers.

It’s worth noting how TzKT indexer handles missed rewards. It is very different from existing APIs such as tzstats. For example, if Baker0 skips a block with priority 0 (expected reward is 40 tez) and Baker1 bakes this block with priority 1 (reward is ~26 tez, due to lower priority), tzstats will show that Baker0 missed 26 tez rewards, because actual block reward was 26. Honestly, we disagree with this approach.

When handling missed rewards, TzKT does answer the question “How much would a baker earn if he didn’t miss blocks or endorsements?”. In the example above, if Baker0 had not missed the block, he would have got 40 tez, rather than 26 tez, so the missed rewards are actually 40 tez. We believe that this is more accurate estimation of how much a baker would earn if everything was fine on his side.

Moreover, a missed block may lead to less endorsement reward as well as a missed endorsement may lead to less block reward. This is also taken into account.

Baker rewards:

Delegator rewards:

Reward splits

Finally! We implemented a working analogue of TzScan’s reward split, which perfectly fits bakers’ and payment tools’ needs. Moreover, we made it even better, more detailed and more accurate.

In addition, we added the “emptied” flag to all delegators in the reward split which indicates whether the delegator is emptied (removed from the node context) and should be re-allocated. This helps to determine if a payout transaction will burn 0.257 tez allocation fee without making extra requests and checking every delegator separately.

/v1/rewards/split/{baker}/{cycle}

Snapshots

Obviously, we’ve added account snapshots indexing. However, all snapshots are stored only for the last 13 cycles (in the mainnet), and for older cycles only selected snapshots are available. This allows us to significantly optimize storage usage.

Actually, we could store even less: only PRESERVED_CYCLES + 2, because as we know “at any point, the shell will not implicitly accept a branch whose fork point is in a cycle more than PRESERVED_CYCLES in the past” and at the beginning of the cycle N a snapshot is taken from the cycle N - 2. This actually means that the maximum mainnet roll back is 5 cycles and 2 cycles more for taking a snapshot, so we need to store snapshots at least for the last 7 cycles. However, at this stage we store a little more for debugging purpose.

TzKT API doesn’t provide separate endpoints for accessing selected snapshots, because it is already possible with existing endpoints. For example, snapshots of a particular delegator can be accessed this way:

/v1/rewards/delegators/{address}?select=cycle,balance,baker

Cycles

We’ve also added cycle data, including a brief staking summary such as how many bakers were active in the particular cycle, how many rolls participated in baking rights distribution, how many tez were delegated, etc.

The most interesting thing, in our opinion, is the total rolls count, because together with the rolls count of a particular baker it allows to estimate such metrics as historical capacity, luck ratio, etc.

/v1/cycles

Breaking changes

Fortunately, there is only one breaking change, concerning the “select” query parameter.

By specifying this parameter, one can request only required data. This significantly reduces network traffic and also response time, because SQL queries would likely contain fewer JOINs.

In the current version “select” parameter works only in the “values” mode:

?select=cycle,balance => [10, 1234]

In new version “select” parameter has been extended with the “fields” mode which is now a default mode:

Fields mode: ?select=cycle,balance => { “cycle”: 10, “balance”: 1234 }

Values mode: ?select.values=cycle,balance => [10, 1234]

So, in order to reproduce the previous behavior “?select=” should be replaced with “?select.values=”.

Also, deprecated parameters “n” and “p” have been removed, as it was mentioned in the documentation: “Deprecated parameter. Will be removed in the next release.”

Staging environment

We decided to run a staging environment before deploying to production. This allows to test latest features and update client applications in advance so that the breaking changes in API won’t affect the production environments.

Staging API is available here: https://staging.api.tzkt.io/, so feel free to play with it. Any feedback would be much appreciated!

Next week we plan to deploy the new version to production for all supported Tezos networks.

Originally published at https://baking-bad.org on May 27, 2020, where you can find full version of the article.