Index Canisters: Account-based search for ledgers on the Internet Computer
Index canisters enhance ledgers on the Internet Computer by introducing account-based transaction queries. They allow users and dapps to quickly query the history of a ledger transaction log and retrieve any transaction that a given account was involved in.
Written by Nikolas Haimerl
Querying transactions without index canisters on the Internet Computer is a bit like searching the whole library each time you are looking for a book. An index canister is equivalent to adding categories, so that you know to find a Stephen Hawking book in the Physics section.
The challenge today with the ICP ledger is that developers who want to access the transaction history of an ICP account would need to process the whole ICP ledger canister to get this list, which is both inefficient and inelegant, or store and update that data manually. The index canister addresses this issue by providing an on-chain solution that can be easily queried.
The Network Nervous System (NNS), the DAO governing the Internet Computer stores transactions of the native token (ICP) on a ledger (ICP-Ledger) that runs on chain. These transactions are stored as a blockchain — a linked list of transactions with each block storing the hash of its parent block. Users can retrieve blocks by querying the canister that runs the ICP-Ledger logic. If users wants to query transactions that belong to a specific account , they would have to fetch all blocks, then search through them to find these transactions much in the same way you would have to search the whole library for the books of a specific author without a catalog systemTo solve this impracticality, the ICP-Index canister was created and added to the NNS to provide users with an account-based search endpoint that returns all transactions of any given account within a block range that is specified by the user.
As ICRC-1 Ledgers are very similar to the ICP ledger and have a similar issue, ICRC-1 Index canisters were also created to essentially serve the same purpose. The key difference between ICP and ICRC-1 Ledgers is explained later in this article. ICRC-1 Ledgers are used in main ICP features such as SNS DAOs and in the implementation of ckBTC, a decentralized bitcoin twin. This article will go into more details on how these two types of index canisters work, what their differences are and how dapps interact with them.
How do index canisters work?
As transactions are created on the ICP or ICRC-1 Ledgers they are stored in a blockchain. As soon as an index canister is created,it will start fetching blocks from the ledger it was pointed to by the entity that deployed the index canister. This fetching of blocks will be triggered at a fixed interval, usually every few seconds. This means, there will always be a small lag between the ledgers and the index canisters. The interaction between an index canister and its assigned ledger is a one way communication. So a ledger does not call any of the index canisters’ endpoints. Rather, the index canister calls the ledgers endpoints. The ledger is also the only canister that the index canister interacts with. No other canister-to-canister communication takes place.
When fetching blocks the index canister will store the blocks in stable memory logs. The index canisters also store the account id together with the index of the block in stable memory in a BTreeMap format. The key of this BTreeMap is the account and the index of the block where the transaction is stored. The blocks are stored in an append only list.
When retrieving transactions for an account this BTreeMap is used to search through the keys and extract all block indices corresponding to the account. The stable memory Log is then accessed at the specific indices where the transactions occurred. It then extracts the transactions from the block and serves them to the user. This allows for fast search and response time for any entity requiring transactions of an account from an index canister.
Differences between the ICPIndex and ICRC-1 Index canisters
There are two major differences between the two index canisters. Firstly, there is only one ICP-Index canister which queries, processes and serves requests for transaction data created on the ICP-Ledger. It lives on the same subnet as the NNS. ICRC-1 Index canisters, on the other hand, are multiple, ,each being connected to a specific ICRC-1 Ledger. These can be the ledgers used for SNS DAOs, for example, or the ledger used to record ckBTC transactions.
Another aspect that sets these canisters apart is their fundamentally different types of account representations. The ICP-Ledger uses AccountIdentifiers, while the ICRC-1 Ledgers use Account as their respective user account representation. The specifications and details are explained in the linked resources. The key takeaway here is that these two account representations are not interchangeable. An AccountIdentifier is essentially a hashed version of Account to increase privacy. This also means that an AccountIdentifier cannot be converted back to an Account. This difference has implications for the format of the returned transactions of the index canisters. The transactions returned by the ICP-Index canister will contain AccountIdentifiers, while those of the ICRC-1 Index canisters will contain Accounts. A user querying both index canister types will have to account for this key difference in whatever task they seek to do with the data fetched from the index canisters.
How to interact with the index canisters
The interfaces to interact with the index canisters can either be found by looking at the interface of one of the currently deployed index canisters (ICP-Index, ckBTC-Index) or through the candid files of the reference implementations.
https://raw.githubusercontent.com/dfinity/ic/d628aa32d72565c9e5258d033b3c0be194c77b24/rs/rosetta-api/icp_ledger/index/index.did
The index canisters make it possible to not only retrieve the current balance for a specific account, but also to fetch a range of blocks or a range of transactions for specific accounts. For instructions on how to interact with canister interfaces see this guide.
More useful links
Deploy an ICP-Ledger locally
Deploy an ICRC-Ledger locally
Deploy a Candid UI to interact with canisters through a front-end
