Animagus Part 1: Introduction
by Xuejie Xiao
This is part one in a series that will introduce and explain Animagus, the problem it aims to solve, and how you can make the best use of it.
We believe that Nervos Common Knowledge Base (CKB) has unmatched potential among modern generation blockchains. Currently, specially built tools are required to tap into CKB’s full potential. Animagus is such a tool.
The name “animagus” comes from the Harry Potter series. A witch or wizard is called an “animagus” if she/he can transform into an animal and back again. Depending on the specific witch or wizard, the animal they can transform into also differs. I personally believe the name “animagus” fits well with the scope of this project — hopefully you are with me after reading this series.
Seriously, what is Animagus?
Historically there have been many ways to describe what Animagus does. Some would say it’s a dApp framework, some might say it’s a layer atop CKB, but the description I like these days is “an account layer for CKB.”
While most trending blockchains these days use an account model, CKB uses a UTXO-like model which is inherited from Bitcoin. While the UTXO model certainly has a lot of advantages, there is one signficant drawback: it is harder to program when compared to the account model.
This is where Animagus comes into play. As we will see in this post, and with future posts, Animagus provides a solution to most — if not all — of the programming hurdles one encounters in the UTXO model. We are hoping Animagus can fill in the gaps of the UTXO programming model, and enables the realization of the full benefits of CKB.
On the technical side, Animagus can be viewed as an AST runner:
- A developer designs the functionality they want into an AST, or Abstract Syntax Tree. (If this AST thing sounds strange to you, please bear with me for a second. For now, you can just assume it is a short program)
- The AST will be expressed in the Protocol Buffers format, so Animagus remains language agnostic
- When booting, Animagus loads the AST. Based on the definition of the AST, Animagus can provide different behaviors:
- It can read CKB state and index certain cells matching predefined conditions, such as all Nervos DAO deposit cells
- It can provide RPCs that a developer can call from other services for certain results, such as fetching the balance of an account or assembling a transfer transaction
- It can generate validator smart contracts used on chain from AST definitions
Example
I’m sure this all sounds quite complicated right now. Let me explain how this works with a real example: in the UTXO model, an account can just be viewed as the set of live UTXOs (or cells in CKB’s terminology). The balance of an account in CKB is just the sum of capacities from all the live cells in the set.
The core CKB has never provided an RPC to fetch the balance of an arbitrary account. Yes, the CKB node’s indexer has a way to fetch the current balance of indexed cells, but what if we want to fetch the balance of any random account? This is where Animagus can help: we are going to design an AST here, so that when Animagus boots, it provides an RPC which can be used to fetch the CKB balance of any account.
To make our example suit the introduction to Animagus better, let’s also create the following restrictions:
- Our AST only supports accounts using CKB’s default lock script.
- It really is the
argsthat define the account a cell belongs to. In this case, we will write the RPC construct so it accepts a series of bytes, the RPC will then return the balance of the account when we treat the series of bytes asargs(recall that when using the default lock script,code_hashandhash_typeare fixed values).
As mentioned above, Animagus is language agnostic. While Animagusitself is written in Go, you are free to use any language to build the AST and for calling Animagus, as long as GRPC is supported in language of choice. In our example of this series, we will use Ruby to build the AST and call Animagus, but you are not limited to this.
There are 2 parts required to build an Animagus AST: querying and transforming.
Querying
Querying in Animagus serves 2 purposes:
- Indexer uses querying to index needed cells
- RPC server uses the querying part to fetch cells needed to serve the RPC.
In our balance example, the query actually queries for all cells belonging to a specified account, but this is completely up to the defined AST. A different AST might be querying cells from multiple accounts, or cells that have other things in common. Later in the transforming phase, we would extract capacities from all queried cells and do a summation to get the final balance.
To define a querying part in Animagus, all we need to do is define a function that returns true for those cells we want to query for, and returns false otherwise. The following snippet shows how we can define such a function in Ruby:
PARAMS = ["0x12345678ab12345678ab12345678ab12345678ab"]
def filter(cell)
cell.lock.code_hash == "0x9bd7e06f3ecf4be0f2fcd2188b23f1b9fcc88e5d4b65a8637b17723bbda3cce8" &&
cell.lock.hash_type == "type" &&
cell.lock.args == PARAMS[0]We use PARAMS to denote parameters provided by RPC clients. In this case, the RPC request accepts one parameter, which is the script args part denoting the account owner. To make this resemble an AST more, we can modify the Ruby code here a little bit:
PARAMS = ["0x12345678ab12345678ab12345678ab12345678ab"]
ARGS = [cell]
def filter
ARGS[0].lock.code_hash == "0x9bd7e06f3ecf4be0f2fcd2188b23f1b9fcc88e5d4b65a8637b17723bbda3cce8" &&
ARGS[0].lock.hash_type == "type" &&
ARGS[0].lock.args == PARAMS[0]While typically we won’t write code this way, it more closely resembles what an actual AST might look like: special ARGS and PARAMS nodes are used to represent arguments and parameters passed to the function (either by parent AST node or RPC users).
Tip: notice there are 2 args used: we use ARGS to denote a special AnimagusAST node, representing that a part of AST is actually a function. ARGS will be filled by the parent node when executing the AST; while script args represent the args part in CKB’s script structure, which is filled by user to denote the account owner.
Now we can construct an AST based on the above code snippet:
This is what AST means: it’s a tree representation of a piece of code, where values in the code become a node in the tree, and operations/functions in the code also become nodes with arguments as child nodes. AST is actually widely used in compiler technologies and almost all the programming languages we use are first parsed into an AST before more operations are executed.
When we have the querying AST function, we can feed it to a special QueryCell node, which then completes the querying part.
For space consideration, we have omitted certain child nodes that already exist in the previous AST graph.
Why AST?
One question you might have is why use AST directly here? Why not just build a small programming language? We actually thought about this in the design phase, but we finally voted against it for several reasons:
- There are just far too many programming languages out there, whether they are general purpose programming languages or smart contract focused languages, it really seems that everyone is busy implementing their own language and each person has their own preferences when it comes to syntax. The result is that whenever someone wants to pick up a new blockchain, the first thing they need to do is learn a new programming language. We want to change this, so we are only defining the core AST we have here. Everyone is free to implement whatever language style they want and compile their mini language to Animagus AST. This way, we can ensure maximum freedom when using Animagus.
- It’s actually more than just programming languages that compile directly to AST. People with deep learning backgrounds will know that compute graphs and DSLs are very popular in certain fields. In those situations, you can just continue to use your existing language and use all sorts of DSLs to build the logic you need. Later you can export the complex logic defined via DSLs and run them on modern GPUs. In fact, later we will show exactly how you can define Animagus AST in this way, in plain Ruby.
- There is actually more to the capability of an AST than just programming languages. In different fields, many people are also building awesome things without programming at all. For instance, talented game designers are building awesome new games via things like Unreal Blueprint everyday. These developers don’t need programming languages to fulfill their choice of game mechanics. We believe in a diverse future, by providing Animagus AST directly, we could also enable blueprint-style dapp building, where talented people without programming skills can also craft the dapp they need.
In a nutshell, just as CKB provides maximum flexibility to the blockchain world, we believe an AST-based design also provides maximum flexibility to Animagus, which unlocks many more possibilities.
Transforming
With a query part in place, we can start to build the transforming phase. 2 more functions are needed in the transforming phase. The first one is used to extract capacity from a cell:
The second one just adds 2 capacity values together:
As you have seen, not all ASTs have to be super complex, some can be really simple as long as they serve their purposes.
Now we can assemble the transforming part together:
There are 2 new types of nodes in the graph:
- “Map” accepts a list (in this example, the list of cells returned from QueryCell node) and a function. It applies the function to each of the elements in the list and returns the resulting list.
- “Reduce” accepts a list, a function and an initial value. It applies the function with the initial value and the element, uses the return value to replace initial value, and goes on until it runs out of elements in the list. The result of “reduce” is a single value.
If you are familiar with functional programming, these are exactly the plain map/reduce functions you will love. The reduce node thus contains the transformed value, which is the total capacities of all cells in the specified account.
Piecing Together
The remaining work here is trivial, we give the RPC method a name, put it together in a Call node, then create a Rootnode containing it:
When Animagus boots, it accepts a protobuf serialized message of Rootnode. A Root node might have multiple Callnodes (later we will see it might also contain multiple Stream nodes, but this is outside the scope of this initial post). Animagus analyzes each Call node and performs each operation:
- It extracts all the
QueryCellnodes from all theCallnodes (note it’s allowed for oneCallnode to have multipleQueryCellnodes). For each new cell in each block, Animagus runs the cell against the function described in theQueryCellnode. If the function returnstrue, Animagus will store a record of the cell (a.k.a. index the cell). - When an RPC request is made to Animagus, Animagus locates the corresponding
Callnode and runs the AST. When aQueryCellnode is encountered, it will fetch the corresponding cells based on indexing records.
As shown in the above example, we’ve implemented a way to index cells and provide RPC calls for fetching the CKB balances of each account.
Conclusion
By designing Animagus, I started to realize that from a programming perspective, the account model and the UTXO model might not be so different. While the account model provides account state directly, the UTXO model just spreads account state across a set of UTXOs (or live cells in CKB’s case). There’s nothing stopping one from using tools like Animagus to help organize and transform the UTXOs into a form that is directly readable, just like the account model.
I hope you are not bored by this first post :) In the next post I will show you how you can build the balance AST in Ruby, and run it for real in Animagus.
