A Daml ledger tells a story — it’s time to show it to everyone
A Daml-based application seems to be an ordinary state machine. Sure, the application UI only displays the current state of the system and the transition options. But the unique feature of a Daml ledger is that it stores not only the state of the system but also the state transitions. For explaining the idea of Daml ledgers and the story told by a specific ledger (and for audit, graph neural network based fraud detection, etc.), it’s a good idea to show the full picture.
As a preview of upcoming attractions: at the end of this blog post you will understand what this event graph means and how it can be created:
The Daml ledger‘s event graph
The summary of the working mechanism of a Daml ledger, based on the Daml ledger model is the following:
- The ledger stores an event graph, containing creation and exercise event nodes, as well as “consuming” and “consequence” edges.
- The event graph grows in an append-only manner during normal operation, which guarantees atomic transaction handling
- The current state of the ledger (also called the active contract set, ACS for short) consists of such creation events for which no consuming exercise was recorded on the ledger.
- Ledger parties have access to a subset of the event graph containing the events they are entitled to see. This guarantees sub-transaction privacy.
- The ledger can be pruned under certain circumstances, removing events which are not needed for operation.
Let’s see these points in more detail.
The ledger stores an event graph.
(Please not that the Daml documentation speaks about the ledger causality graph which is a related but somewhat different concept, and is important for validating transaction requests which I don’t cover in this blog post.)
The nodes of the graph are creation events for the creation of contracts, and exercise events for exercising choices on previously created contracts. (Plus some other kinds of events which I omit from this account for clarity.)
For those of you who are not closely familiar with what Daml contracts and choices are, here is a short detour.
In the Daml code, the equivalent of contract creation and choice exercise is the contract template and the choices on these templates.
type AssetId = ContractId Asset
template Asset
with
issuer : Party
owner : Party
name : Text
where
ensure name /= ""
signatory issuer
observer owner
choice Give : AssetId
with
newOwner : Party
controller owner
do create this with
owner = newOwner
The above code is an Asset contract template. The data content (payload) of the contract instance created from this template will be what is specified in the upper “with” block.
Contracts represent legal relationships, so the payload must contain at least one (and usually more than one) ledger party. (Ledger parties are the on-ledger representations of the actors of the legal relationships, namely people of organizations.) In this case the contract represents a legal relationship between an issuer and an owner. It’s good practice to choose field names for the parties which indicate the roles of the parties in the relationship.
Such an Asset contract can be created by any party existing on the ledger. There are some restrictions though:
Anyone who submits the contract creation request to the ledger, identified as a ledger party, must specify themselves as the issuer of the Asset. This is indicated by the “signatory issuer” clause. The party id of the submitter will be stored not only in the “issuer” field of the created contract instance, but also in the “signatories” list which is created by the platform.
The Asset name must be a text, which is not empty. This is indicated by the `ensure name /= “”` clause.
The creator of the contract can specify any existing ledger party as “owner”, including themselves. The owner doesn’t have any obligations as a consequence of being mentioned in the contract (of course, as they didn’t sign anything so far). Any ledger party mentioned as owner can see the contract creation in their ledger views. (This is specified by the “observer owner” clause. Similarly to the above mentioned “signatories” filed, there is an “observers” field on contract instances, created by the platform. In this case the party specified as owner will be filled in.)
The owner party is granted a right by this contract, specified in the “choice” block. They can give it to any other ledger party, by exercising the “Give” choice, specifying a new owner. (The required input data for the choice are specified in the “with” block under the choice name.) The “do” block specifies the explicit consequences of exercising the choice: the creation of another instance from the same template, with almost the same data as the underlying contact, just changing the owner to the new owner.
I wrote above that the “do” block contains the explicit consequences of the choice exercise. This is suggests that there can be some implicit consequence of the choice exercise. And there is indeed, in the default case. If it is not indicated otherwise, a choice exercise “consumes” the underlying contract, meaning rendering it unavailable for further use. The creator of the Daml model can change this default behavior by putting the “nonconsuming” keyword in front of the “choice” keyword.
The creation and exercise events don’t store the choices associated with them on the ledger, just the payload of the contract and the input data of the choice exercise (plus some metadata).
Another short detour about how to know what rights and obligations are associated with contract instances:
You may ask: how anyone inspecting the ledger can know what rights and obligations are associated with the contract instances? The answer is that from inspecting the ledger alone they cannot. They can only know this by also inspecting and understanding the source code of the Daml model.
From which arises the next question: how can they know that a certain contract instance belongs to a certain Daml model? The answer is that every Daml model has a unique id, called package id, which is the hash of its source code + its version + its dependencies. Every contract instance on the ledger contains the package id of the Daml model it belongs to.
The edges of the graph are the “consuming” and “consequence” relationships starting at exercise events and pointing to other creation or exercise events. We can picture these relationships as arrows for ourselves. The “consuming” arrow points to previously created contracts, “consumed” by the choice exercise. The “consequence” arrow points to the events listed in the “do” block of the choice body in the source code.
The event graph grows in an append-only manner during normal operation.
When receiving a transaction request, the ledger performs some general checks and examines if all the necessary authorizations were given in advance. (It doesn’t wait for any ledger party to authorize the request — if such post-hoc authorization is needed, then the Daml model needs to contain appropriate contract templates for that).
If the request is valid and well-authorized, the corresponding event nodes and relationship edges are added to the event graph. If anything is missing, nothing gets added to the ledger. This guarantees atomic transaction composition.
The current state of the ledger (also called the active contract set, ACS for short) consists of such creation events for which no consuming exercise was recorded on the ledger.
When we speak about “active contracts” and “archived contracts”, this is an abstraction. The “archival” of the contract doesn’t change anything on the creation event of it. It just adds an “arrow” to the ledger pointing back to the previously recorded creation event, from a consuming exercise event.
You may be tempted to think that “active contracts” are on-ledger, while “archived contracts” are off-ledger objects. This couldn’t be further from the truth — until, of course, the ledger gets pruned (see later).
All this means that a Daml ledger is an event sourcing system. In other words, the current state of the system must be computed from the whole event graph, which is usually slower than just retrieve the state snapshot from a database. Speeding up this computation is a challenge. One option is to use the query store feature of the Daml JSON API. Another option can be to store the event graph in a graph database which is optimized for fast graph query (see later).
Ledger parties have access to a subset of the event graph containing the events they are entitled to see. This guarantees sub-transaction privacy.
Such subsets of the event graph are called “ledger projections”. The magic of Daml is that the consistency of projections guarantees the consistency of the whole virtual ledger. (Here I am simplifying things, because consistency is actually a property of the causality graph.)
The ledger can be pruned under certain circumstances, removing events which are not needed for operation.
This serves two purposes: 1) some kind of “garbage collection” 2) is necessary to comply with the “right to be forgotten” mandate of GDPR. See the details of ledger pruning here.
Why we usually cannot see the whole event graph
For obvious reasons, the UI of Daml applications usually only display the “active contracts”, or in the case of the Navigator UI, on demand also the “archived contracts” (and we already know how these are abstractions).
Though exercise events are the links between ledger states, the dynamic elements which tell the story of the ledger, they remain the “missing links” in this way.
I hope you are convinced by the reasoning put forward in the previous sections that the event graph is worth displaying. Let’s see how this goes.
Displaying the event graph
The Daml docs use causality graphs for explaining the Daml ledger model. Below is an example. Single arrows together with the “Exe” keyword signal “consuming” relationships, double arrows “consequence” relationships. (Nonconsuming exercises were signalled with the “ExeN” keyword.)
The direction of the arrows is not crucial but can convey a subtle message.
This is a causality graph, so the single arrow means that the creation of a contract must happen before a choice exercise on that contract.
For event graphs, I prefer the reversed direction of the “consuming” relationships. On graph representations of directed graphs, the edges are usually so represented that the set of the ending nodes is a property of the starting node. And this is how the Daml platform actually returns the event graph.
Drawing the “consuming” edge starting at the consumed contract may suggest that the creation event of the “consumed” contract is changed afterwards. As we saw earlier, this is not the case. The creation events, once recorded, are never changed because the event graph grows in an append-only way.
A similar graph representation of the Daml ledger can be seen in Daml Studio on the “Script result” pane, choosing the “Show transaction view” option. The below graph represents the ledger after three events, happening with Asset contracts explained in previous sections. The events are:
- Alice creates an Asset contract instance with the name “TV”, specifying herself as issuer and owner.
- Alice gives the TV asset to Bob.
- Bob gives the TV asset back to Alice.
The same as expressed in Daml Script:
aliceTV <- submit alice do
createCmd Asset with
issuer = alice
owner = alice
name = "TV"
bobTV <- submit alice do
exerciseCmd aliceTV Give with newOwner = bob
submit bob do
exerciseCmd bobTV Give with newOwner = aliceAfter these three events, the ledger graph looks like this:
There is a lot going on here, I won’t explain every detail just what is important for us now.
Transactions are marked with the TX keyword, a growing number and a timestamp. The number stands for the combination of a unique, randomly chosen transaction id and a growing index called “ledger offset”. We can see here three transactions, corresponding to the three events described above.
Events are marked with a compound id, consisting of the transaction id and an index as suffix. E.g. event #2:1 is the second event of the third transaction. Event descriptions also contain the actor, the “creates” or “exercises” keyword together with the template name or the choice name, and the contract payload orexercise inputs in the corresponding “with” blocks.
Transactions always have a top level action, which is directly requested by a ledger party. These are marked with zero suffix.
The list of consequences of top-level exercise actions is marked with the “children” keyword. (Create actions, naturally, don’t have consequences. The list of consequences can be empty when a choice only consumes its contract, or is only used to signal some information to a client application without changing the ledger.) The list of children in these cases is always a singleton list, because the “do” block of the “Give” choice on the “Asset” template only contains the creation of an “Asset” template with a modified content.
“Consuming” relationships are marked with the “consumed by” purple clauses. My warning regarding the direction of the “consumed” arrow previously is also valid here. Marking the creation events themselves as “consumed by” may lead to the false assumption that the creation events on the ledger are modified afterwards by the consuming exercise, which is not the case.
This graph view is nice for small ledgers and small contracts, but can quickly become unintelligible. An additional limitation of it is that it can be displayed for displaying a series of hypothetical ledger submissions expressed in Daml Script against Daml Studio’s script running service, and not for displaying the actual content of a running ledger.
What a specific party can see on the ledger?
(Please note that “the ledger” is an abstract concept. You may be tempted to ask “What is on the ledger?”. This question is not a meaningful one. You can ask instead: “What a specific party can see on the ledger?”.)
The full view of the ledger for a specific party is available through the transaction service, with the GetTransactionTrees subscription. This command returns the stream of transaction trees as a gRPC message stream.
The ledger view can be returned as a message stream because the ledger grows in a pure additive manner. The ledger view after the first transaction was the first message, the ledger view after the first two transactions was the first two messages, etc.
For prototyping, the message stream can be fetched by e.g. gRPCurl. The returned JSON representation of the first two transactions from the above mentioned three transactions in Alice’s view looks like this:
{
"transactions": [
{
"transaction_id": "1220bb3313bff96d52932e0e088e60cdbc3e4165292d89c0c76a588f1e9dc911bb09",
"command_id": "ce783485-56eb-4823-8086-2a109ff32651",
"effective_at": "2023-03-07T14:02:49.476346Z",
"offset": "000000000000000007",
"events_by_id": {
"#1220bb3313bff96d52932e0e088e60cdbc3e4165292d89c0c76a588f1e9dc911bb09:0": {
"created": {
"event_id": "#1220bb3313bff96d52932e0e088e60cdbc3e4165292d89c0c76a588f1e9dc911bb09:0",
"contract_id": "00a87fa6e78e7c059f0f192eea08bdb38cd1339103bee0116339dce93abc9caa7dca01122083a06b269dab95a5d1abd78bb35cdad628d4f0c0b8a7751c01fb56b6d734d527",
"template_id": {
"package_id": "e44c38bd3be067c6f812661268514b1570c9bd03be97fda6f6a59e714b00dde9",
"module_name": "Main",
"entity_name": "Asset"
},
"create_arguments": {
"record_id": {
"package_id": "e44c38bd3be067c6f812661268514b1570c9bd03be97fda6f6a59e714b00dde9",
"module_name": "Main",
"entity_name": "Asset"
},
"fields": [
{
"label": "issuer",
"value": {
"party": "Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67"
}
},
{
"label": "owner",
"value": {
"party": "Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67"
}
},
{
"label": "name",
"value": {
"text": "TV"
}
}
]
},
"witness_parties": [
"Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67"
],
"agreement_text": "",
"signatories": [
"Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67"
],
"metadata": {
"created_at": "2023-03-07T14:02:49.476346Z",
"driver_metadata": "CiYKJAgBEiCbJmDtJrpjE9ux5M2GS268rbcgvEt1/T6OgDm+U9VPgw=="
}
}
}
},
"root_event_ids": [
"#1220bb3313bff96d52932e0e088e60cdbc3e4165292d89c0c76a588f1e9dc911bb09:0"
]
}
]
}
{
"transactions": [
{
"transaction_id": "1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3",
"command_id": "cdbc4694-35b7-4493-8350-6cff66d43c84",
"effective_at": "2023-03-07T14:02:54.749560Z",
"offset": "000000000000000008",
"events_by_id": {
"#1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3:0": {
"exercised": {
"event_id": "#1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3:0",
"contract_id": "00a87fa6e78e7c059f0f192eea08bdb38cd1339103bee0116339dce93abc9caa7dca01122083a06b269dab95a5d1abd78bb35cdad628d4f0c0b8a7751c01fb56b6d734d527",
"template_id": {
"package_id": "e44c38bd3be067c6f812661268514b1570c9bd03be97fda6f6a59e714b00dde9",
"module_name": "Main",
"entity_name": "Asset"
},
"choice": "Give",
"choice_argument": {
"record": {
"record_id": {
"package_id": "e44c38bd3be067c6f812661268514b1570c9bd03be97fda6f6a59e714b00dde9",
"module_name": "Main",
"entity_name": "Give"
},
"fields": [
{
"label": "newOwner",
"value": {
"party": "Bob::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67"
}
}
]
}
},
"acting_parties": [
"Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67"
],
"consuming": true,
"witness_parties": [
"Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67"
],
"child_event_ids": [
"#1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3:1"
],
"exercise_result": {
"contract_id": "00027c63797ce8dce9e793ae740770334c72e499c9e7b9c7bc72b2ab7cf978c07aca01122056a1884001af49b183f1a7a3cba394e616dc68e881f17f5f76c07f68c8f4179c"
}
}
},
"#1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3:1": {
"created": {
"event_id": "#1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3:1",
"contract_id": "00027c63797ce8dce9e793ae740770334c72e499c9e7b9c7bc72b2ab7cf978c07aca01122056a1884001af49b183f1a7a3cba394e616dc68e881f17f5f76c07f68c8f4179c",
"template_id": {
"package_id": "e44c38bd3be067c6f812661268514b1570c9bd03be97fda6f6a59e714b00dde9",
"module_name": "Main",
"entity_name": "Asset"
},
"create_arguments": {
"record_id": {
"package_id": "e44c38bd3be067c6f812661268514b1570c9bd03be97fda6f6a59e714b00dde9",
"module_name": "Main",
"entity_name": "Asset"
},
"fields": [
{
"label": "issuer",
"value": {
"party": "Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67"
}
},
{
"label": "owner",
"value": {
"party": "Bob::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67"
}
},
{
"label": "name",
"value": {
"text": "TV"
}
}
]
},
"witness_parties": [
"Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67"
],
"agreement_text": "",
"signatories": [
"Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67"
],
"observers": [
"Bob::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67"
],
"metadata": {
"created_at": "2023-03-07T14:02:54.749560Z",
"driver_metadata": "CiYKJAgBEiCcT0sakwjtEG60q0pCsNfx5ZvaZxPIwRkynyzeLWsh6A=="
}
}
}
},
"root_event_ids": [
"#1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3:0"
]
}
]
}As expected, the first transaction contains one event, the creation of an Asset contract by Alice. The second transaction contains two events, 1) the consuming exercise if the “Give” choice on the previously created contract (identified by contract id), 2) the creation of a new Asset contract with modified content.
Please note again that the first “created” event is never changed after having recorded on the ledger. The fact that the corresponding contract was “archived” is recorded on the “exercised” event by setting the value of the “consuming” field to “true”.
Show, don’t tell
Now finally we have arrived at the point where we can create a nice visualization of the event graph.
After some transformation (see the source code on Google Colab) we can create from the transaction trees JSON the Cypher code to create the graph:
CREATE (Event_7_0:created {offset: 7, event_id: "7_0", template: "Asset", issuer: "Alice", owner: "Alice", name: "TV"})
CREATE (Event_8_0:exercised {offset: 8, event_id: "8_0", choice: "Give", newOwner: "Bob"})
CREATE (Event_8_1:created {offset: 8, event_id: "8_1", template: "Asset", issuer: "Alice", owner: "Bob", name: "TV"})
CREATE (Event_9_0:exercised {offset: 9, event_id: "9_0", choice: "Give", newOwner: "Alice"})
CREATE (Event_9_1:created {offset: 9, event_id: "9_1", template: "Asset", issuer: "Alice", owner: "Alice", name: "TV"})
CREATE (Event_8_0)-[:CONSEQUENCE]->(Event_8_1)
CREATE (Event_9_0)-[:CONSEQUENCE]->(Event_9_1)
CREATE (Event_8_0)-[:CONSUMING]->(Event_7_0)
CREATE (Event_9_0)-[:CONSUMING]->(Event_8_1)Which, in the Neo4j graph database creates the following graph you’ve already seen in the introduction:
