Photo by AJ Garcia on Unsplash

Solving Severe Asynchronicity with Gnarly

Optimistic UI for blockchain interfaces.

Gnarly reduces blockchain events into a steady state with confidence.

And that’s fuckin’ gnarly.

What is Severe Asynchronicity?

Severe Asynchronicity is the experience of using a first-layer blockchain today:

• transactions publish within a reasonable timeframe (ms) but at very low confidence—it’s hard to know if and when they will succeed
• transactions are finalized within an unreasonable timeframe (minutes/hours) but with very high confidence
• Off-chain state is uncertain due to [1], [2], block re-orgs, short-lived forks, uncles, etc,
• Off-chain software isn’t perfect; it can lag behind the blockchain (if waiting for confirmation blocks), fail to replay state updates in the event of reorgs/forks, improperly handle unconfirmed transactions, and much, much more.

Users have no insight into what’s happening to a transaction and how their future actions are affected—and if they do have insight, it’s within an unfriendly loading screen that doesn’t indicate the confidence that a transaction will succeed.

Waiting on transactions is a huge sticking point for the user experience. I can’t move forward and perform other tasks if I’m not confident that the first thing I did will succeed.

Giving the user confidence in their actions is necessary, especially for high-performance applications and games.

Features

• Enables declarative, reactive clients,
• “Instant” updates with confidence intervals—Optimistic UI,
• The optimistic ui pattern—apply expected changes immediately but revert to source of truth as soon as it’s known, reducing blockchain events into a steady state—the client need only know what the state is right this second with appropriate confidence in order to do its business,
• Reducing blockchain events to a steady state optimizes client queries and allows interfaces to design better data stores,
• Graceful block reorganization and invalid state handling,
• Friendly error management means (i) developers get reasonable error contexts and (ii) consumers get explanations about errors,
• Friendly error management allows anyone to know (i) that something occurred and (ii) why it occurred and (iii) how it affects their actions,
• supports replay from arbitrary blocks to (i) bootstrap the steady state and (ii) resume after failures,
• default output is catered towards a graphql consuming client

Gnarly is available in server-side environments to provide optimistic shared state and is available client-side for optimistic personal state (the fully trustless approach).

Key Ideas

You can think of it as a “server-side” apollo-client where the blockchain you’re using as source-of-truth is the apollo-server.

Gnarly uses the ideas behind redux and MobX to convert imperative blockchain events to declarative, reactive state.

Transactions and Blocks

Every write to the blockchain is dependent on (i) the transaction being included in a block and (ii) that block being various states of valid.

The system keeps an entire log of everything that occurs to the transactions and blocks we care about, including which state is derived from them, allowing it to cascade changes throughout the database in the event that the optimistic view is incorrect.

For example, if a transaction adds a number to a counter, the optimistic view is that it will succeed and the counter should be equal to prev + 1 . The counter’s state is now dependent on that tx, which is dependent on its block, either of which can be determined to be invalid at any point in the future. These things that have dependencies on their validity are called artifacts. In the event a dependency is invalidated, all artifacts of that dependency must be invalidated as well — and the user notified of this change.

Artifacts

Artifacts are state (specifically, state changes) that depend on some variable-validity artifact before it. A tx is an artifact of a block. A block is an artifact of the previous block. A counter’s state being prev+1 is an artifact of the tx that caused that state change.

All artifacts:

• are universally addressable by a UUID
• have 0 or more dependent artifacts upon which their validity depends
• are never deleted; only validated (with confidence intervals) or completely invalidated—this allows stale clients to request information for any artifact they know about, regardless of its validity
• have an event_log associated with them, providing context and cause

This architecture allows a client to never be “unknowingly wrong” about the state of the world. That is, if it thinks that the state is 6 due to artifact abcd, but artifact abcd had been invalidated without the client’s knowledge (due to a disconnect, for example), future requests for information about artifact abcd (and artifacts that depend on abcd) still contain the context that (i) it was invalidated and (ii) it was invalidated because it was in an uncle block (for example).

An artifact’s confidence interval—how likely it is valid—is a function of all of its dependencies (namely, blocks). As the blockchain grows, later blocks are considered more likely to be finalized (e.g. 100% validity), which affects the validity of all blocks after it (and the transactions within those blocks and the artifacts that result as a state change cause by that transaction).

The artifacts and the dependency chain is represented as a directed graph. Confidence in an artifact’s validity cascades through the directed graph to affect the artifact you’re interested in. This model of confidence can also be used to inform gas prices; how likely do you want your transaction to succeed within the next 5, 10, or 15 minutes?

Filter—Map—Reduce

You can think of gnarly as server-side MobX, persisted either to client-side memory or server-side database.

The core functionality is a filter, map, and reduction over all blockchain actions. An “action” is literally anything you want to describe; perhaps a transactions to your contract produces an action. Perhaps every single transaction ever created produces an action. Perhaps an Event() fire produces an action. It’s arbitrary, much in the same way that redux actions can describe anything; all that matters is firing them appropriately.

The system looks primarily like:

# filter, flat map across blocks to produce actions
(block) => [action]
# produce state modifications—just like MobX
(action) => [...direct modifications...]

GraphQL

The resulting state is persisted into a datastore of your choice. Perhaps CouchDB for realtime capabilities.

This datastore is then fed to the client by a GraphQL server (apollo) supporting subscriptions that enable highly reactive UI.

Optimistic Transactions

Without optimistic transactions, this steady state would stay in lock-step with the blockchain (or at least, your node’s view of the blockchain). That is, if a tx takes 10 minutes to get into a block, your persistent state has no knowledge of it until it has been included.

To enable truly responsive blockchain interfaces, we can apply learnings from web development—optimistic UI.

Upon every transaction submitted to the network, the client also implies its expected state transitions to gnarly. Each request is:

1. The transaction details and the signature against it, which allows gnarly to calculate the transaction hash and observe it across its lifetime, as well as confirm the signer’s address, and

2. The application-specific action(s) that will be produced by this transaction, which allows us to quickly propagate the expected state to the store.

So the client says something like

buyKitten(txData, signedTxHash)

from which the backend can get the address from the signature, do any validation of the transaction—like reversing the abi to confirm the function call and arguments—, and produce the correct action(s) corresponding to this event.

These actions are then presented to the system as a new, potential block with the appropriate confidence.

When using gnarly in a server-side environment, sybil defenses are necessary to avoid greifing the shared state; minimizing attack surfaces with rate limiting and authentication schemes will be necessary. For example, only users with an ID associated with your project (perhaps an ENS name that resolves to their address) should be able to write to your optimistic shared state.

System Components

Actions

Actions follow the flux-standard-action (FSA) pattern, with some additional requirements. All actions must have a _REASON key that corresponds to a platform-specific, but well-known reason. AKA, a key that you can look up to communicate to a user why this action was created.

{
  type: ‘YOUR_ACTION_TYPE’,
  payload: {
    // any data necessary for
    //  reduction later down the road
  },
  _REASON: ‘MY_REASON’,
  meta: {
    // any metadata necessary for
    //   communicating context about the action
  }
}

When referencing an artifact later down the line, it’s easy to see the list of things that affect it along with why those things occurred.

Action Reducers

We still need to turn these actions into some sort of state. We can use the MobX model of reducers to produce these effects. All state modifications are wrapped in a block-scoped transaction.

Of note, the state view provided to the reducer is time-dependent; it provides a snapshot of the state at the given block, not as it is at the present moment, important for stateless replay.

Almost every state transition produced by this reducer that affects an artifact should come with a _REASON to be applied to the event log.

const counters = (state, action) => {
  // whatever this looks like for the db adaptor
  const artifactId = state.counters.add(1)
  // produce an event log using action._REASON, action.meta
  state.events.produce(artifactId, action)
}

Persistence and State Updates

All state transitions that are produced by the reducers are then executed within a single transaction to apply them to the store.

Every change to an artifact can be tied to a block number (and then invalidated in the case of a rollback).

Loading Blockchain Data

Gnarly needs to know about everything that could possibly happen to a blockchain so that it can produce actions about them.

One option is to require gnarly to ingest the full blockchain, requiring a high-bandwidth connection to a full node, as well as the local processing and memory required to compile the steady gnarly state. This works well for shared, server-side environments where computation is freely available, and is generally performed only once for multiple users. This implies trust, however, and is generally not reasonable for client-side (web, mobile) environments.

To provide for client-side—but trustless—usage of gnarly, iterative snapshots of gnarly state can be computed using Truebit and stored on IPFS. Then a client must only load the latest finalized state from IPFS and then “fast-forward” to current-time.

I love how this can help make ‘live use’ of a dApp way better. But also very very important too is computing the original state without every user having to traverse a huge event chain locally.

Truebit could be used for this. Having an off-chain state reducing function that takes the state and logs of a contract and returns a JSON file with the processed state that can then be published to IPFS. The hash could be stored on chain and there could be a verification game to claim the state has been computed incorrectly.

You can do this sort of snapshots every finalized block, to be sure it is unlikely the chain will reorg. — Jorge Izquierdo

@TODO

There are a ton of unanswered questions.

Ethereum transactions are nonce-based and require that the next valid transaction must be the next available nonce. Hence, if a single transaction from an account fails, any transactions that use the next nonce are no longer valid. This makes it more difficult to “queue” transactions. There may be native solutions to this in the future, but until then, UX is the only option.

In the event of a rollback (for whatever reason), it should be a relatively straightforward process to:

• find all artifacts of a block
• find all artifacts of the txs of a block
• etc etc all the way down
• invalidate them
• roll back any state nicely so that the steady state is again accurate (and make sure confidence intervals are re-applied correctly)

🚧 Want to build Gnarly?

Gnarly is 100% owned by the community. It’s designed to be blockchain agnostic (although starting with Ethereum) and enable Very Good UX™ for the vast majority of blockchain interfaces. If your app is a user interface that talks to the blockchain, you need gnarly.

1. Join the #gnarly channel on XLNT.chat 🚀 and contribute ideas and discussion.
2. Join us on the github repo at XLNT/gnarly.

Thanks to Paul Fletcher-Hill, Feridun Mert Celebi, and Jorge Izquierdo for collaborating on the ideas behind gnarly. ❤️