The potential of decentralized applications (Dapps) to bring disruptive innovations to many sectors, from banking (DeFi) to gaming, is widely recognized. However, even the most innovative solutions will struggle to gain acceptance if they can’t meet the expectations of consumers.
Consumers expect smooth and sophisticated user experiences, and the developers of Dapps on Ethereum have found this to be a major challenge.
In this article, I’ll give an overview of the classic Dapp architecture, and point out several inherent limitations in the standard Ethereum stack today that make it challenging for developers to create compelling user experiences.
I’ll then describe some new innovations in Ethereum infrastructure that can help developers overcome these challenges, such as dfuse. At dfuse, we are excited to enable a better developer experience, to accelerate mainstream adoption of Ethereum applications.
The Classic Ethereum Dapp Architecture
(Experienced Ethereum developers: feel free to skip to the next section!)
A Dapp on Ethereum has traditionally consisted of three main components:
- A smart contract, most often written in Solidity, often built using a framework such as Truffle Suite, and deployed on the Ethereum blockchain.
- Front end code, written in Javascript.
- A back end — traditionally, a standard Ethereum blockchain node. The front end communicates with the back end using either the JSON-RPC or the GraphQL API provided by the node.
Various libraries are available to facilitate the front end’s communications with the Eth node, the most popular being web3.js and ethers.js. There are also web3 libraries for many other languages (Java, Python, Rust…).
Private Back End Nodes
In the early days of Ethereum, it was necessary for developers to operate their own Ethereum nodes for development purposes. They also had to operate production-quality nodes (or clusters of nodes) when they released their Dapps. Operating a blockchain node was a significant effort, and reduced developers’ productivity.
Nodes as a Service
These challenges led to the rise of “nodes as a service” platforms from several companies, such as Infura, and more recently Nodesmith, Quiknode, Blockdaemon, Ethernode, Chainstack, Alchemy, CloudFlare, and others.
These platforms provision cloud-based Ethereum nodes for developers, sparing developers the effort of operating a node. Solutions are available both for development and for production. These platforms let developers offload system administration tasks such as patches and updates, both of the underlying operating system and of the node software itself.
Inherent Limitations of Ethereum Nodes
However, even though they succeed in sparing developers from sysadmin duties, nodes as a service do nothing to help developers build better Dapps with better user experiences. This is because these challenges arise from inherent limitations of both the nodes as a service architecture and the JSON-RPC and GraphQL interfaces supported by Ethereum nodes.
The main limitations include the following:
Inconsistent Views of State
In order to scale beyond the capacity of a single node, and provide higher reliability, nodes as a service platforms provide access to a pool of nodes through a load balancer.
Since each of these nodes operates autonomously as a peer in the Ethereum network, as information propagates through the network at a given moment different nodes may be at different block heights, or even on different forks. This means that a Dapp can receive inconsistent information about the blockchain state as its requests are served by different nodes behind the load balancer.
Nodes as a service platforms often attempt to solve this problem by means of session stickiness on the load balancer, which endeavors to always send a given front end’s queries to the same back end node — but this method fails in multiple cases:
- when a front end generates more requests than a single back end node can handle;
- when network issues cause a front end to disconnected from the back end, and it has to reconnect;
- and because many node as a service platforms route different types of front end requests (e.g. sending a transaction vs. searching the chain history) to different groups of back end nodes optimized for that query type.
As a result of the fact that the front end often accesses several backend nodes, which may not have views of the blockchain state consistent with each other, it is difficult for Dapps to deal with chain reorganizations. When following the chain backwards, a Dapp may suddenly find that the parent block it is expecting doesn’t exist (because it is now talking to a different node that is on a different fork). Then the burden is on the Dapp developer to write code that can cope with this (generally by reconnecting repeatedly until it finds a node on the right fork). This adds needless complexity to the Dapp, and can result in inconsistent information being presented to the user.
Searching the Blockchain is Slow and Limited
Dapps are limited in their ability to search for transactions, or other past events on the chain, because standard Ethereum nodes are not well suited to support accurate searching, or to perform specific searches of streaming realtime data. Doing so in a performant way requires rich indexes over many millions of blocks and transactions, but:
- Ethereum nodes index only certain fields from the Logs emitted by transaction execution (the fields to be indexed have to be identified by the developer at the time the contract is deployed)
- Ethereum nodes do not index internal transactions (which occur when a smart contract invokes a method on another contract) data at all
- Developers are reluctant to add extra indexed fields, because indexing costs the users of the contract extra gas — the cost of every transaction increases for each field that is indexed
- Ethereum nodes perform searches using Bloom filters, so searches are always fuzzy, and can yield false-positive matches. Exact matching requires extra work by the front end, which has to retrieve the whole block or transaction containing a fuzzy match and double-check for exact matches. This not only requires developer effort but also wastes bandwidth between your front-end and the node
- The search syntax available is very limited — it supports only basic selection plus simple alternation.
- Getting search results is slow — it can take hours to search over a large range of blocks.
- JSON-RPC is very wasteful of bandwidth — it returns much more data than you need. The GraphQL interface uses less bandwidth but does not provide streaming (your front end must poll for updates)
Lack of Atomicity
Transactions are expected to be an atomic operation in most modern environments, like relational databases, but are not atomic on Ethereum (or other blockchains). Each transaction transitions through a sequence of many states, and can get stuck or fail in numerous ways. Dapps have to make many API calls, querying many different data sources (blocks, the mempool, network conditions) in order to follow the lifecycle of a transaction through to completion.
Again, the burden falls on front-end code to poll repeatedly and figure out what’s happening, and users of the Dapp experience delays and refreshes as the Dapp performs all this extra work.
Nodes are Passive
Ethereum nodes are passive, in the sense that they can’t generate events or callbacks or invoke webhooks. All actions must be initiated by the front end, which has to poll the node to obtain updated information. The event streaming capability of Ethereum nodes is too limited to meet the needs of most dapps, and is available only in JSON-RPC interface, not when using GraphQL (see here)
Rethinking Dapp Infrastructure With dfuse
dfuse is a platform that provides higher-level blockchain APIs, which do more, with less effort, than the native APIs provided by blockchain nodes. dfuse has been designed from the ground up to empower Dapp developers with the capabilities they need to build modern blockchain applications with fast, fluid interfaces that deliver exceptional user experiences.
dfuse aims to overcome the limitations of traditional Ethereum nodes by addressing all of the above limitations.
A Single Consistent View
dfuse is an integrated hyperscale data platform, not a collection of Ethereum nodes clustered behind a load balancer. The dfuse Platform provides a single consistent view of the chain state, across all connections and at all points in time. The dfuse Platform either sees a block (and is then always able to automatically navigate chain reorganizations for you), or it never reports it at all (in the case of blocks that are reorged out quickly and don’t propagate very far).
Dapps never face inconsistent views of the chain state, and can focus on their primary functions rather than the details of blockchain bookkeeping.
Fast, Granular Searching
dfuse empowers Dapp developers to search through the history of the chain with great granularity, with extraordinary speed and efficiency, and with streaming realtime results over GraphQL, gRPC, and a websocket interface.
- dfuse fully indexes all Log fields — all the data emitted in Logs by each transaction is automatically available for highly granular searching.
- dfuse fully indexes all internal transactions (sender, recipient, value, method, input parameters), enabling complete tracking of operations of your contract throughout the call tree
- Indexing does not cost your users any extra gas — dfuse’s indexing is an integrated feature of the dfuse Platform, and does not increase the resource cost of contract execution
- Searching yields exact matches, rather than fuzzy results. There is no need to write additional front end code to double-check search results, and no bandwidth wasted by bulk data retrievals to support such checking
- dfuse provides a structured query language, similar to Kibana’s or GitHub’s, with full boolean operations and the ability to zero in on the exact transactions or calls you want to see
- dfuse provides exceptional performance — you can search the whole chain history for a set of exact matches specified by your search expression in under a second
- dfuse supports the best of both worlds, with GraphQL for succinct responses, but without sacrificing streaming capabilities — our GraphQL interfaces provide full streaming search, enabling efficient dynamic updates to your users
- The performance of dfuse is constant regardless of traffic levels on the Ethereum network
Transaction states
dfuse provides a single streaming endpoint that understands all the intricacies of the states a transaction can get into, and that notifies you when your finality criteria are met. There’s no need to laboriously follow a transaction’s state by repeated polling or to check multiple data sources — you can simply push a transaction and stay connected to receive streaming updates, enabling you to provide realtime transaction status displays to your users.
Active Back Ends
The dfuse Platform gives you an active back end that can initiate events. For example, dfuse can call a lambda function (or cloud function) of your choice, on precise criteria you specify (with all the power of the search features described above). This enables asynchronous styles of Dapp architecture, where updates can be posted to users across multiple communications channels, fluidly and in real-time.
A Modern Platform for Cutting-Edge Dapps
dfuse provides you with a modern infrastructure layer for your Dapps that is:
- Fast,
- Scalable,
- Gives you highly granular, streaming access to blockchain events,
- Supports active webhook-style callbacks,
- all with the highest reliability in the industry.
Try dfuse today. Reach out to us on twitter or email with any questions/suggestions or to talk about your Ethereum dapp building experience — we’d love to know if you were delighted by the service.
