The Cardstack Ethereum Plugin: Technical Overview

Exposing your smart contract as a RESTful service

I’m really excited to write about one of the new features of the Cardstack Hub that we unveiled during our MVP: The Cardstack Ethereum Plugin. For those new to my blog, I’m an Ethereum developer at Cardstack, and I routinely post about fun challenges we are solving.

At Cardstack we are building an application framework for decentralized applications (dApps) that puts user experience first. One of the foundational aspects of our framework is our ability to expose on-chain information to dApps in an easy-to-consume manner that allows developers to query the blockchain without any special Ethereum libraries (like web3.js).

The Cardstack Hub provides a “data fusion” layer for decentralized applications built using Cardstack. Using the Hub, each Cardstack application can use off-the-shelf plugins in order to ingest data from some particular data source — be it a cloud service or a blockchain—into the Cardstack Hub’s index.

Examples of plugins for different data sources that we have built so far are:

• PostgreSQL plugin that can ingest content from a PostgreSQL database,
• Git plugin that can ingest content from a Git repository (e.g. GitHub)
• Drupal plugin that can ingest content from a Drupal content management system
• Ethereum plugin that can ingest content from smart contracts (including Oracles) running on Ethereum

Additionally, we have a development kit that can be used to create new data source plugins for ingesting content from other sources.

Today, let’s take a deeper dive into the Ethereum plugin to see how it works.

How Does a Cardstack Plugin Work?

To understand how the Cardstack Ethereum Plugin works, first we need to take a step back and understand a bit more how any Cardstack Plugin works.

When the Cardstack Hub is booted, the very first thing it will do is to start crawling the directory structure of the application to discover the various plugins that are installed and the different features available for the plugin. The most basic feature that any data source plugin must have is an indexing feature.

The indexing feature’s main job is to ingest content from a data source and place the content in the Cardstack index. To accomplish this, the indexing feature describes to the Cardstack Hub the schema of the data source: what are the different entities contained by the data source, what are the attributes of the various entities, and how are those entities related to one another.

Additionally, the indexing feature contains logic that knows how to ingest the content held by the data source, and to ingest the content in such a way that we are not doing wasted work, so that if we have previously ingested the content, we won’t waste time trying to re-ingest the information that we already know about.

The information that we ingest into the Cardstack Hub is placed in the Cardstack Hub’s index. The index is a fast means to query the content based on the schema of the content that was indexed. This means that once the content is indexed we can now search on the data and explore its relationships in a manner that is more sophisticated and performant than if an application were to talk to the original data source directly.

The index can then be exposed to the outside world via our RESTful API which utilizes a system of permissions and grants to ensure that users and groups are only authorized to view and manipulate resources for which they have been granted access.

Plugins can have other features too. A writer feature allows the Cardstack Hub to write information back to the data source. A searcher feature allows the Cardstack Hub to perform a search against the backing data source. A middleware feature allows the plugin to register API endpoints to perform specialized tasks for the plugin. An authenticator features allows the Cardstack Hub to authenticate sessions. (We’ll continue to explore these features in upcoming blog posts.)

The Ethereum Plugin

The Cardstack plugin that we will be focusing on is the Ethereum Plugin (check it out on Github).

In a nutshell, the Ethereum plugin makes it so that:

1. The Cardstack Hub can introspect your smart contract to derive a schema.
2. The Cardstack Hub can ingest content from your smart contract as it emits Ethereum events.
3. Web clients can query your smart contract using simple RESTful web requests without any specialized Ethereum libraries.

Alright, let’s dive in.

The Cardstack Ethereum plugin handles Ethereum-based smart contracts data sources, including oracles. It is the responsibility of the Cardstack Ethereum Plugin to ingest the state of specified smart contract(s), and reflect that within the Cardstack index.

The Cardstack Ethereum plugin currently supports the indexing feature. In the near future we intend to create a writer feature for the Ethereum plugin, but in the meantime the Cardstack Hub only performs reads from the blockchain.

Schema

When an application declares that it wishes to use the Cardstack Ethereum plugin, it provides the Cardstack Hub with a configuration file that describes, among other things, the ABI of the smart contract to index. The ABI of a smart contract is basically Ethereum’s way of describing the schema of a smart contract. Within the ABI, you can find all the events that the smart contract will emit, as well as all the different methods that can be invoked by a smart contract.

The Cardstack Ethereum plugin decomposes the provided smart contract’s ABI into Cardstack’s own schema representation. The entities that are created include:

• An entity that represents the contract itself, whose attributes include a property for each read-only function that accepts no parameters.
• A one-to-many relationship between a contract entity and child entity for each read-only function whose parameters accept a single address type. In Solidity these are commonly referred to as mapping and are used to represent ledgers.
• The child entity for a contract’s mapping relationship will contain attributes for each of the return values of the read-only function described above (in Solidity a function can have multiple return values).
• Coming soon: An entity that represents each type of event that a contract can emit whose attributes are the event arguments.

Right now we only create child entities from read-only functions that have a single address parameter (as it is very well suited to ledgers), but this can certainly be extended in the future to read-only functions with different signatures.

Within Cardstack we define the schema of an entity as a content-type. We use a very simple naming convention to define the name of each content type and the content-type’s attributes.

Let’s take a look at the unit tests that we use for the Cardstack Hub’s Ethereum plugin as an example. (For those that are curious, we integrate Truffle into our testing framework to be able to fully test Ethereum contracts in our tests.)

In our unit tests we leverage a simple ERC20 contract that has a few extra methods to make things interesting. (You can find the Solidity code for the contract we use for testing here.) See how we assert the schema is built correctly from our unit tests:

Reference 1: Ethereum data-source configuration test-setup. GitHub.

The code snippet in Reference 1 above is the test setup for the tests where we make assertions for the correct behavior of the Cardstack Ethereum plugin, specifically you can see the data-source configuration on lines 15–33.

Let’s focus on line 15 for now. On line 15 we are creating a resource that describes the Ethereum data source that we want to ingest into the Cardstack Hub. Specifically, we name this data source “sample-token”. The data-source “sample-token” refers to the smart contract that was deployed to the blockchain (in this case the Truffle private blockchain) on line 12, which is the smart contract I referred to above.

Reference 2: Smart contract schema assertions. GitHub.

In the code snippet of Reference 2 above we are showing how the “sample-token” smart contract is converted into the Cardstack Hub schema’s content-type. On line 4, you can see that the name of the content-type for the “sample-token” smart contract is just the plural inflection of the name of the data source, which is sample-tokens.

The attributes for the sample-tokens content-type are the read-only functions of the smart contract that take no parameters. The naming convention here is the dasherized name of the smart contract function prefixed with the name of the smart contract (attribute schema is shared across all content types, so we must namespace our attribute names). Since sample-tokens is an ERC20 contract the typical ERC20 functions will be included in the attributes. For instance, the ERC20 function name() appears as the attribute sample-token-name (line 26). Same for the ERC20 function symbol(), which appears as the attribute sample-token-symbol (line 42). We have a bunch of non-ERC20 functions in this smart contract as well, like tokenFrozen(), which returns a boolean indicating if the token is frozen. This attribute appears in the schema as sample-token-token-frozen (line 49).

Additionally, for smart contracts we also add 2 extra attributes:

1. ethereum-address which is the Ethereum address that the contract is deployed to
2. balance-wei which is the amount of wei held by the contract’s address

Reference 3: Ledger entry schema assertions. GitHub.

In the code snippet of Reference 3 above, we are showing the content-type of the sample-token’s ERC20 functionbalanceOf(). Because balanceOf() is a read-only function that takes a single parameter of type address, we decompose it into its own content-type. This content-type’s name appears on line 5, and is the plural inflection of the dasherized method name and is prefixed by the contract name: sample-token-balance-ofs (which, admittedly, is a little awkward in this case). The idea is that there will exist a sample-token-balance-ofs record for each ledger entry in the sample-token contract, that will have a relationship to the original contract (line 15).

The attributes for the sample-token-balance-ofs are based on the return values of the sample-token contract’s balanceOf() function. In this case, there is only a single return value for an address’s token balance which is a uint256. The Cardstack Hub interprets that particular Solidity type as a number, and adds a mapping-number-value attribute to this type which holds the token balance for a ledger entry of the sample-token contract. In the interest of conserving attribute field names, all the smart contract child content-types leverage a mapping-*-value field where * is the type of the mapping.

So in the case of the sample-token’s Solidity function:

mapping(address => bool) public approvedBuyer;

The sample-token-approved-buyers content-type will possess a mapping-boolean-value field to represent the boolean value for each address that is an approved buyer.

Additionally, each child content-type, as with the contract content-type, possesses an ethereum-address attribute field that represents the address of the ledger entry of the child content-type.

Reference 4: Ledger entry schema assertions for functions with multiple return values. GitHub

For functions that return multiple named values, the attributes of the content-type for the function is the dasherized name of the function followed by the dasherized name of the return value. So for example, the content-type schema for sample-token-vesting-scehdule of the Solidity function vestingSchedule() in sample-token (GitHub) would possess the attributes asserted in reference 4 above.

Indexing

The Cardstack Ethereum plugin additionally defines rules around what aspects of the smart contract are indexed as well as when the smart contract is indexed. These rules are encapsulated in the Ethereum data source configuration. You can see the code snippet of Reference 1 on lines 25–29.

In the current configuration, the Cardstack Hub will index all data sources every 10 minutes. Additionally, the Cardstack Hub will index the Ethereum data-source every time an Ethereum event is emitted from the configured smart-contract.

For the periodic 10 minute indexing, each data source can decide the content that is eligible to index. The Cardstack Hub provides each data source with essentially a notepad that the data source can jot down notes to help it to decide what to index when the next 10 minute indexing cycle begins.

For the Ethereum data source, we keep track of the block height for each indexing cycle. When the 10 minute indexing cycle occurs, the Ethereum data source indexes all the activity that has occurred against the smart contract since the last block height indexing happened.

For Ethereum events that are emitted from the smart contract configured as the data source, we use the data-source configuration to tell the Cardstack Hub which Ethereum events should trigger content indexing.

Consider the data source configuration for the “sample-token” contract in our unit tests:

Reference 5: “sample-token” data-source configuration from unit tests. GitHub

On lines 17–20 of Reference 5 you can see the Ethereum events that have been configured as events that that trigger indexing of the contract.

Let’s take a deeper dive into line 17 of Reference 5 above and break down how this works:

Transfer: [ "sample-token-balance-ofs" ]

The property name Transfer is the name of the Ethereum event that triggers indexing. The Transfer event is defined as part of the ERC20 specification and is emitted whenever a token changes hands. So line 17 above means that whenever the “sample-token” contract emits a Transfer event, indexing will occur. The array ["sample-token-balance-ofs"] refers to the content-type that will be indexed — in this case, the content-type sample-token-balance-ofs, which corresponds to the balanceOf() ERC20 function of the “sample-token” contract.

When the Transfer event is emitted, the Cardstack Hub will parse the event arguments that are of an address type from the Transfer event, and then invoke the balanceOf() function for each address in the emitted event. The resulting balanceOf() responses are then ingested into the Cardstack Hub as new/updated sample-token-balance-ofs content.

In this manner we indicate to the Cardstack Hub which Ethereum events are of interest to the indexer, which functions should be called against the contract, and what content should be ingested into the Cardstack index.

Reference 6: Ethereum event triggers indexing. GitHub

In reference 6 above on line 17, an Ethereum transfer() function is invoked against the contract and it triggers the Transfer Ethereum event. Subsequently, the emitted Transfer Ethereum event causes the Cardstack Hub to index both the sender and the recipient sample-token-balance-ofs documents based on the Transfer Ethereum event arguments.

Reference 7: Ethereum event triggers underlying contract indexing. GitHub

In addition to updating the specified content-type, sample-token-balance-ofs for the Transfer event, the Cardstack Hub also updates the sample-tokens content-type for the underlying smart contract, as the events that are fired may actually trigger changes to the other attributes upon the underlying smart contract. In which case, the Cardstack Hub will invoke all the read-only functions that do not accept a parameter upon the underlying contract in order to update the sample-tokens content. This is demonstrated in reference 7 above.

In the data-source configuration on line 18 in reference 7, we have configured the Cardstack Hub to index when the MintingFinished event is emitted from the “sample-token” contract. On line 38 in reference 7, the invocation token.finishMinting() emits the MintingFinished event.

Finally, on line 42 in reference 7 we assert that the underlying contract, whose content is a sample-tokens type, has updated its sample-token-minting-finished attribute.

Smart Contract API

Now that we have covered how content is ingested into the Cardstack Hub via indexing, let’s cover how we can access content from the Cardstack Hub. The Cardstack Hub uses a RESTful API based on the JSON:API spec that web clients can use to interact with the Cardstack Hub. This allows web clients to perform find, create, update, and delete operations against the content within the Cardstack Hub.

Note: we’d love to include support for GraphQL too! If you’re interested in helping out, we’re very appreciative of pull requests.

Along with this, Cardstack Hub leverages a sophisticated system of grants and permissions to ensure that clients of the API can only view and modify content for which they have been granted access — stay tuned for a future blog post on this! The Ethereum blockchain is inherently globally readable. Accordingly, the Cardstack Hub’s Ethereum plugin applies a global read-only grant to all the content that is indexed from the Cardstack Ethereum plugin.

At this time we do not have a writer feature for the Cardstack Ethereum plugin, so there are no grants within the Cardstack Hub that bestow the ability to update or create new content (nor delete content) from content-types that originate from the Cardstack Ethereum data-source through the RESTful API. This will change in the future when the Cardstack Hub supports writing to the blockchain from the API.

JSON:API

The Cardstack Hub exposes its underlying content via JSON:API. JSON:API is a very powerful specification for building API’s.

From the JSON:API website:

“JSON API is a specification for how a client should request that resources be fetched or modified, and how a server should respond to those requests.

“JSON API is designed to minimize both the number of requests and the amount of data transmitted between clients and servers. This efficiency is achieved without compromising readability, flexibility, or discoverability.”

Let’s take a look at some example queries for Ethereum content made from our CARD token dashboard on cardstack.com.

On cardstack.com, we have defined an Ethereum data-source with the content-type of card-tokens. Our current CARD token smart contract is deployed to the address: 0xB07ec2c28834B889b1CE527Ca0F19364cD38935c. To get the document that represents the CARD token smart contract, a web client issues the request that specifies the content-type of the document and the ID of the document (in this case the ID is just the address of the smart contract) : GET https://hub.cardstack.com/api/card-tokens/0xB07ec2c28834B889b1CE527Ca0F19364cD38935c.

The response looks like this:

Reference 8: CARD token API response of https://hub.cardstack.com/api/card-tokens/0xB07ec2c28834B889b1CE527Ca0F19364cD38935c

The data structure should look similar to the concepts introduced when we discussed the schema. Within the attributes property of the response you will find the output of all the read-only functions of the smart contract that take no parameters. Within the meta property of the response is the blockheight field, which provides context for all the attributes.

Let’s now look at a specific token holder’s balance. As with the unit test configuration in reference 5 line 17, the configuration for the CARD token smart contract used on cardstack.com also includes this configuration in the eventContentTriggers:

Transfer: [ "card-token-balance-ofs" ]

This means that whenever the CARD token contract emits a Transfer Ethereum event, the Cardstack Hub will index both the sender’s and receiver’s card-token-balance-ofs documents.

From etherscan.io, we can choose an arbitrary token holder and supply this to the API. Let’s use the token holder address 0x274f3c32c90517975e29dfc209a23f315c1e5fc7. In this case we want the output of the balanceOf() function for the token holder’s address. To get this we can issue the request: GET https://hub.cardstack.com/api/card-token-balance-ofs/0x274f3c32c90517975e29dfc209a23f315c1e5fc7.

The response looks like this:

Reference 9: CARD token holder API response of https://hub.cardstack.com/api/card-token-balance-ofs/0x274f3c32c90517975e29dfc209a23f315c1e5fc7

As we have discussed in the schema section, the attribute that holds the value for the output of the function balanceOf(0x274F3c32C90517975e29Dfc209a23f315c1e5Fc7) would be the mapping-number-value (line 7 of reference 9) as this function has a single return value that is a uint256which is interpreted as a number by the Cardstack Hub.

Note that a current limitation is that you must use a lower-cased token address string for the ID of a child document of a smart contract when issuing the API request for the document. We use the ethereum-address attribute (line 6 of reference 9) to preserve the EIP-55 case sensitive form of the Ethereum address for the token holder.

The Cardstack Hub also supports much more sophisticated queries of documents. For example, if you wanted to get all the vested CARD token grants that that expire in 366 days you could issue a query that looks like this: GET https://hub.cardstack.com/api/card-token-vesting-schedules?filter[vesting-schedule-duration-sec]=31622400 which would return the response:

Reference 10: Querying the Cardstack Hub API https://hub.cardstack.com/api/card-token-vesting-schedules?filter[vesting-schedule-duration-sec]=31622400

We’ll expand on this in future blog posts by explaining:

• How to query using the Cardstack API
• How to create server-side computed properties to supercharge your ability to introspect data-sources
• How to correlate disparate data-sources that are used by the Cardstack Hub in an up-coming blog post.

Putting It All Together

Whew! That was a lot of information, and we’re still just scratching the surface of what is possible with the Cardstack Hub.

The takeaway is: building an easy to consume interface for your Ethereum smart contracts has never been easier.

With the Cardstack Hub introspecting your smart contract and ingesting Ethereum events as they occur, web clients can query your smart contract using simple RESTful web requests without any specialized Ethereum libraries. That’s a game changer for the development of decentralized web applications.

We have so much more that we want to show you! Stay tuned for our next blog post where we’ll walk through the process of configuring and running a Cardstack application for your Ethereum smart contract.

Read More

• Growing a Healthy Software Ecosystem by Lead Developer Ed Faulkner
• Building a Token Vesting Contract by Senior Blockchain Developer Hassan Abdel-Rahman
• How the Cardstack Framework Powers Decentralized Applications
• The Cardstack White Paper

Get Involved

Join the discussion about Cardstack on our official Telegram channel: https://t.me/cardstack.

Are you a developer? Join our Slack channel: https://join.cardstack.com

Important Reminders

• We will never, under any circumstances, solicit funds from you via email, private message, or social media.
• If you are in doubt or notice any suspicious activity, please message the admins in our official Telegram group: https://t.me/cardstack.