March Development Update
Issue 05 / March 2018
Our engineering team has been working on a ton of new features since ETHDenver. On the way back we collected feedback we received and mapped out the work for the next few months. Our focus now is on knocking out all the little things that could help more projects launch with us in production. At the end of this push The Graph should be usable for 90%+ of Ethereum projects. Let’s take a look at how we’re tracking.
New Features
Big decimals
In order to support big number math, we’ve added support for big decimals across the board. Entity fields can now be declared as BigDecimal or BigDecimal! in the subgraph’s GraphQL schema to make use of the new BigDecimal type in entities and GraphQL queries:
type User @entity {
balance: BigDecimal!
}In mappings, BigDecimal values can be created from BigInt and strings with BigDecimal.fromBigInt(x) and BigDecimal.fromString("33.45"). In addition to this, BigInt values can now be divided by BigDecimal in order to obtain a precise result of the division:
let x = BigInt.from(100)
let y = BigDecimal.fromBigInt(BigInt.from(33)) # y is now "33.0"
// or: let y = BigDecimal.fromString("33.0")let z = x.divDecimal(y)
// z is of type BigDecimal and has the value "3.0303030303..."
IPFS stream processing
The existing ipfs.cat API provided by @graphprotocol/graph-ts is convenient for small to medium sized files. To account for larger files as well, IPFS stream processing has been added in the form of ipfs.map and its convenience companion ipfs.mapJSON. Using the latter, it is now possible to map over all JSON values in e.g. a JSONLines file and process them in a dedicated callback:
export function handleTransfer(transfer: Transfer): void {
ipfs.mapJSON(
// IPFS hash
"Qm...",
// Name of a callback function
"handleItem",
// User data, can be Value.fromNull()
Value.fromString("parentId")
)
}export function handleItem(value: JSONValue, userData: Value): void {
let obj = value.toObject();
let id = obj.get("id").toString();
let title = obj.get("title").toString();// Callbacks can also created entities
let newItem = new Item(id);
item.title = title;
item.parent = userData.toString(); // Set parent to "parentId"
item.save();
}
GraphQL interfaces
To allow expressing information that different entity types have in common, support for GraphQL interfaces has been added. Subgraphs can now define interface types in their GraphQL schemas and have entities implement and refer to them. Like with entities, query fields and arguments will be generated for interfaces in the subgraph’s GraphQL API:
interface SimpleWallet {
id: ID!
balance: BigDecimal!
}type NamedWallet implements SimpleWallet @entity {
id: ID!
balance: BigDecimal!
name: String!
}type User {
wallets: [SimpleWallet!]!
}
Thanks to the generated GraphQL API, queries for entities implementing SimpleWallet and entities that refer to entities that implement SimpleWallet are now possible:
{
simpleWallets(first: 5, orderBy: balance, orderDirection: desc) {
id
balance
}
users {
wallets(where: { balance_gt: "0.0" }) {
id
balance
}
}
}GraphQL arguments for relationship fields
As of the latest graph-node release, it is possible to use the same arguments that are available for top-level query fields — first, skip, where, orderBy, orderDirection — for many-to-many and one-to-many relationship fields as well.
Subscription throttling
Initially, GraphQL subscriptions were checked and updated on every entity change. This caused a substantial amount of queries and traffic, particularly while subgraphs were still syncing. To mitigate this without skipping any relevant changes, the implementation was altered to throttle subscriptions to one update per 500ms while syncing, and to only update them once for every block with relevant entity changes.
Testnets on the Hosted Service
We’ve launched support for Ethereum testnets on the Hosted Service. In addition to mainnet, subgraphs can now be deployed against Kovan, Rinkeby and Ropsten. To achieve this, simply write
network: kovan # alternatives: rinkeby, ropsten, mainnetin the subgraph manifest under the data sources. When deploying subgraphs to a local Graph Node, this field has no effect, since Graph Node currently connects only to the Ethereum network that is passed in via --ethereum-rpc, --ethereum-ipc or --ethereum-ws.
Logs search and filtering in Graph Explorer
Subgraphs generate detailed logs of their indexing progress, including which blocks were scanned, skipped, which ones included events for the subgraph, which events were processed using which event handlers in the mapping and why indexing failed in case there is a bug in the subgraph or Graph Node.
To make it easier to find the information you are looking for, Graph Explorer now supports searching logs, sorting them by date and and filtering them by log levels.
graph init
We want to make it as easy as possible for developers to get started using The Graph. We’ve added a new Graph CLI command called graph init that takes a subgraph name and creates a local project from our example subgraph.
With the latest Graph CLI release, graph init has been extended to support a second mode: creating a subgraph from an existing contract that indexes all events from the contract and stores them as entities. This gives smart contract developers a subgraph that they are immediately familiar with. In addition to this, graph init walks users through an interactive form, allowing them to enter bootstrapping parameters as they go.
Subgraph migrations in Graph CLI
As we improve the The Graph, we want to make it easy for developers to stay on top of the latest developments. We have therefore added automatic subgraph migrations to Graph CLI. Whenever it is possible, graph codegen, graph build and graph deploy will automatically update subgraphs to the latest manifest changes, mapping APIs and more.
A first migration has been shipped with the latest Graph CLI release, incrementing the apiVersion of mappings to make input data on Ethereum transactions available inside the mappings.
Automatic version switching
Being able to atomically update subgraphs without downtime is an important feature of the Hosted Service. In order for others to benefit from this as well, support for subgraph versions has been built directly into graph-node. With this feature, every deployment made to a subgraph creates a new subgraph version. Each subgraph also stores a reference to its current version. Updating this current version is implemented in the form of two version switching modes: instant (default) and synced (used in the Hosted Service).
With instant version switching, a new deployment instantly becomes the new current version of the subgraph. This mode is particularly useful in local development. In synced mode, a new deployment only becomes the new current version of the subgraph after it has been indexed up to the current chain head or if there is no already synced current version yet for the subgraph.
Indexed event parameters and other Ethereum improvements
We recently landed a number of fixes and improvements related to Ethereum ABI decoding/encoding. Specifically, indexed event parameters are now properly decoded as arrays of 32 bytes and treated as such in mappings. Overloaded events with different signatures are now correctly resolved when deciding which handler should be called. Last but not least, contracts compiled with older Solidity compilers — causing event data to not always have a size that is a multiple of 32 bytes — are now handled gracefully.
Extended documentation, FAQ & support repository
We wrote new getting started guides for local development and development using the Hosted Service. We also added detailed documentation for the AssemblyScript API available in mappings.
Other improvements
Various smaller improvements have been made to our GraphQL implementation: @include directives are now handled correctly, @derivedFrom now works for one-to-one relationships, query variable parsing has been improved and where filters have been added for list fields.
On the database side, indexes were added for entities and their attributes to speed up queries. Several of the procedures involved in block ingestion and subgraph status updates have seen performance improvements. The amount of data stored by Graph Node has been reduced significantly by only storing entity history data that is needed.
Several tuning parameters supported by Graph Node are now exposed via documented environment variables. This includes GraphQL query timeouts, event handler timeouts, IPFS file resolution timeouts, IPFS file size limits as well as the main parameters that affect Ethereum block ingestion and request parallelization.
What’s next?
This month, the team is working hard on delivering the rest of the adoption blockers. Support for Ethereum tuples/structs is already in code review. Block and transaction triggers are being tested in our staging environment this week. Dynamic data sources for registry/factory-style smart contracts are close. On top of these features, we’ve kickstarted an effort to start optimizing query and indexing performance to make everything blazing fast. It’s going to be a good month!
Get Involved
Would you like to get involved in developing The Graph? We have easy starting points for new contributors: good first issues in Graph Node, good first issues in Graph CLI.
We always appreciate feedback about The Graph. Don’t be afraid to file issues against Graph Node, Graph CLI or Graph TypeScript.
Have questions or want to chat with the team? Come join our Discord!
If you’re excited about The Graph and would like to explore a position on the team, take a look at our careers page.
Happy Hacking!
— Jannis & The Graph Protocol Team ✌
