OrbitDB: Deploying the Distributed IPFS Database in the Browser

OrbitDB breakdown and how to utilise it today to store decentralised app data

Prelude: the Distributed Web is moving full steam ahead

Distributed protocol development is moving fast and will only accelerate heading into 2019, a year looking to be very exciting for decentralisation in general. Databases are no exception in this evolving software movement, and as such will be the focus of this article.

Gone are the days of in-house protocol development. The open-source ecosystem and emerging decentralised economics are paving the way forward for a more collaborative effort in shaping the future of software. Because of this, we can get a taste of what is coming by deploying the latest open-sourced efforts today. OrbitDB is one such effort leading the way in distributed databases; a peer to peer database run on top of IPFS.

What makes OrbitDB impressive is when it is running on top of the IPFS Javascript implementation in the browser. Doing so allows us to replicate and manage databases on a per-user basis. We will explore more about what this actually means, as well as the use cases it offers.

What is OrbitDB

The OrbitDB Github page bills the package a peer-to-peer database for the decentralised web. To achieve this decentralised nature the database is run on top of IPFS, as mentioned above. But unlike a traditional centralised database, where data is stored in one central location, (and perhaps replicated or sharded to other centralised locations), an OrbitDB database is serverless, and is replicated to all peers that are using it. And because of IPFS’s hashing protocols that do not allow duplication of a piece of content, there is no central point for such a database.

The hash is the true source, which could exist on hundreds or thousands of nodes across the network. With this in mind, the high level features of the database are as follows:

  • OrbitDB is compatible both server-side and in the browser.
  • OrbitDB is serverless. It does not require a central holding server, however, it does require at least one IPFS node to be willing to persist the database so it is not lost upon a browser disconnect, or garbage collected with go-ipfs.
  • Databases are automatically synced across peers. The IPFS pubsub protocol keeps databases up to date across peers, automatically. Pubsub is short for publish — subscribe, a model that reliably handles events and updates across peers.
  • OrbitDB databases are eventually consistent, achieved with conflict-free database merges (CRDTs). In short, database replicas across peers can be updated concurrently without the need of a coordination mechanism between them. Therefore, it is guaranteed that every peer of a database will eventually have the same database as everyone else; the same records, in the same order, etc. This is great under-the-hood management that front end developers don’t have to think about.
  • Because an OrbitDB database is replicated by peers and stored locally, the application then becomes offline-compatible. This offers more accessibility but will sacrifice reliability; data becomes more outdated the longer the app is offline.
  • OrbitDB supports write permissions. We can restrict write access to databases based on a special hash key, unique for each peer, that we will visit further down the article.

Note: At the time of writing, we are limited to defining write permissions only when a database is first initiated. However, OrbitDB core devs are working on more flexible permission management features, including read permissions, that will be rolled out in a future update.

  • OrbitDB data is public. Private IPFS protocols are being worked on, albeit in early stage, therefore will not be explored here. For the time being, make sure no sensitive data is stored on OrbitDB — stick to data you want the world to know about.
  • OrbitDB offers 5 datatypes to work with out of the box. Let’s visit those next.

OrbitDB Data Structures

OrbitDB offers a range of datatypes to work with out of the box, as well as the ability to add custom datatypes. We define a datatype to adhere to when initialising a database. We will visit the code this entails further down.

The datatypes we have out of the box are:

  • A log (append only, immutable) structure with traversable history. Useful for queueing and logging applications where you don’t want to delete entries.
  • A feed (mutable) structure with traversable history. Entries can be added and removed. Useful for shopping carts, comments, responses to articles.
  • A keyvalue store. As the name suggests, a simple key-value structure. Good for persisting app configuration or app state.
  • A document store. Similar to a document-based database like MongoDB, the document datatype allows us to store JSON documents as entries. The _id field can be utilised to lookup entries of a document. This more flexible datatype can be used for describing items such as products and articles.
  • A counter. The simplest of datatypes, a counter comes with an inc() function where we can pass positive and negative integers to manipulate our counter. Counters are useful for counting events in conjunction with a log or feed database.

Installing the required packages

To get OrbitDB up and running install both IPFS JS and OrbitDB, which can be installed with npm or yarn:

npm i orbit-db ipfs
#or
yarn add orbit-db ipfs

Note: If you are experiencing errors installing a particular package, as I did with tiny-secp256k1, documented in this issue, the solution entailed reinstalling a fresh npm with homebrew:

sudo rm -rf /usr/local/lib/node_modules/npm
brew reinstall node

IPFS JS is in alpha. Use with caution

It is important to stress that IPFS JS is in an alpha state, and APIs may indeed change as the project comes closer to a public release. However, we require it to run in conjunction with OrbitDB to run in the browser, so will be adopting it here.

It is generally not advisable to include pre-release software in your mission critical apps, but I fully recommend creating prototype versions of your apps with pre-release software to test the stability and learn from the implementation— you may be pleasantly surprised to see your app perform flawlessly.

Keep up to date with the IPFS JS via the Github project page:

It’s your call whether you wish to slowly roll out assets onto IPFS, release a beta build to a subset of your users, or simply wait until the final release before introducing it into your production infrastructure.

With that in mind, let’s now look to instantiate a database.

Instantiating an OrbitDB Database

With the ipfs and orbit-db packages now installed we are able to spontaneously instantiate an IPFS node in the browser, followed by initiating an OrbitDB database.

The code required to do so is as follows:

const IPFS = require('ipfs')
const OrbitDB = require('orbit-db')
const ipfsOptions = {
EXPERIMENTAL: {
pubsub: true
}
}
const ipfs = new IPFS(ipfsOptions)
ipfs.on('ready', () => {
//Create OrbitDB instance
const orbitdb = new OrbitDB(ipfs);
   //orbitdb is now the OrbitDB instance we can use to interact with databases
})

Instantiating in React

At this point you may be wondering how to adopt this code into a React state-managed app. The following gist addresses this concern.

Copy and paste the following into your React project (with ipfs and orbit-db installed) to observe the speed at which IPFS is instantiated, followed by OrbitDB:

On my local machine IPFS instantiation took little less than a second followed by a split second before OrbitDB instantiated — consequently triggering a re-render to reflect the updated state.

The Database Creator + Database Persistence

At this point it is worth briefly visiting the issue of which node creates a database. If we allow an end user to do so, running a browser-based IPFS node, we will witness the node being disconnected when said user leaves the webpage, or simply loses connection. The just-created database then becomes unreachable and destroyed.

What is needed is a highly available node to instantiate the database, and have its peers replicate and pin the database for it to become widely accessible.

The solution here is to have a server-side IPFS node as part of an IPFS Cluster create the database. IPFS Clusters are a subject for another talk, however, an IPFS node as a peer of a cluster will have its files automatically replicated and pinned by every peer in said cluster. This can be achieved with IPFS’s ipfs-cluster-service and ipfs-cluster-ctl packages on their Distributions page.

Learn more about installing and configuring an IPFS cluster with this introductory article:


Creating a Database

Now, with our orbitdb object instantiated we are now able to create and interact with databases on the IPFS network.

Creating a database is simple — a keyvalue database can be created with the following:

#keyvalue database
const db = await orbitdb.keyvalue('my-database')

OrbitDB methods are asynchronous and support the async await implementation in Javascript.

For completeness let’s run through the other database creation methods based on the other out of the box datatypes:

#log
const db = await orbitdb.eventlog('site.visitors')
#feed
const db = await orbitdb.feed('orbit-db.issues')
#docs
const db = await orbitdb.docs('orbit.users.shamb0t.profile')
#counter
const counter = await orbitdb.counter('song_123.play_count')

A resulting address string is generated once a database is created allowing peers to replicate and write (if they have write access) to the database. Let’s take a look at the address string next.

Note: If we wanted to replicate an already existing database instead of creating one, the database’s address string is passed into the above methods instead of a name string.

The OrbitDB database address string

OrbitDB database string consists of the following:

/<protocol>/<database_manifest_hash>/<database_name>

The protocol in question is always orbitdb. The database manifest file consists of vital metadata about the database. Lastly, the database name is the name we defined when creating a database.

A practical example of an orbitdb address can be the following:

/orbitdb/Qmd8TmZrWASypEp4Er9tgWP4kCNQnW4ncSnvjvyHQ3EVSU/first-database

To save repetition of readily available resources, more documentation of the address string can be found on Github.

Obtaining a database address

It is important to know a database string in OrbitDB as other peers will need it in order to interact with or replicate the database. The address can be obtained with db.address. db being your database instance:

const address = db.address.toString();

Next let’s visit accessibility, and how to configure peers to have write access to a database.

Giving Write Access

OrbitDB currently only supports configuring write access — by default write access is given only to the database creator, and read access is global. In order to give write access to more peers we pass in an access object when creating the database. That’s right; the only way to define write access currently is when creating the database. This will undoubtedly change in the future, but is a limitation we currently have to adhere to.

Passing the access object in practical use is done the following way:

const access = {
// Give write access to ourselves
write: [orbitdb.key.getPublic('hex')],
}
const db = await orbitdb.keyvalue('first-database', access);

The database is created in the same way as before, but this time passing the access object as a second argument. access is an object consisting of a write field, consisting of an array of public keys of peers that require write access.

Now, these public keys are not your IPFS identity hash. Instead, these are a peer’s OrbitDB instance key. In order to obtain your desired write-priviledged peers’ instance keys, they will each need to run orbitdb.key.getPublic('hex') after instantiating OrbitDB:

console.log(orbitdb.key.getPublic('hex'));

Once obtained, they can be plugged into the access object.

const access = {
write: ['key1', 'key2', 'keyN'],
}
...

The official documentation on Access Control can be found here on Github.

Database Events

At this point we are ready to interact with a database, but you may be wondering how we can listen to updates to a replicated database in order to keep your UI in sync.

We understand from earlier in the article that a database is automatically kept up to date between peers, so all that is needed now is to react to incoming changes from other write-able peers.

An events mechanism is built into a database instance. We can react to incoming updates with the replicated event (example taken from documentation):

// When the second database replicated new heads, query the database
db2.events.on('replicated', () => {

const result = db2
.iterator({ limit: -1 })
.collect()
.map(e => e.payload.value) {
console.log(result.join('\n'))
}
});

For further reading on replication and events, there is good documentation on Github.

Querying a Database

You are now in a great position to dive into the documentation and investigate the CRUD methods for each database type. There are not many, and are simple to use:

  • keyvalue put, set, get here.
  • log add, get, iterator here.
  • feed add, get, remove, iterator here.
  • docs put, get, query, del here.
  • counter inc here.

All OrbitDB methods are asynchronous, remember to treat them as promises with await.

Closing Comments

This has been a breakdown of using OrbitDB. Now is a great time to start experimenting with OrbitDB, and IPFS in general, to fathom its use cases in relation to your app infrastructure and to benefit from its distributed architecture.

Before deploying any OrbitDB-required assets in your production environment, remember OrbitDB is still in active development. Per their notice on Github:

Status: in active development
NOTE! OrbitDB is alpha-stage software. It means OrbitDB hasn't been security audited and programming APIs and data formats can still change. We encourage you to reach out to the maintainers if you plan to use OrbitDB in mission critical systems.

But with this in mind, there have been some impressive demos including orbit’s real-time chat app, at https://orbit.chat, combining Ethereum and web3 with uPort integration, and IPFS and OrbitDB for the chat content.