IOTA Meets Couchbase

A step closer to a (semi) permanode.

Olaf van Wijk
Feb 10, 2019 · 5 min read
Image for post

The discussion of permanodes is one with a long history and is one that always seems to come back. With the release of local snapshots it becomes more and more uncertain if your data will stay alive for longer then 30 days in the tangle. However with local snapshots separation of concerns also become possible! And this is really great!

For our project, just like many other projects, we want to choose if we keep data available on the tangle or not for application specific purposes. Snapshotting is simply put -a pain in the ass- when it comes to this and with localsnapshotting being activated it makes harder and harder to keep data alive in the tangle. Ideally we want to selectively keep data and prune what we don’t want to store but to do so we first need to be able to store everything and move from there.

So I decided to run a little experiment to implement a persistence provider other then RocksDB and ZeroMQ. One for an external database, couchbase!.

The benefits!

By using an external persistence provider we can keep the running instance of IRI relatively small without sacrificing the storage and data availability capabilities.

  • We can run multiple IRI nodes that use a clustered back-end storage so we can run with a near 100% update.
  • We can setup IRI nodes with different roles: Active larger part of the live tangle and a few IRI nodes exclusively for accessing historical parts of the tangle.
  • The API didn’t change!
  • We can independently scale the back-end storage service without the need to ever shutdown our IRI nodes.
  • Opens up research for semi permanodes that can store data selectively.
Image for post

Why couchbase?

A few months ago during a great community effort to create a permanode (localsnapshotting etc was all unknown back then) I already wrote a document with some considerations that still hold in this discussion(Wide-column, Graph, Key value etc vs CAP):

The conclusion there was to choose for RiakDB with a personal bias to having a better experience compared to couchbase and the more advanced features being part of the open source version(like cross data center replication). Community members notified that Riak’s team from Basho was in some trouble with a big chance of Riak being discontinued.

So now… why couchbase?

  • As from the document : A key value store is the best all round storage for tangle type of data.
  • Couchbase has a master-less setup, for use this means all nodes can participate in write operations. This opposed to other setups like mongodb which are master-slave where writes are a bottleneck. Master-less allows for proper horizontal scaling on both writing and reading.
  • Since the tangle is asynchronous in nature having backend nodes and indexes not immediately consistent is not an issue.(It is writer consistent, meaning that if you write something to a node, the same node will have a consistent view and update of the data)
  • It allows for key-index eviction: most systems keep primary keys in memory. For something like the tangle with very high-cardinality data this feature is of utmost importance to keep the memory usage of an ever growing database system in check.(Most data is unused)
  • It is darn easy to setup and manage
  • It can handle binary data, this is important since storage of Trits in a UTF-8 encoded text format is just not efficient on a massive scale.
  • Services(Storage, indexing, caching etc) are independently scalable. For performance tuning each specific use-case this is very important.

Side note

This was an effort to show to myself, mysoundsafe and the community it can be done and totally love to get some feedback on this. The project is in absolute infancy (1 commit) and far from finished even though most API functionality works normally. Some side notes on the current version:

  • No tests yet (all previous tests pass and is backwards compatible!)
  • It has only been tested on a personal testnet (with coo and all) easily dealing with 40tx a second (a bit cheated with MWM3)
  • It has been a very long time ago I did Java, constructive feedback is welcome ;)
  • Ordering of transactions changed, with IRI the ordering is dependent of the order IRI ‘saw’ the transactions and with couchbase the ordering is based on the ASCII order 9ABCDE… of the transaction ID’s
  • Probably more things pop up

Test it yourself!

Follow the normal installation instructions about building from git but use my repo instead:

Install couchbase on your machine and create two buckets: ‘tx_bucket’ and ‘tx_metadata’ and add a separate user with application rights to the created buckets (don’t feel scared, it is really really easy with the UI) and manually execute these query comments to create the relative indexes:

CREATE INDEX ix_address ON `tx_metadata`(address);
CREATE INDEX ix_bundle ON `tx_metadata`(bundle);
CREATE INDEX ix_trunk ON `tx_metadata`(trunk);
CREATE INDEX ix_branch ON `tx_metadata`(branch);
CREATE INDEX ix_tag ON `tx_metadata`(tag);

Add these lines to your iri.ini:

COUCHBASE_NODES = then your node ofc
COUCHBASE_USERNAME = iri //With your user
COUCHBASE_PASSWORD = testiri // And your password

That should do it! Enjoy your couchbase database backend!

Any questions feel free to contact me on the IOTA discord: Olaf van Wijk#1273


Like the work?

Iota-pay donationlink: OR


Image for post

Get Best Software Deals Directly In Your Inbox

Image for post


Coinmonks is a non-profit Crypto educational publication.

Sign up for Coinmonks

By Coinmonks

A newsletter that brings you week's best crypto and blockchain stories and trending news directly in your inbox, by Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Olaf van Wijk

Written by



Coinmonks is a non-profit Crypto educational publication. Follow us on Twitter @coinmonks Our other project —

Olaf van Wijk

Written by



Coinmonks is a non-profit Crypto educational publication. Follow us on Twitter @coinmonks Our other project —

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store