Enterprise Readiness
After months of hard work we are proud to announce that DataBroker DAO is now running on the new version of our Mint middleware solution. The new version is simply called Mint 2.0 because as we are developers we’d very quickly come up with crazy names that seem funny at first but are eventually just a nuisance to explain to new joiners. I kid you not: I once refactored a micro service and we ended up renaming it to ‘Natalie’, after Natalie Portman, for no reason other than the apparent one.
The diehard followers of our project will know that not so long ago we raised money through a token sale, and we are still very grateful to all the people backing our project and giving us the opportunity to keep working and building out our Data Exchange platform. In light of this we felt it was our responsibility and an opportune time to go a little deeper into the inner workings of the backend now that our End of Summer release is out. Over the past 3 months most work has sunk into re-engineering the tech on which our platform is built. I won’t go into detail about the numerous improvements to the frontend, which were discussed in previous posts already.
TL;DR: Mint 2.0 is bigger, badder and bolder; and DataBroker DAO’s Data Exchange platform is now running on it. And by the way you might say “boy this post is way too long”, but I looked it up and apparently I only overextended the ideal blog post length by 1.4%.
Some history
As a background into what exactly Mint is I’d like to refer you to the following overview of its functionalities. In short, Mint is a blockchain agnostic layer that sits on top of the blockchain and acts as both an indexer and adapter layer for all in- and outgoing traffic to and from the blockchain. Mint 1.0 was built using NodeJS and mostly relied on the well-known Ethereum Javascript API web3.js, in combination with MongoDB 3.2 and IPFS as data storage layers. One of the main issues we were facing with this setup was that we needed to maintain an enormous amount of watchers in the web3.js library, one for every sensor deployed to the system. We discovered however that these watchers were suffering from a memory leak. For us this was a good point in time to decide to mature the entire stack and move to newer, more cutting edge technologies.
The new stack
TypeScript
One of our first decisions was to rewrite our entire codebase in TypeScript, which many of you know is a purely syntactical superset of Javascript offering static typing and thus type safety.
Many articles have been written about this already so it’s not very meaningful to go into great detail here. Suffice it to say that programming in TypeScript is bliss, especially for engineers with an object oriented programming background. Once you’ve worked in TypeScript, there’s just no going back, up to the point that even looking at plain Javascript hurts my eyes every time I see it.
Ethers
After being stuck in limbo for the past year between web3 v0.2 with all of its known issues, or upgrading to v1 which is still in beta and thus possibly comes with some instability, we were glad to see a new library was gaining some traction. When we discovered our watchers were leaking memory we decided to move once and for all and rewrite the whole thing using the ethers.js library. After a couple of months of working with it, and having encountered numerous corner cases making me dig deeper in their lower level API, I can say it’s very readable and easily extendable. Kudos to Richard Moore who’s behind the project and is very actively tackling all open GitHub issues on a daily basis by the way.
RabbitMQ
Mint 1.0 was mostly purely synchronous behaviour, which means that when you were interacting with the blockchain through our REST API you were bound by the default 60s Nginx timeout window before your calls would end up in gateway timeouts. Other than this we also felt we were very limited in the amount of open connections our servers could handle at any point in time. If some requests would tend to stall a little, this number became very unpredictable and our overall system ended up becoming rather unstable during moments of heavy load. As we are eyeing to become an enterprise level data marketplace for worldwide IoT sensor data this obviously was not an option for us.
By rewriting all our endpoints to handle requests in an asynchronous manner, meaning that POST requests immediately return a request identifier for which you can poll the response later on, we brought back stability to our systems, even in times when the platform’s under stress. This comes in very handy as not all blockchain applications are necessarily a continuous stream of transactions that need to be sent to the chain. On the contrary, that traffic is more often than not very bursty and using this technique we managed to not only buffer incoming requests but we’re actually able to optimise the processing up to a point we can fit more transactions into a single block, thus increasing our overall TPS.
Aside from that it enabled reworking the codebase to a fully decentralised architecture, we ended up containerising every service just because it felt natural to do so. We’ve also managed to make synchronous communication between consumers possible by implementing RPC calls using simple direct replyTo queues. Obviously these should be treated with a little caution but they are extremely useful for nonce tracking for example. It allowed us to make a nonce tracking microservice able to keep track of your nonce on multiple networks, or even if you’re interacting with multiple Mint instances at a time, again increasing the overall TPS as we’re able to correctly predict nonces without running into too many invalid nonce errors. For the most part though our workers are mere small snippets of code executing on certain data and republishing messages to a wide array of queues for further processing, which means we can scale horizontally.
Another great technology which we implemented and which goes hand in hand with AMQP is MQTT. Coincidentally this protocol was designed for enabling communication between IoT devices, but we just use it for the slimmed down pub/sub messaging protocol it is. The rabbitmq plugin does nothing really other than host a socket on which frontends can subscribe to topics they’re interested in; we however ensure that our entire stack religiously communicates every interaction with the blockchain, tagged by your request id for example. Now if you use your imagination and combine this with react and redux, I’m sure you can imagine just how powerful this solution really is.
Mongo 4.0
What can I say ... I freakin’ love Mongo. We decided to step away from ODMs like Mongorito or Mongoose and write all our queries raw using the incredibly awesome aggregation pipeline. I remember back when WiredTiger was first introduced I was amazed at the power of their engine and with the new versions 3.6 and later 4.0 they’ve just blown me away with the capabilities of their tech. One of the great additions I feel in the latest version are the change streams cursors enabling you to have very fine-grained listeners on all your Mongo collections, triggering the necessary queues and updating whatever’s required.
Webhooks
I think it was a Friday in August when Roderik opened up a medium prio ticket stating “Hey don’t you think it would be cool to add webhooks?”. Well by Saturday evening we had implemented three different types of webhooks, so users of Mint can now subscribe to:
- Events that get triggered
- Addresses that are interacted with
- Timed hooks that relay the user’s wallet balance for example
ERC20 token bridge
I guess you can say this token bridge we’ve built is part of our new stack as well. The reason we need it, is to allow for scalability of the marketplace and still maintain a real life use case for our DTX token on mainnet. It mitigates scaling issues because it enables using a private PoA network, where you can decide the gas cost and blocktimes yourself. Furthermore it protects you against the congestion that we see happening on the mainnet from time to time.
It’s actually pretty cool, I’ve pulled this schematic from the internet somewhere describing the deposit flow. In short it’s basically just 2 contracts on 2 different chains that communicate with each other through events for which you have to listen to in a couple of validator nodes that are signing requests using seeds the contract on the mainnet knows the addresses of. Read that twice, it’s really not that difficult when you get how it works. Implementing it however is a real PITA, but that’s just because... blockchain.
What’s next
All of the above doesn’t even describe half of what we’ve been doing, but the rest is top secret and we will never tell you. Most of all we’re glad to have the ERC20 token bridge in production, allowing everyone to transfer DTX tokens from the mainnet onto our private PoA network and making the DTX token truly useful and usable on our platform. With that I’d like to welcome every data sensor owner or buyer to start interacting with it. Withdrawals are disabled for the time being.
Now I hear you all thinking ... why is this blog post titled Enterprise Readiness? Well we, the DataBroker DAO team, feel that with this End of Summer release we are finally ready to welcome enterprise level players onto the system. If you have data just sitting there, collecting dust in your data silos, please don’t hesitate to contact our admins. We will guide you through the enlisting process and answer any questions you may have so you can turn that pile of bits into a big pile of gold, DTX tokens, whatever you want to call it!
Oh, and I was kidding about the withdrawals by the way, of course we made these possible too!
Stay tuned, because winter is coming and we plan on working even harder, now there’s no real reason to go outside anymore.
