The Etch Journey — the last three months — from a single smart contract to Google’s apigee and load balancers

“Did you hear that? This *Ethereum* will give us programmable money…. think of all the cool things we can do with that!”

But the guy sitting next to me at the London Bitcoin Expo back in January 2015 was only interested in the free Dogecoins that this crazy guy was throwing around.

Sigh!

I decided there and then that I was going to do my bit to make “programmable money” a reality.

When the Etch whitepaper was published nearly eighteen months ago we had a clear vision of the end goal — bring real-time wages to hard-working employees and introduce them to the Etch Economy (this is all the cool stuff you can do with money when it is programmable and flows like a stream directly from the employer). What we did not realise or appreciate at the time was to do this we had to come up with a solution to the scalability and security limitations of the blockchain first.

This last year has been about experimenting with the technology with small pilots and pitching the idea of Etch without giving the audience the classic “Future Shock”.

It was from these experiments that the idea of creating an employee bonus platform was born, for the following reasons:

  1. It gave the innovation team of interested companies a “small step” that was easier to sell to the FD without disrupting payroll.
  2. It allowed employees to be comfortable with dealing with crypto — wallets, private keys etc.
  3. It shows that we can create scalable architecture.

The early days of our employee demo — which was to dynamically add solidity contracts onto the Ropsten test blockchain — now seems no more than a distant memory as we embark on the exciting journey of implementing scalable solutions for blockchain implementations.

In this twelve week journey we have taken a set of ideas and built a solid business model fit for a corporate implementation at some of the world’s finest companies. We have created a layered solution taking the user from a single URL to a scaling blockchain solution.

In this post, we highlight this journey from javascript/HTML/Metamask/ and solidity to using Google’s apigee with target servers, docker containers, multiple cloud servers, and load balancers with docker/swarm/nginx.

These solutions when fully implemented will be able to process millions of transactions an hour putting us on the map.

How it started

In July this year, we started thinking about how users could really implement functions onto the blockchains, even if they did not know any solidity. This led us to experiment with Browser-Solc to see if we could dynamically deploy contracts onto Ethereum. Browser-Solc allowed us to compile contracts dynamically in the browser and then deploy them.

The contract code itself could be pieced together from keywords given by a user. Hence the user could enter a series of pseudo code type of instructions from which we could build the solidity. We tested with various demos and the deployments all worked.

A website at https://etch-prepare-for-takeoff.com/index.html was created.

Video Walkthrough

It was immediately obvious that a drag/drop system would allow the user to build a contract far more easily using a toolbox type of implementation. It was clear that the implementation itself suffered from gas costs, dependence on Metamask (unavailable on android/chrome), and scaling issues. We mused about what to do next.

Gas and Money

Of course, one enormous problem was Ethereum’s gas cost. As we wrote contracts, we were also dogged with “stack too deep” errors and “gas limit exceeded” errors. The economic and technical limitations (for now) of Ethereum were self-evident and also any dynamic deployment of contracts would be met with severe problems due to these errors.

The only way to really solve these issues, was to separate the UI and contract deployment completely — to implement an API solution.

Removing Metmask and APIs

Metamask had hampered us as it is was not available in Android/Chrome and also needed users to pay gas costs by approving transactions. In our testing, that presented major issues on mobile and also required users to have Metamask installed on their desktops.

We decided to use server side processing for the deployment and implement API solutions. We were not aware then that this would open an interesting door to the API economy.

APIs

Once we decided to use APIs, we could then produce a far more flexible solution in the cloud, free from the limitations of response times, Metamask, or types of browsers. This flexibility also opened up far greater access for users.

We were free to use sendRawTransaction and store keys under dotenv as a degree of security. This was our first implementation in testing contract functions in a static contract.

By using Postman we quickly confirmed that we could post directly to cloud servers and deploy solidity code. We had started to use Infura as a web3js provider which worked excellently.

Since Ethereum had released Web3 1.0 the formats in online posts had challenged us at first but we chose to use Web3@0.20.0 as more comments were about the older version.

The sample code is shown at the end of this post.

This code confirmed that from the cloud we had access to Ethereum via APIs and we could also break up contracts into smaller ones. This made deployment far easier.

APIs and Scaling

It was immediately apparent that the API implementation allowed for considerable growth under the principles of an API economy and Microservices.

This text from Google motivated us -

“A microservices approach is a significant departure from traditional software development models in which applications are built and deployed in monolithic blocks of tightly coupled code. These legacy approaches can make updating applications time- consuming, increase the potential for updates to cause bugs, and often limit how easily and quickly an organization can share or monetize its data, functions, and applications. Microservices, in contrast, are fine-grained, single-function component services that can be scaled and deployed independently, enabling organizations to update or add new features to an application without necessarily affecting the rest of the application’s functionality.”

It was clear that Microservices could be used to build a component based model for smart contracts and then later link smart contracts.

Hence we set about testing apigee.

Google’s Apigee & Load Balancing

The initial implementation of solving scaling for accessing the APIs was a solution using Google’s apigee. Apigee was superbly documented and we were quickly able to build flows and load balancer balanced on target servers -

We defined a flow structure which coded Apigee to send http requests to two servers at Digitalocean.

Testing in postman showed it all worked — we had solved a load balancing problem. A user could send API requests to Apigee using a single URL and Apigee would then send the requests to different machines.

The first layer was done but now a more interesting problem surfaced — fault tolerance on the servers and server management — we turned to Docker.

Docker and Load Balancing

We chose to use Docker initially to create containers of the core server.js file and this requires (dotenv, express, web3); then we tested with Postman and Postman posted successfully to ports on the containers.

We had solved the basic issue of containerisation of the server based solution.

The next problem to solve was one of using load balancing which at the time of this article is on-going. This would form the basis of layer two of our solution.

What’s Next

We are working industriously now on docker and load balancing. Then the work will focus on how contracts can be split and implemented using contract interface methods. These solutions will also allow new blockchains such as Hyperledger and Corda to be seamlessly added. That will allow additional expansion.

We also will examine Mulesoft.

Summary

About twelve weeks ago, we set about deploying contracts onto Ethereum to allow users to write contracts in the browser by using a series of keywords. It was soon clear that the solutions we had engineered had a wide scope and we chose to use an API based solution and then apply Microservice designs.

We tested with docker, apigee, digitalocean servers, and Ethereum. Our solution has shown a scalable solution is viable for deploying smart contracts on to Ethereum, and assessing functionality all via API calls and without the user ever coding solidity.

You can join the discussion on telegram: t.me/ThatsSteady

… and you can contribute to our open source code here: https://github.com/EurosEvans/employee-bonuses-etcheconomy

I would like to thank Trevor Lee Oakley for his help in preparing this post.

Sample Code

sendRawTransaction example for an addBonus function.