Scaling A Cryptocurrency Exchange Using Go & Microservices

Written by Chad Peterson, a software engineer at Luno.

When we first started Luno, we wanted to build an intuitive app, that was both secure and robust. We needed to move quickly but didn’t want to compromise on user experience.

Our architectural solution needed to include simple, widely accepted design principles and patterns. To “move fast and take action” (one of our Moontality values) we needed to take a pragmatic approach.

We built three monolithic systems using Go programming language

We decided to build the platform with Go, as it is a modern systems programming language with an amazing concurrency model — and also widely used in the cryptocurrency community.

The three monoliths were built within a monorepo, utilising Docker containerization (and as much off-the-shelf AWS tooling as we possibly could).

This resulted in the API system taking on the responsibility of handling all public API requests, mobile application requests and web traffic. Matcher became our exchange matching engine to match buy and sell orders, along with sync handling our blockchain and banking integrations.

Before 2017, our initial architectural design worked well. The three monolithic systems were interacting with a single RDS MySQL database.

By the end of 2017, the bullish market saw an influx of new customers, with many customers making multiple daily transactions during this period.

The single database concept started becoming a bottleneck during the December price surge.

We realised that we needed a more scalable solution, and our architectural design needed to be revisited.

This spike in customer activity, placed a lot of strain on our single RDS MySQL database instance. To meet the growing demand of activity, the scope of these three monoliths started growing disproportionately. For example, Sync started sending out emails and push notifications for transactions.

Microservices

To solve this engineering challenge we turned our focus to building microservices. We needed to ensure the approach would allow us to get something out quickly, in order to immediately alleviate strain on our systems.

We looked to the community to help identify design ideas and share best practices. The principles stood out:

• Creating pure functions
• Idempotency
• Writing “golden path” functions
• Choosing explicit functions parameters over global variables
• Keeping it simple

Before we could start building microservices, we first needed to deepen our understanding of what microservices are. In our context, a microservice is “a development technique that structures an application as loosely coupled independent services”. These services should have a well defined bounded context in which to operate, should have a single purpose, and ideally own its own data and resources.

Apart from microservices being isolated, they should also be capable of communicating with each other over a well-defined protocol. At Luno we chose Google’s gRPC framework along with protocol buffers for microservices. gRPC is an RPC framework that enables clients and servers to communicate transparently over HTTP/2 with each other. It generates cross-platform client and server bindings for many languages. On the other hand, we use protocol buffers as the IDL for communication that help define the services provided as well as the objects that are transmitted across the wire.

Our approach to Microservices

When building out a microservice we first define the bounded context we would like to extract out of our monolith. We build an interface that defines the behaviour our new service will be offering. Then slowly port over the functions from the monolith into its new home within the microservice.

Our main rule when building microservices is to make the services client interface the heart of the microservice. Thus ensuring that we only expose properties and functions that are allowed to be accessed from outside the microservice. Finally, we add metrics to monitor and measure the microservices’ effectiveness.

The diagram above shows an example exchange Go package we could build to enable us to move the exchange out of the Sync service and into its own microservice. We would follow the steps to help guide us with designing our microservices. It enables us to operate at our scale as well as help us collaborate better, and improve the maintenance quality of the microservices we produce.

Root package

The root package in our design defines the microservice interface that specifies the functions that the service provides and would be defined within the exchange.go file. The root package also contains the domain-specific types and enums that we would like to expose for other microservices to make use of. This is possible since we use a monorepo. These types and enumerated values are defined within the types.go file. This is beneficial as other microservices doesn’t need to traverse the packages too deeply in order to make use of domain objects and types.

We also define metrics that we’re keen to measure within the root package. We might be interested in measuring request latency or monitoring the number of errors our service produces to help us maintain the SLA that we define for the service.

This touches on another one of our key Moontality values — “we prefer facts to fiction”. We need to ensure that our services are resilient and that they work as expected. If anything impacts the SLA’s that we define we need to have the appropriate data to make decisions.

The root is also the home for microservice documentation. We strive to create amazing software that our engineers enjoy working on so we document our services well within the docs.go file. This file’s only purpose is to provide clarity to the engineers looking to work on the microservice and those looking to leverage the functionality provided.

Client and Server packages

These client packages houses various implementations of the microservices interface. We use gRPC to enable microservice communication and one of the clients could be a gRPC client that allows other microservices to access functionality via gRPC. The server package, on the other hand, takes requests from a client, invokes the various dependencies it has to handle the request in an attempt to fulfill the request. The server package speaks to internal packages, such as the db package and ops package in order to execute the logic required to fulfil the request.

The glue used to propagate a request and response among a gRPC client and server would be the protocol buffer messages. Therefore, within a separate package called exchangepb we define the RPC service functions along with the request and response messages. This enables the client and server to concretely define the structure of the data that is being transmitted over the wire as well as expose the operations the microservice has available over gRPC.

Internal and cmd packages

The internal package houses all the logic that we do not want any other microservices to make use of. These could be functions that are unsafe to invoke outside of the context of the microservice, or types and enums we do not want to expose. These could also be operations and resources we want to strictly enforce data ownership by the microservice.

The internal package is a special Go package that provides the above out of the box. It allows us to hide away certain abstractions, types, functions and resources that the outside world should not make use of, as they are potentially unsafe to invoke from outside the context of the microservice and breaks the design we want to build.

A few great candidates for enforcing these rules in Luno’s context are database queries and business logic operations. Within the db package we add multiple database entities and define the CRUD operations on those entities. The ops package uses the db package to add further business operational logic to fulfil business requirements. The functions in the db and ops packages are then used within the server package to help fulfill the requests from clients. Therefore, external packages or microservices should not need to access the properties and functions within db and ops. Hence they can be moved into an internal package.

The cmd package defines the many driver programs that the microservice has that you could run as separate binaries. This could contain a cli, a daemon, an HTTP server or anything else that you can think of.

The above defines the package structure we love at Luno.

We find its design simple yet expressive. We have been playing with this design for a while now and have 27 microservices that have been chipped away from our monolith that were structured in this manner. We find this design is easy to understand and reason about. It also eases new feature development and maintenance.

If you enjoyed reading this, and are interested in working on similar projects, why not check out our current software engineering opportunities based in London, Cape Town and Johannesburg?

Originally published at www.luno.com.