My Elixir journey in GO.Exchange
What is GO.Exchange
GO.Exchange is a cryptocurrency exchange with the aim to be a secure, sustainable platform. However, due to the regulation intensive in the cryptocurrency area. Sadly, GO.Exchange was closed on 15th March 2020.
I’ve worked for almost 2 years on GO.Exchange as a developer and started building the product from the beginning. We adopt Elixir as a programming language and use it in production. The reason we chose Elixir because it has a good concurrency model, very good performance, the syntax is nice and readable.
I think would benefit to share some technical stuff that I’ve learnt in this journey with myself and also along with the team.
Elixir processes
Elixir processes are very useful and increasing our productivity. Elixir processes played in major parts for 2 reasons.
As a periodic job. There are a lot of open source libraries. For example: Quantum, the most popular one. However, we decided to implement by ourselves which is more fit for our use case. It was a good decision which currently we have the flexibility to handle many use-cases base on business requirements.
As a consistency control, In order to prevent double-spending and race conditions in the Exchange. It leverages a process/mailbox behavior in Elixir. The mailbox mechanism is a battle-proven which make us very confident. We use processes quite a lot underneath core business logics inside Matching Engine module, which is the module that handle incoming buy and sell orders. Once the price between buy and sell is matching together. Those orders will be executed as a trade. The logics of the matching engine such as matching limit orders, market orders or cancel orders. The challenge is how to design processes to work together. There are many ways to do this with the pros & cons. For example: We can design only single process per matching engine. It will guarantees that every orders will be proceeded in order, no double spending while we can keep the code simple. However, It could cause big bottleneck in the application. We can bring up more processes such as per trading pair or per price level. (What is price level? you can checkout this talk if you want to learn more) It will reduce the bottlenecks but also will introduce more complexity to the code and more points of failure. Batch processing and back pressure could also be introduced as well. Those idea I have mentioned are just examples. There are plenty of creative ways we could do with the Elixir processes. And the design decision may depend on many factors like trading behavior of the users, user experience or business requirements.
We did spend quite a lot of time to improve the bottleneck. And the effective way to find the bottleneck was investigating the actual trading behavior from the users. Building the exchange, we were very serious about the performance. However, in the same time we have to keep the balance between performance and not over engineering. It is not easy job. There was a moment where we have to split up the team to dedicate on continuous improvement on bottleneck and performance areas.
Using the process requires some level of learning curves. Also, it could cause unexpected behaviors when it crashes. Even we always keep in our mind about “Let it crash” However, there are always edge cases that make us headache. In a serious situation, it could bring the entire application down.
Phoenix Framework’s Context
Phoenix’s Context is so confusing to me and always be. It is hard to make it right in the first place and requires ongoing refactoring in order to make it right. One of our microservices, we excluded Context in the app structure and it helped because we don’t have to spend our time thinking about it. Even I could see in the future that we have to introduce it. But I think that it is easier to introduce Context in the latter stage than introducing it in the first place once we have good understanding about our domain.
Arbitrary precision decimal arithmetic
The Exchange has to deal with numbers with long precisions. For instance 0.000000000001ETH. We cannot just use the basic float type in Elixir since it cannot calculate the exact number in precision. I was in Ruby world for a long time where BigDecimal or Money gem is pretty handy. In Elixir, We use Decimal library to deal with decimal places. It is not as straightforward as Ruby. There is a concept of context introduced which makes the code a bit more complicated at some point.
Testing
Unit test: Mox was used in the beginning. Even the Elixir community loves it, I personally don’t feel it is productive and there are some limitations. For example: Not every functions that are easy to get dependencies injected. The codebase doesn’t seem to be easy to read when passing the dependency parameters around. After a while, We have decided to switch to Mimic. Our life is getter better. Mimic is very smart testing library. What I like about Mimic is: 1) We cannot just create a mock with random/unknown behavior. The behaviors have to be existed. 2) With less dependencies injection, the codebase is much cleaner. 3) Easier to TDD
Contract Test: It is not easy to test the collaboration between services in microservices world. Consumer Driven Contract Testing is one of the testing technique that help a lot. PACT is a one of the framework which available in many programming languages such as Ruby, Javascript, GO. Sadly, For Elixir It is still in PRE alpha phase. (PACT_Elixir) What a pity!
Request test: We use it to test API requests as the higher level above controller test. It doesn’t come with default Phoenix. We just set it up our own. Request test will allow us to focus on request parameter, route path and response and let the controller tests and other unit tests handle the rest. In the request tests level, it really works well with API document generation. We also used Bureaucrat to generate the document. Bureaucrat takes the test description, request parameters, response and map into the API document. We don’t have to maintain API document our own, just need to generate it along when running the tests. It is really handy. We also can set the output format such as API blueprint, OpenAPI spec, Markdown and etc.
Umbrella apps
We started from one application and when it grew up. We were trying to split it into Umbrella apps. Good thing about the umbrella apps is we can see how much domain coupling in our application. The team will have a good virtual when looking at the code to discuss the solutions for decoupling. Then we can make a better decision in order to move into microservices or not.
However, There are also some challenge topics related to how to deploy each umbrella apps separately. For example: How can we detect the changes which umbrella apps has been modified and will be deployed specifically? How can we make sure that umbrella app is compatible with other umbrella apps?
Libraries
Elixir is a new language and the community is not so big comparing to other programming languages. We often faced issues related to libraries. such as outdate, not compatible with others, bugs. BTW, I believe it will be getting better and better when there are more Elixir adoption. Also we used some of Erlang libraries. For example: Pot, for handling multi-factor authentications. It worked really seamlessly and didn’t feel that I wrote another language inside Elixir or making the code messy.
Blockchain
We also assessed several Ethereum libraries such as Ethereumex, Eth. Those libraries work well as expected. However, We decided to write our own implementation due to the better code quality control and security reasons
Deployment
We created releases using Distillery since version 1, 2 and mix release respectively. Since, we‘re trying our best to follow the 12 factor app. One release should be able to run in against any environments, Test, Staging, Production. It makes us really have to aware of run-time environment variables. We did some mistakes where the environment variables was taken when compiled time instead. Luckily that the issue that we found was a minor one. However, it could be very risky since we cannot detect easily. Our workaround was setting up a convention in team, using code static analyzer, manually testing the release inside Docker to make sure of everything.
Erlang cluster on top of Kubernetes pod was also a challenge. There are some configurations that the team between DevOps and developers have to work very closely. I did use Peerage to manage it. Why do we need the cluster? Actually, we don’t need it if the application is simple and stateless. However, if we want to use the power of Elixir processes then most of the time we will need it. One obvious example is scheduling jobs in the application. Let’s say we have 3 pods running. If we don’t setup the cluster, all the pods will execute the same jobs. With the cluster, these 3 pods will have a way to communicate among them to make sure that only 1 process will be executing the job. CAP theorem is becoming important since now we talk about distributed system already.
Last but not least
I still believe that adopting Elixir in GO.Exchange was a good decision. Hiring developers who has experiences in Elixir is very hard and rare. As a result of that, we tended to hire good developers and transform them into Elixir developers instead. More than half of the team doesn’t know Elixir before joining the team and it successfully transformed Javascript devs or Ruby devs to Elixir devs. Actually, Elixir is the programming language of happiness. Some of us don’t want to come back to the old day of Javascript again. 🤞
Lastly, I am sure that there are many things I haven’t covered here. Hopefully, I could share more next time. Thank you for reading! I hope this article is helpful. And thanks to our team that gave me a chance to learn a lot of great stuffs along this journey.
