Technical Strategy Principles at OY!

We started OY! Indonesia in 2017 to build a chatting application that allows fund transfers between users. In late 2018, we pivoted to focus solely on building the best fund transfer product for end-users and businesses in Indonesia. It has since grown to a company that processes transactions worth billions of dollars annually.

Underlying all of this growth are systems designed, implemented and maintained by members of our engineering organization. Our business pivot and growth has forced us to swiftly build (and sometimes, rebuild) these systems. Over the past couple of years, we have converged on a set of technical strategy principles that we refer to when we want to make big technical changes.

We are sharing our technical strategy principles to provide a window into how we think about making big technical changes; from re-architecting our micro-services, choosing a new framework to adopting a new programming language. We want to help prospective candidates understand the technical strategy principles at OY! and what they could expect when joining.

Context

In order to better understand our principles, it is useful to first look at the context in which our engineering organization is operates at. These are some of the most important ones:

• Team Size: We have less than 30 people in our engineering organization in total who are responsible to ensure that our systems scale with the growing traffic, while at the same time iterate on our existing products or launch new ones quickly. Engineering resources are scarce in our company.
• Scale: We are processing a large volume of transactions monthly and the numbers are growing rapidly month-over-month. It is important that our systems and processes continue to keep up with the product growth.
• Mission-Critical: Businesses and end-users use our fund transfers platform to power critical transactions, so it is important that funds transferred through OY! are delivered securely, reliably, and swiftly. Some mistakes (e.g. double disbursement for the same transaction, hacked accounts, etc) could seriously jeopardize the well-being of our users and/or the company.

The Principles

With the context above, these are the principles that we have converged to:

• Velocity + Simplicity: We are a small startup and our ability to execute is essential to compete with the incumbents. Thus, it is important to continuously optimize our development velocity whether that is in the form of code refactoring, building tools or revisiting the velocity of our existing technical frameworks. A key element of development velocity is the simplicity of our technical stack. We should not add a new technical framework to our stack unless there is justification for this added complexity.
• First-Principle Thinking: Countless new technologies are created every year and most turn out to be fads; from microkernels to 1990s’-style neural nets as noted by Slack. In order for us to be able to distinguish revolutionary technologies from fads, first-principle thinking is essential. A first-principle is a foundational proposition or assumption that stands alone and can not be deduced from any other proposition or assumption. There are a number of great frameworks in software engineering that have been written to apply first-principle thinking on; from making a build-vs-buy analysis, modeling engineering problems as systems, to anti-principles (e.g. Shiny New Toy Syndrome, Not Invented Here Syndrome, etc) that we should just avoid.
• Alignment: Anyone is free to experiment with and propose new technology at OY!. However, they are responsible for getting buy-ins from other stakeholders in the company. Fortunately, an advantage of having a small team size means that it is not that challenging to find and gather the stakeholders in a (virtual) room to discuss the proposal.

Although we converged on these principles, we always applied them with great caution as they come with tradeoffs. For instance, if everyone always try to get alignment from everyone else at OY!, we would be in meetings all day and our velocity would halt to a crawl!

To ground these abstract principles in something more concrete, let’s look at some of the technical areas we optimized thus far.

Backend Programming Language

When OY! first started, there is no guidance around what languages we should use in our backend; there were 6 backend engineers and we ended with 4 programming languages: Erlang, Python, Java and Go 🙃

After the business pivot, we evaluated more closely what should our ideal backend programming language(s) be. Firstly, it should have strong typing and rich OOP features. Correctness and cleanliness is critical for a product that move high volume of money and we believe strong typing and rich OOP features helps enforce that. Secondly, it should have extensive library that allows easy integration with legacy systems and protocols such as SOAP and NDC. This is because we need to integrate with financial institutions which only have legacy protocols sometimes and we should not implement these protocol libraries ourselves to optimize our development velocity. Finally, there should be a deep talent pool at all levels of experience that are familiar with the language. This is because we are already operating at scale and we sometimes need to hire more experienced engineers to scale our systems even further.

We finally decided to use only Java since it ticks all the boxes. All of the other languages that we used do not have libraries as extensive as Java. Availability of talents in Erlang or Go is sparse especially at an expert level. Neither Python nor Go has strong typing or OOP features at a level that we expect.

We acknowledged that our choice is rather unusual for a startup and that there are downsides to using Java, e.g. it is relatively more verbose than other languages, slower than natively compiled languages, etc but we think the pros listed above outweigh its cons.

Since then, we have removed Erlang and Go from our technical stack, centralized all of our Java code in a monorepo and we are currently in the middle of migrating our last Python micro-services to Java. This highly contributes to our simplicity and development velocity.

Mobile Development Framework

Since the inception of OY!, we have been developing our mobile app natively. However, as we developed more features, it became increasingly challenging to maintain a high quality Android and iOS mobile app and enforcing feature parity between these two platforms given our small team size.

In late 2019, a couple of our engineers proposed to explore both Flutter and React Native. After exploration, we found that our development velocity would effectively double by migrating to either framework since there is more than 90% code-sharing between the two mobile platforms. This also increases the simplicity of our technical stack. However, between Flutter and React Native, our engineers find it easier to learn Flutter and that there is less performance degradation and increase in app size.

After carefully evaluating the tradeoffs, we came into the conclusion that Flutter is the most ideal framework for our needs. We then needed to convince our stakeholders (i.e. other mobile engineers, designers and product managers) to carry out the migration. This is challenging because we chose full migration instead of partial migration; this means no new features will be rolled out to our users until the migration is finished. However, since we are able to explain concretely how much higher our engineering velocity is and that we agreed to implement new features along with this migration, we are able to get the buy-ins from our stakeholders.

For further reading, we documented our journey towards migrating to Flutter here.

Code Repository Structure

For our last example, let’s look at our code repository setup. In the beginning, we started with a multi-repo setup in which each micro-service has its own repository. However, as we onboarded more engineers, we observed a number of problems due to our multi-repo setup that slowed down our velocity.

The first problem is consistency. Different people defined own codebase structure differently, used different libraries than other repositories, and/or used different versions of libraries than other repositories. This makes it hard for people other than the authors to read through and iterate on the codebase. The second problem is the challenge of making changes across multiple services. Code change across multiple services can not be considered and reviewed as a single atomic unit. It will need to be spread across pull requests in different repositories. Building a common tool or abstraction for multiple services (e.g. code coverage tool, linter, unit test abstraction) will be a challenge too.

Because of this, we began to evaluate a monorepo setup. Some of the most technologically-advanced companies such as Google and Facebook have found successes with it. We found that it solves the aforementioned problems too. It is easier to enforce consistency since all of our codebase is centralized and that people could open pull requests that modify multiple services at once. Furthermore, whenever someone created a new service we could ensure that the new service leverages common tools or abstractions that we already built.

However, there are several drawbacks to a monorepo setup. The first drawback is increased rigidness of our codebase. We would be more constrained in terms of the language and frameworks that we could use. Considering that one of our principles is simplicity, this drawback helps to enforce our principles. The second drawback is increased complexity in our toolchains. We would need to invest in our toolchains (e.g. deployment tooling, linter, etc) to ensure that it works well with the monorepo. We considered this drawback to be a one-time cost that we were willing to make. Given these tradeoffs, we decided to move forward with migrating our codebase to a monorepo setup.

The Future

The above principles have shaped the technologies in our current technical stack and the ways we make big technical changes. We do not consider these principles to be permanent but, in the model of Hegelian dialectic, something that continuously evolves, especially if there are changes in the context our engineering organization operates at.

We hope our post on the technical strategy principles provide sufficient insight into how we approach big technical changes and that it contributes to the wider engineering community.

If you find these principles to resonate with you, join us!

Acknowledgements

(In alphabetical order) Many thanks to Aldo Adiyasa, David Chou, Dandi Diputra, Herbert Bintoroe, Nikolaus Indra, Nino Aquinas, Ozzie Osman, Roberto Fernandez and Veni Johanna for reviewing early drafts of this.