Why we bet on Scala at SwissBorg

Scala has been with us since the company’s inception, and we are really satisfied with how it served us in all these years. In this article, we want to share our journey with Scala, highlighting the challenges we encountered, the solutions we crafted, and debunking some common myths about Scala based on our experiences.

2017: The Year of Bootstrap

As SwissBorg was taking its first steps, our CTO, Nicolas Remond, a former Gatling contributor, had to make the decision of selecting a programming language for this new venture. This decision was guided by several critical questions:

Can We Get An Advantage?

It’s important for any startup to Beat The Average. Scala was SwissBorg’s way to do it. By recruiting talented people from EPFL, Scala’s birthplace, we were able to rapidly bootstrap the system leveraging the full potential of Scala expressiveness and its ecosystem. At this early stage, with a lean team and a small system, the language’s complexity was manageable. We were able to invest in skilled professionals who could navigate Scala’s intricacies effectively, yielding a high return on investment.

How Are We Going To Scale?

During the initial phase, it’s important to plan for growth after the market validation. In the early days, a lot of software needed to be built fast. This was not a problem because of a few compounding factors. First, we had good luck in finding our founding engineers (Lomig and Arnaud, who are still with us after all those years!) through the internal network. Then there were (and are) plenty of Scala-specialized consulting companies that are happy to help businesses grow. We entered into partnerships with some such companies, including SoftwareMill with whom we are still partnering till this day.

On the technical front, Akka and its event-sourcing support enabled us to create system components with virtually limitless scalability from day one. These components have supported our growth by several magnitudes and remain a foundational part of our infrastructure.

What If a Bug Happens?

In fintech, a single bug can significantly impact the company’s future. Having experience with both Python and Scala, Nicolas could weigh the trade-offs between different software development approaches. We chose Scala for its strictness, minimizing the risk of technical failures that could threaten the company’s stability.

Security was another critical consideration. A single cybersecurity breach could have devastating consequences. The Java and JVM ecosystem are very scrupulously monitored by various parties, and this aspect is taken very seriously there, which further solidified our choice of Scala.

2020 & 2021: The Years of Growth

Over the next three years, we continued to expand our team and develop the system powering our product. By early 2020, we had approximately 30 engineers and over 500,000 lines of Scala code. Faced with a significant market opportunity, we needed to double our engineering team. This expansion brought several new challenges.

Can We Hire Enough?

Hiring is one of the most commonly mentioned issues of Scala. Claims range from the costliness of developers to a scarcity in the market. We put these claims to the test and found them not to hold true for us. Our lack of problems in hiring may have been aided by our remote-first approach, allowing us to recruit globally, or our positive work culture, which encouraged referrals. As a result, we were able to meet our hiring goals during these years without any issues.

Can We Maintain the Standards and Culture?

SwissBorg has embraced an engineering culture centered on technical excellence and correctness. With an inflow of new engineers, preserving and evolving this culture was a potential concern.

Scala is a very flexible language. You can write Python in Scala, Java in Scala or Haskell in Scala. This might sound like something that will make it even harder to ensure mindset coherence. But it’s not! Turns out that below the surface the Scala community, or at least the subset that applied to our job postings, is rather aligned. Most people agree on basic principles of functional programming and prioritizing correctness. This mindset aligned perfectly with our approach, and we didn’t have issues with regard to maintaining the culture we had before the growth.

Can We Onboard Junior Engineers to Our Codebase?

Another often mentioned accusation toward Scala and Functional Programming is that they are complicated. Because we hired across the whole seniority range, including interns and junior engineers, we were able to validate that claim. Turned out that we didn’t have any noticeable issues from that perspective.

Our hiring bar for entry-level Scala positions is typically at the level of the “Functional Programming in Scala” course by EPFL or “Programming in Scala” book by M. Odersky. This means that we expect this level of knowledge from candidates, and it also means that we might have a slight selection bias here. But among all the people that were able to get the theoretical basics and had no or very minimal commercial experience, none showed issues during onboarding and contributing to our codebase.

It’s worth noting though that we try to maintain a strong culture of collaboration and we had a lot of seniors around, so all the juniors had someone to help them and explain things when needed.

Can We Support the Community?

Benefiting greatly from the contributions of individual developers, we felt compelled to contribute back to the community. In the beginning, we open-sourced our akka-persistence-postgres plugin, and after some time we became Scala Center partner, launched our Technical Blog and Twitter profile. Despite modest beginnings, we would love to increase our support in the future.

2022: The Year of Consolidation

Following two major black-swan events in 2022, the Luna crash and FTX collapse, the cryptocurrency sector entered a bear market, pushing us to adopt a more conservative financial strategy. How did our choice of Scala influence our response during these turbulent times?

Is Our System Costly?

The first area we examined for cost reduction was our infrastructure. Operating our Scala-based system on the JVM, which is sometimes perceived as a resource-intensive platform, was an opportunity for evaluation. Our analysis revealed minimal benefits from switching to supposedly more efficient languages like Rust or Go, primarily because:

• Memory costs are relatively low
• CPU costs are higher
• The JVM offers excellent CPU performance.

Ultimately, we achieved some savings by reducing the number of replicas, leveraging our system’s horizontal scalability. However, further optimisations would have offered a negligible financial impact on the overall company budget.

Do We Have Redundancies in The Team?

Salaries represent a significant expense on our balance sheets. During these challenging times, it was necessary to critically assess our team’s composition. Despite the company reducing its overall staff by about 15%, our engineering team, particularly the backend part, remained mostly unaffected. This decision confirms our belief that Scala developers, while costly in absolute terms, provide value that far exceeds their expense. This was validated when we needed money the most.

Can We Continue To Build The System With Limited Resources?

The bear market is the time to build, so that when the bull market comes back, we are prepared with a stable system and new enticing features. This period required us to maintain and expand our system without increasing team size. Remarkably, our system demanded minimal upkeep, allowing us to focus on adding new functionalities efficiently and cost-effectively. It’s hard to assess what was the primary contributor to that characteristic, but definitely all the major building blocks of our system (PostgreSQL, Kafka, and Scala with its whole ecosystem) were rock solid.

2024: The Year of Perspectives

Right now we are entering the much anticipated bull market, and this is always an important moment for any crypto company. What does it mean for our system built on top of Scala?

Can We Still Meet Hiring Goals?

We started the year by opening six Scala positions, even before the bull market began. Now it is likely that we will increase that number throughout the year, and so we will have another opportunity to verify the state of the Scala hiring market. Hence, we double down on our efforts to be an enticing workplace.

Can We Migrate Everything?

We face two important migrations: Akka to Pekko and Scala 2 to Scala 3. The Pekko one looks promising because the team really tries to accommodate the process, e.g. through the rolling cluster update or similar improvements. As for Scala 3, we began preparations following its first stable release and recently cleared the last of our obstacles. The migration process is planned to start shortly, with optimism strengthened by detailed guidelines and general buy-in from the community. We anticipate leveraging all the Scala 3’s great features within a few months from now.

Can We Continue To Scale Technically?

​​With the bull market comes an expected increase in user numbers and their activity, posing scalability and stability challenges that have already impacted some of the largest entities in our industry. We are facing the same challenges, but we believe that the architectural choices we made and the tech stack we embraced will allow us to significantly minimize the number of such issues.

Scala Myths Debunked

Let’s now revisit and summarize many popular beliefs about Scala that didn’t match our experience accumulated over all those years.

• Scala is complex. While Scala has the potential for complexity, our approach has mitigated any negative impacts. We focus a lot on the simplicity mindset, especially in the hiring process. We try to balance the team seniority, so there is always someone around to teach & guide less experienced colleagues. And we try to use a “reasonably functional” stack, based on IO, immutability, lack of exceptions, but e.g. with local-only monad transformers, little tagless final and no recursion schemes. All those things combined allow us to benefit from the complexity more than we pay for it.
• Scala is hard to hire for. Contrary to common perception, hiring Scala developers has been no more challenging than filling roles in other areas of our tech stack (e.g. QA, SRE, frontend or mobile). Factors such as our scale, commitment to Scala, strong referral culture, and the attractive aspects of our company may have played a role in this success. Despite the tech community’s mixed feelings towards the crypto industry, we’ve thrived and occasionally opened ad-hoc positions to secure exceptional talent.
• Scala is slow or expensive to run. Operating our system entirely in Scala has not led to prohibitive costs or scalability issues. Currently supporting over one million users efficiently, our infrastructure is poised to handle much more. Scala’s cost-effectiveness was validated during significant financial downturns.
• Scala tooling is bad. We never had any significant issues with either IntelliJ, sbt, scala-cli or scalafmt, all of which we use very heavily. On top of that, sbt-tpolecat transforms for us a compiler into a powerful linter, which we then complement with scalafix. This last one also allowed us to easily migrate our codebase on a couple of occasions. All those tools are not always perfect but never to the extent that would affect our ability to run the business.
• Scala is unstable. By actively monitoring the ecosystem and driving updates, our platform team is minimizing the impact of any instabilities. We successfully went through a couple of big cats-effect migrations, we are using the latest Akka version (from before the license change), we are in the middle of Pekko migration, and we successfully switched our whole codebase to Scala-3-compatible syntax. Neither of those migrations caused any significant issues nor pushbacks/concerns from the business. It’s also worth mentioning that Scala 3 comes with some very strong compatibility guarantees.
• Compilation is slow. Although the Scala compiler performs extensive work, resulting in perceptions of slowness, our largest projects compile in under five minutes on build servers, with incremental compilation speeding up the development cycle locally. Thus, while faster compilation would be very much welcome, it hasn’t notably hindered our operations.

Why Do We Love Scala?

After addressing common misconceptions, it’s time to celebrate Scala’s strengths and the unique advantages it offers to our team and projects:

• Upward scalability with Akka/Pekko cluster and persistence.
  Although our stack is more on the functional side, those two Akka modules power our event-sourcing and allow us to scale the most critical parts of our system.
• Downward scalability with scala-cli. Thanks to it, we managed to use Scala for our automation and tooling, increasing developer engagement and satisfaction.
• Strong community focus on correctness. The Scala ecosystem prioritizes doing things correctly, which provides us with libraries (e.g. cats, refined, circe, spire or ScalaCheck) that enhance code robustness and reliability.
• Ability to model complicated domains and problems. Scala is a very expressive language, and this allows us to precisely express our business domain and its requirements. Our system touches many inherently complicated domains (e.g. accounting, trading, payments, investment) with a multitude of invariants and intricacies. We strive to keep the code as close to those domains as possible, and Scala is a great help in this.
• High quality of developers. Among all the candidates we get, very few get rejected based on their technical abilities. For most hirings, we had the opportunity to select from a pool of highly competent candidates.
• Rich ecosystem of libraries. Over the years, we never had a problem with a missing library, and we always try to leverage libraries natively written in Scala for the best developer experience.
• Stable institutional backing. EPFL, Scala Center, VirtusLab and JetBrains are all institutions that commit (very) significant resources to different aspects of language development and assuring its stability. This stability is reinforced by software houses and companies deeply invested in Scala’s success, contributing through community events, content production, and open-source work. Even the strategic pivot by Scala’s original parent company (Lightbend) did not disrupt this situation.
• Community. The Scala community is not the biggest, but it’s definitely active, vibrant and well organized. It’s straightforward to get help or discuss issues, be it on Discourse, Discord, Reddit or in regional communities. There are also plenty of community events (even after COVID-19 killed some of them). And in recent years we have much less drama! On top of that, a recent license switch for Akka and the Pekko birth showcased the resilience of the community.
• Rock-solid JVM runtime and Java ecosystem. We benefit a lot from the continuous updates of the JVM, be it performance optimisations, security updates or new capabilities. We also benefit from the Java ecosystem’s focus on stability and reliability, especially in critical areas like cryptography, networking, monitoring or logging.

Where is SwissBorg Now?

After more than six years since its inception, SwissBorg has grown to 200 employees, with our engineering department being the largest at 65 engineers. Among these, over 35 developers use Scala daily across nine teams. They are entrusted with the development and oversight of more than 1.1 million lines of Scala code, spread across over 30 services. These services are instrumental in supporting over one million users and managing assets exceeding 1 billion dollars. Our commitment to Scala remains unwavering, and we have no intention of incorporating any other programming languages into our backend codebase.

Oh, and yes, we are hiring!