Tech at Moss — The B2B Spend Management Platform
At Moss, we enable companies to master their finances with our complete spend management stack. We’re a FinTech startup from Berlin, and in just three years, we have built a team of over 500 people in 5 countries across Europe and the UK. We are just getting started on our mission to elevate the finance backbone of the SME economy.
I’m proud to have been part of this journey from the beginning, focusing on building and maturing the system that powers Moss. It’s a modern system that we built using best practices that help us evolve our product quickly while keeping things scalable and secure.
In this article, I wish to share some high-level details of our architecture and the principles that govern our day-to-day work at Moss to inspire, spark conversations and attract smart people to continue building ahead. So let’s get started!
System Architecture
Our early engineering team members were all FinTech veterans, who knew how to build and iterate quickly. As a joint decision, we chose to go with Microservices architecture from the start.
Most of the microservices are Java/SpringBoot apps with Postgres as the database. We consciously don’t use any ORM frameworks as they can often lead to expensive surprises & performance bottlenecks. Instead, we prefer staying in control by writing SQL queries with simple, battle-tested abstractions like JDBC Template or JOOQ.
The microservices are packaged as containers and run on Kubernetes which provides auto-scalability, blue-green deployment, and other benefits that come with it. Services communicate with each other over REST APIs or asynchronously by passing messages over Kafka.
Other languages/frameworks in the ecosystem include Typescript, Javascript, React, Golang in the Platform team & Python for data-engineering needs.
CI/CD
This is the most important part of any software system. It’s the circulatory system responsible for the reliable delivery of new features, incremental improvements, and bug fixes. Because of how critical this system is, we wanted to choose something that was easy to use, simple to configure, reliable, scalable and didn’t have management overhead.
There’s an old joke among Site Reliability Engineers that “Jenkins is the leading cause of alcoholism in the community”.
We didn’t want to manage our own CI/CD system because of previous bad experiences. After playing with a few options, we settled for GitHub Actions. It offers reusable and composable actions that run on GitHub (Azure) machines or self-hosted runners if needed.
We also continuously deploy at Moss. In a continuous delivery system, the generated artefacts (compiled packages, jars, docker images) are kept ready but manually deployed. Continuous deployment goes one step further and automatically deploys the change to all the environments — including production. This is often the biggest surprise for engineers who are joining Moss because they are not always used to it.
Although there are various benefits of deploying continuously that you can read about in different articles, we wanted to be careful before we implemented it. Adopting such practices requires trust in your automated tests, and confidence in your tooling to quickly detect and roll back in case of errors. Developers are expected to write tests for the code that they are working on and make sure that the main branch is always green.
We also measured the number of deployments and how comfortable the developers were in promoting the deployments to production immediately. Once we had the data, it clearly showed that it is best to remove the friction of manual deployment.
Deploying continuously has had an overall positive impact on our engineering culture and helped us avoid a false sense of security. Our engineers are available after their code change is merged to fix any errors quickly since they are the ones merging the change and they are in the context to verify a successful deployment. This leads us to have better practices like deploying smaller changes in increments rather than releasing many, big, risky changes together in batches. We also use feature-toggles extensively to roll out changes silently until they are ready for release.
Platform
We are happy Google Cloud customers. Kubernetes is a core system for us & since it’s a Google product, they know best how to run it.
We don’t try to fool ourselves that we can run complex systems better than the companies that offer them as a core product. So, where possible, we use best-in-class PAAS products instead of self-hosting them. This has helped us keep a very lean & simple Platform & even leaner SRE team.
We have integrated infrastructure-as-code into our DNA. Everything is managed through Terraform and lives in our version control. Living a true DevOps culture, engineers are encouraged to find their way and raise pull requests for required changes instead of coming with tickets. (As a regulated entity, we still need a ticket capturing the business context of the change).
On secret management — we understand that the best way to avoid secret management problems is to not have them. As such, we don’t use static credentials between our systems. For databases, we use dynamic credentials & for cloud resources, we use workload-identity. This helps us avoid the constant work of safe secret storage, sharing and rotation.
As we are building a product to automate the financial tasks of our customers, we ourselves are big fans of automation and use it to avoid toil as much as we can.
Pillars of our engineering culture
This article is more than just about using a set of tools and technologies. It’s about the type of engineering culture that a set of best practices can enable.
We are proud of the choices that we have made along our journey of building the Moss platform and that has led to our unique culture (I am of course biased, so I would encourage you to experience it yourself). Here are some of the principles that are helping us build a healthy engineering culture at Moss:
- Efficient secrets management. No static credentials for databases. Not every company that uses Vault does it to this extent. Nobody knows the admin password to our databases once they are configured in Vault and rotated. Since nobody knows them, we don’t need a process to rotate them. This has saved us countless hours of redundant work & helps us to clear our audits easily.
- Workload identity for authentication & authorisation. We don’t have to worry about securely injecting secrets like Service Account keys into our microservices to access the cloud resources. Workload Identity allows us to assign distinct, fine-grained identities and authorization for each application. CI/CD systems are often a prime target for attacks because they have secrets for authorisation to your critical environments. Again, workload identity helps us avoid having such secrets.
- No ClickOps! We don’t make changes to the system outside of Terraform. Engineers create PRs instead of creating tickets. You can’t just add Terraform to a company and expect this kind of culture. Having a pure infrastructure-as-code DNA gives us the benefits of peer review, revision control, audit, permission control and following the 4-eye principle for all of our changes.
- Continuous deployment. Again, this needs a mindset shift and it is a surprise for most of our new joiners. Our engineers deploy multiple times during the day with confidence. When there is an error, we can quickly rollback. As such, we score high when using the DORA metrics that are used to indicate the performance of teams.
- Zero trust architecture. This is something new that we are working on. More details to come up in the following article.
I’m proud of what we have built so far and we have a great team, full of energy and passion, continuously working to build more. We are just getting started and it still feels like day one.
Liked what you read so far? Let’s connect. We are hiring!
