How we developed our deployment model in Inventa (Part 1)
In the early days of Inventa's engineering, all deployments were made from the engineers’ own machines using AWS lambdas, especially because it was a fast way to get things to end users while also not having to worry about scale and infrastructure. Although this works fine at the beginning of a company, it does not scale the way we wanted, so one of our first challenges was how to migrate from this fully serverless model to a serverful-first approach. In this article, we'll show what we took into consideration and the decisions we made (as a complement, I suggest you read this other article in which we tell how we rebuilt our backend tech stack).
We can separate our deployment model into two phases: the pipeline and the release itself. Before we implemented our official deployment model, most of the deployments were made manually in the engineers' machines; this has multiple drawbacks, from security to a larger likelihood of errors, and our strategy was to automate the process as much as possible. So the first step was to understand how we would do this automation.
A few reasons why we should have a standard on how deployments were done:
- Risk mitigation since humans are error-prone
- Make rollbacks easier and deterministic
- Team communication and knowledge sharing since all teams use the same stack
- Reduced operational and software maintainability overhead
- Software delivery with fewer bugs and less risk
- Release new features to market more frequently — and learn faster from user feedback
By that time, we were starting to use GitHub as our code base tool and GitHub Actions were growing massively in the field of CI/CD pipelines. It made a lot of sense for us to have our pipelines in the same place our code is, reducing the cognitive load to follow deployments for example. Besides, GitHub Actions have a set of pre-implemented actions that help us do a lot of operations, from logging into AWS (our cloud provider) to recognizing which files changed in a PR (which is very useful for our Infrastructure as Code (IaC) strategy for example), that allows us to focus on the specifics of our pipelines instead of having to build a lot of basic operations from scratch. Therefore, we decided to use GitHub Actions as our CI/CD pipeline tool (we have a pretty neat CI/CD pipeline here at Inventa by the way).
Okay, now we have a place to write pipelines that will release applications — but where will they be released?
One thing was certain for us — we would go in the way of containerized applications, more specifically using Docker as our Platform as a Service (PaaS) solution. In a nutshell, using applications in the form of containers, we can abstract all the infrastructure and hardware involved in running these machines, which makes the deployment model much more reliable, flexible, and deterministic.
Therefore, all applications must define a Dockerfile that will generate its executable (a jar for our Kotlin applications for example), and the pipeline will call this Dockerfile in order to build the application image to be deployed. After the image is built, it must be stored somewhere so it can be deployed. As expected, there are quite a few options here as well, being the most used DockerHub and AWS ECR. We decided to go with the latter for being a native part of our cloud provider, making it easier to comply with our access management and infrastructure as code strategies.
Now the image of the application is stored and can be launched. But where it will be deployed? How will it be accessed? How will it use secret environment variables? This is where things become very complex and there are a lot of different approaches. We considered two tools to be our container orchestrator: AWS ECS (EC2 Container Service), and Kubernetes.
Both of them provide a secure and reliable container orchestration solution, but we decided to use Kubernetes for the following reasons:
- Cloud-agnostic, so if we ever need to deploy our applications on GCP or Azure or any other cloud for any reason, we will not have to build the deployment model from scratch
- Kubernetes is much more flexible in terms of configuration than ECS, so we could configure our cluster to better suit our needs
- Given ECS is a much more self-managed tool, it does a lot of stuff under the hood, so troubleshooting is harder when things do not work the way we expected — ECS does not give access to cluster nodes (the EC2 machines in which the cluster is running) and this makes observability and monitoring much harder.
- Kubernetes has been battle-tested hard in previous years, as being the leading solution in container orchestration. This also results in multiple forums and content related to the platform, making the learning phase less painful and helping in troubleshooting.
Ok, so now we have chosen Kubernetes. Decision time is over, right?
Not at all. Choosing Kubernetes opened a vast spectrum of how we were going to set the foundations of our cloud architecture — will we use a self-hosted cluster or choose a solution from an established cloud provider?
Considering that at the time we had a team composed of a single platform engineer, it made all the sense to go with a Kubernetes solution offered by a cloud provider. Given we use AWS as our major cloud provider, we decided to go with EKS to manage our clusters in all environments. This would immensely reduce the engineering overhead to implement and manage the application cluster.
Here at Inventa, we don't reinvent the wheel: if there is no aggregate value in building something, we use a SaaS that does this for us and focus our efforts on initiatives that differentiate us as a company.
Last but not least, we faced a final decision: Amazon EKS comes in two flavors:
- AWS Fargate mode: in this mode, you don't have to provision the cluster machines yourself — it works as a serverless environment in which AWS controls the number of machines in the cluster and the resources (CPU, memory) of these machines, that is, both vertical and horizontal scaling.
- EC2 mode: in this option, you control the machines (EC2) of the cluster, defining how many machines there will be, which is their family and configurations, and also the autoscaling groups to guarantee the horizontal scaling of the cluster nodes.
The image below illustrates these options:
Needless to say, Fargate mode is much simpler to deploy and manage, given its serverless paradigm, and that is why we chose it at first.
But we regretted this decision shortly after since like everything that is serverless running as a black box, you lose control over what is happening and the troubleshooting is much harder. Especially for a young tech company such as Inventa, which was provisioning this kind of infrastructure for the first time, it was important to have better troubleshooting over the infra, even if this meant having a larger management overhead.
Another important point that made us question the serverless cluster was that using Kubernetes add-ons was much harder in this case. For example, in order to provision our observability infrastructure (theme for another future article), we had to provision a stateful set on Kubernetes, that required a file system (EFS) and this was nearly impossible to do with AWS Fargate. Adding functionalities to the cluster is critical for us that are aiming for continuous improvement of the platform, hence this was a deal breaker for us.
So we reviewed this decision and rebuilt the cluster using the EC2 mode. Currently, our production cluster contains more than 30 different microservices and supports most of our product features. In order to guarantee the horizontal scaling of the nodes, we have a cluster autoscaler running on it, that continuously checks the resources of the nodes and evaluates if it should provision another machine or not. This guarantees our cluster is able to keep itself healthy and useful automatically.
But that's not all foks!!! This is the part 1 of our deployment model article. In part 2, we will cover in details the architecture we are using inside Kubernetes (ingress, service, deployment and all that stuff…).
Conclusion
In this article, we explained all the steps we passed through in order to go from a completely serverless and chaotic environment to define a standard cloud architecture and deployment model using a serverful approach so engineering teams could easily implement, test, and deploy their features and generate value for our users. In a nutshell, we have (as in the image below):
- GitHub actions as our CI/CD tool
- AWS ECR as our container image registry
- AWS EKS as our Kubernetes cluster provider, running on EC2 mode
This infrastructure has served us very well in the last 6 months and we have already done a lot of improvements to it, such as adding a cluster backup strategy, implementing the cluster via Terraform, building a robust deployment pipeline, etc. So stay tuned in our next articles because there will be a lot of content about them in the days to come :)
Did you like the challenge? How about joining us!? We are looking for software engineers who enjoy the state-of-the-art platform and application development, and want to help build a very reliable, observable, and scalable product in Inventa! Hit here to apply and, if you have any questions, don’t hesitate to call me on LinkedIn.
