Speed up DevOps with the Humanitec Internal Developer Platform (IDP)
We worked for quite a while without a DevOps engineer and split DevOps tasks between individual team members. We were going through migration of our cloud provider (from Hetzner — a small German hosting provider — to GCP) as well as some fundamental refactoring. This process forced us to set up our development and staging environments once again and revise our processes as a team. The decision to onboard to the Humanitec IDP was taken at the right time as it helped us not only to set everything up faster but also to structure our workflows better from the get-go. We can now better focus on feature development as a team as we significantly reduced the DevOps tasks including managing deployments and provisioning services such as databases.
Workflow #1 Cloning of environments in Kubernetes
Engineer perspective
Crafting an environment for a long-term project is often a complex task. You have to think about how your architecture is hosted, set up your continuous integration (CI) pipelines, as well as consider logs, monitoring, and security. All of this has to be scalable, maintainable and accessible to multiple stakeholders.
Furthermore, my team is focussed on creating and writing code and we want to deploy fast and often to have his/her work approved by the quality assurance (QA). But deployment scripts are unstable and we cannot rely too much on DevOps support in the development operations. So how could we get this done elegantly?
Real-world example in a cloud
We are a small team of three FE and two BE-engineers working with a microservice architecture, in BE for FE pattern. We are using an API gateway, which proxies the user’s requests to two different frontends, one for desktop and one for mobile, and six different microservices in the backend. Having all those factors in mind, the decision to operate in a cloud was made early on.
Our Docker Containers allow our applications to run consistently in the cloud and all operating systems. On Kubernetes, these containers can scale as pods. But the learning curve was very steep. When you are new to Kubernetes, it takes a lot of effort to get into the concepts of nodes, clusters, load balancers, ingresses, pods, deployments, services, volumes, storages, databases, secrets, config maps, and namespaces. Without the Humanitec IDP, we would have taken weeks or months to have the first development environment deployed. And afterwards, there are still two other environments waiting to be set up, one as staging or pre-production for the QA and product owner and one for the live production usage. It would have taken days again to clone and set-up all configurations in a secure isolated way without interdependencies.
Product manager perspective
Often, development teams start with one environment which eventually evolves into two and three over a course of six months to a year. This is due to the complexity and time efforts required for each environment (could be ±1 week per environment).
Also with each additional environment, the team needs to adjust processes and their day-to-day development workflows. Not having those environments often results in buggy software. Teams do not know where to test their code and split responsibilities across stakeholders.
With Humanitec’s IDP we were able to easily set up all environments at the initial stage of development with just a few clicks. This means, that on the first day we already had:
1. Developer environment, where developers test basic functionalities and integration of their code and run unit tests.
2. Staging internal, where the product manager, QA, and UX can perform user acceptance testing.
3. Staging external, where stakeholders can check against the business or market-fit requirements.
4. Production, for end-users.
Having this structure helped us to focus on developing quality code from the get-go.
Workflow #2 Management of microservices
Engineer’s perspective
We are using GitHub as the version control system, which offers a well-predefined workflow for updating and versioning. We use GitHub webhooks to run tests that must complete successfully before a pull request can be merged and on a new tag. Creating a new tag triggers the creation of a new docker image. In this case, Humanitec’s IDP allows us to easily integrate it’s CI/CD pipeline and helps to keep our test coverage high. Without Humanitec’s IDP we would have needed to come up with our own strategy, manage all the credentials for accessing the git repos and docker registries. Additionally, we do not have to take care of a private instance of a CD Platform like Jenkins, Travis or CircleCI.
Reusability and internal libraries
Our gateway and some of our microservices are shared among different projects. That way we can reuse our code like authentication, permissions or messaging without copy & pasting it. If we need to make changes for the project we create a fork through GitHub and modify this project’s version. In case we perceive our change as reusable, we push back to the upstream version and create a pull request. The other way around we are able to pull updates. This gives us more touchpoints with other projects and lets us contribute to our internal library on the go.
Product manager’s perspective
Humanitec IDP provides a good overview and easy updates of services that can be accessed by all team members. Without Humanitec IDP our team members had to manually check the status of services and this purely relied on the BE developer. With Humanitec IDP anyone in the team can easily review and update the versions of all microservices on each environment. This has allowed us to streamline the development process and enable faster bug fixing.
Workflow #3 Monitoring high frequent deployments
Engineer’s perspective
Humanitec IDP really supported us when developing our microservice architecture. In one page we can see an overview of all currently deployed versions in one environment, identify invalid images and be notified on a high-level about errors on deployments. We can choose the available versions to update or rollback in case we had errors. If we want to dive deeper, we can take a look into the container logs with two clicks.
Without Humanitec IDP we would have to authenticate and browse through at least four portals or type CLI-commands to get this condensed information from Kubernetes, GitHub, the CD platform and the docker registry. With Humanitec’s platform: no need to manually check if the image is available. No need for memorizing version numbers while browsing through the pre-production environment. It’s all on one page.
This simplicity reduces the human error rate to nearly zero and enables us to effortlessly deploy small changes at high speed. The platform hides the noise. Just click and done.
Prospects
It would be nice to see an excerpt of the git tag annotation bringing some more light into the version number. We are also curious about the upcoming improvements like real-time CPU and memory load to observe the availability in high traffic peaks.
Product manager’s perspective
Teams that are just kicking-off their development usually don’t have the time and capacity to set a full infrastructure for monitoring, therefore the downtime is relatively high. Additionally, the teams are not big enough to have a full-time equivalent (FTE) to keep track of the occurring issues around the clock. Humanitec IDP provides out of the box monitoring which is also easily accessible to any team member — from PM to QA or Developer.
For our team, working in microservices architecture this was extremely useful as we were able to track all our services in one place and have it as a shared team responsibility without allocating any FTEs for DevOps.
Having a platform that hides the complexities of Kubernetes from developers and is able to supply support on-demand allows us to provide a more reliable application to our end users.
Summary
The Humanitec IDP helped us to abstract away the low-level configuration and enabled our team to focus on engineering. We didn’t introduce any new tools into our code stack as we could easily integrate the platform into existing infrastructure. We have significantly speeded up our DevOps and enabled our team to easily spin up new feature environments on-demand.
Written in collaboration with Domile Janenaite, Product Manager at Humanitec and originally published at https://humanitec.com.
