Project Blackbox — Our mysterious staging environment
It’s been six years since NoBroker has started, gone are the days where we have our applications running on VMs, there we spend most of our time fixing the dependency issues. As we grow in all directions we started to embrace containerisation.
With great power comes; great responsibility ( issues)🤙
This is the story — of our staging environment.
Staging environments are notoriously difficult to set up and maintain, staging environments are usually different from production environments, and because of that, are fraught with problems — from failing deployments to out-of-disk-space errors, and various other errors. Even when the staging environment is great, it has one problem — there’s only one. If you want to test a feature branch, you have to “allocate time”, or alternatively install the feature branch and risk disrupting other testers.
We had to set up a lot of site environments to overcome these issues that would act as a stage environment based on the project.
Problem 1: Dependencies
It varies from a project, code packages, and their versions. though docker solves most of this problem. still, we have to manage resource utilization, and exposing the application with endpoint URL ( https😯 ) takes a domain name mapping.
so we want to solve this problem, by giving the developers a single environment with all the dependencies packed into a box.
Each developer will get a unique box with their code running inside along with the dependencies they can choose to test their codes and APIs.
Problem 2: Accessing the logs
how can we give developers to access their logs? ssh? — No way
adding ssh keys of all the developers is not a good way to run the show.
To remove all these issues, we have set up a single environment that is code agnostic and reliable and easy to manage with less overhead.
A whole new idea came into our minds : if we could use swarm to change the way we saw environments for developers ! — Blackbox 😀
Come, let’s unwrap and see what’s inside our mystery box
Docker Swarm
Though over production workloads are running on Kubernetes, we have settled with docker swarm ( because “it just works”).
Installing and maintaining Kubernetes needs an extra bit of effort, on the other hand, managed Kubernetes providers ( like EKS, GKE, AKS) are pricey too.
we are currently using 1 master and 3 slaves cluster, it tends to grow as the workloads started increasing
Jenkins
Jenkins is an automation server that allows you, amongst many things, to create a continuous integration (CI) and continuous deployment (CD) pipelines. We will be using Jenkins and its pipelines to automatically test, build, and deploy our project.
It came in handy as our projects are language agnostic, so it’s an all in one tool for our build needs.
Portainer
Portainer is a nice tool/service that makes it extremely easy to maintain and inspect your docker environment through a web UI instead of doing it all through the terminal. Along with that it has some basic security and exposes an API which we will use to rebuild our Docker images and deploy our projects.
so our developers can quickly find details about their running containers. that solves the problem of force them into **ssh*ing onto the server and docker inspect through containers to extract a database IP, for instance.
Sonatype:
we wanted a trusted and private registry to store the artifacts that generate during the build process.
so, we have created a Docker Registry with Nexus Repository OSS where you can store, manage, and deploy Docker container images.
Traefik
Traefik is a fast reverse proxy ( thanks to lightweight Golang ) and load balancer which will redirect all incoming traffic to the appropriate docker container(s). It will watch for the docker daemon events know when the containers are started or stopped, and will automatically add or remove their DNS rewrites accordingly.
Additionally, the proxy supports automated handling of Let’s Encrypt certificates out of the box. This will allow our environments to run under https very easily.
How does it work?
Each stack has the working code with unique endpoint as the
- Developers push their code to the branch through PR or with a deploy tag.
- Jenkins, using multi-branch pipeline — scan the project for a new branch with the deploy tag and build the code from the branch. The build artifacts ( docker images ) will be stored in the private repository.
- Jenkins calls portainer API to deploy the stack ( dependencies packed inside it) along with a new build number as a parameter.
- Portainer → Docker daemon —this gives us the dashboard to manage deployments and viewing the logs.
- Traefik reads the “Labels” we attached to the docker services and create a subdomain with SSL enabled ( using Letsencrypt ).
so, every build which deployed into the black box comes out as https://URL
we managed to make this URL unique to the code and to test our application on a server, in almost real-time.
https://<branch_name>.<project_name>.blackbox.nobroker.in
Logs? — We have given role-based access to the Portainer dashboard to view the container logs.
we at NoBroker loves open source, in fact, all the tools we have used here in the making of Blackbox are open source.
Do you want to see the code? — 👩💻
DeepDive to the stacks? — Part 2 💥
