Say hello to Devboxes
Introduction
This article is a follow-up to “An Introduction to Developing Microservices on Kubernetes (at Pipedrive),” published in February 2021. In that article, we introduced you to Pipedrive’s local development environment in Kubernetes, a.k.a KWS. We have seen quite a few developments since. In this article, we review them and introduce you to Pipedrive’s next-generation development platform for microservices in Kubernetes, a.k.a the devboxes.
Local vs Remote
In the last article about the development environment at Pipedrive, we introduced a new platform for the local development environment in Kubernetes. Today, the solution can run on multiple platforms like macOS (Intel and ARM64), Linux and Windows+WSL. To run Kubernetes on top of Docker containers, you can use Docker Desktop (macOS and Windows), k3s, MicroK8s or Docker Engine on Linux.
As promised in the previous article, here’s a video that illustrates the usage of KWS.
While we were all excited about the local development environment on top of Kubernetes, it hasn’t been a smooth ride. We realized it more and more as the tool’s adoption increased. Some of our key pains were:
- Poor performance of underlying technology (mainly running Kubernetes on Docker Desktop for Mac)
- Resources (CPU and memory) constrained by the specs of a workstation, a pain we had anticipated, but not so soon down the road
- Slow startup of the environment while deploying many services or huge stacks (a group of services)
- Non-reproducible environment issues — certain issues would affect some developers and workstations, but not others
- Observability limitations — missing a clear understanding of what our developers are experiencing
- Maintainability — too many platforms to support, including Docker Desktop, k3s, MicroK8s, etc.
- Developers not upgrading KWS and complaining about issues that had been fixed in the newer releases
We mainly experienced issues with Docker Desktop for Mac. Back then — before the release of Apple M1 silicon — Docker Desktop for Mac saw exceptionally high CPU usage on Intel. Consequently, our Macs would ventilate whenever we used KWS. Following a round of feedback, we started thinking about solutions and alternatives. Just as a side note, I don’t mean to imply that KWS is insufficient in and of itself. It has served us extensively, just not for this particular cause.
One core skill of all competent developers is their ability to build excellent software fast. They want their toolbox to contain robust and reliable tools. When we built KWS in spring 2020, we toyed with the idea of creating a development environment in a remote Kubernetes cluster (a.k.a remote development environment). We prototyped it and released the proof of concept together with KWS. The setup was clunky but still worked, which got us acknowledgment and praise. Among the tools the market offered for remote development in Kubernetes at the time was Devspace. It was packed with features that we liked quite a lot and worked well, so we ended up choosing it. Kudos to Devspace creators and maintainers for doing a great job!
By 2021, we had already finished our migration from Docker Swarm to Kubernetes. Yet, at this point, KWS had gained popularity, and our views on the concept of a local development environment in Kubernetes started to change. The challenges started weighing on us, and we realized we needed to change our perception of the development environment. One of the biggest shifts from an organizational perspective was assigning a dedicated developer experience (DX) product manager. We wanted this person to own the product’s vision, future and challenges.
Luckily, we found a fantastic DX product manager who got straight to business by tackling the remote development environment in Kubernetes while basing it on our prototype. In September 2021, we started implementing the interface of the remote development environment in Kubernetes, a.k.a the devboxes. In November 2021, we released devboxes for alpha testers. Today, the platform is available to all Pipedrive engineers.
Implementation
The concept of devboxes was built on an existing concept: the testboxes. Testboxes are demo environments, which Pipedrive uses for internal, isolated testing without any distractions. We wanted to create a similar concept for developers. That way, we could run the services in the development mode by applying file sync and hot reload, enabling the debugger and setting up port forwarding.
Technically speaking, a devbox is just a namespace in a remote Kubernetes cluster. The namespace contains all the dependencies to deploy and run Pipedrive’s microservices. Here’s what the architecture of the devbox platform looks like:
The interface for the devbox platform is called Devbox CLI, which is written in Go, like Kubernetes and most of its tooling. The CLI integrates with kubelogin, Helm, Go client for Kubernetes and Devspace. All integrations are compiled into the CLI and aren’t used as standalone executables. The purpose is to ship a single binary that includes all necessary dependencies. That way, you can get started right away.
Here’s an example:
$ curl -s https://devbox-releases/install/current.bash | bash -
$ cd ~/git/my-service
$ devbox service developOne of the key requirements for the CLI is creating commands for the most frequently used developer workflows, including:
- Creating users and companies in the Pipedrive application
- Opening various URLs for tools and services
- Deploying and removing services and stacks (related groups of services)
The CLI is not only about core commands. It also offers commands for expediting some operations on the command line, rather than following through with the login flow and via a web browser.
Via the CLI you can also control the devbox lifecycle:
- devbox status — see the status of your devbox
- devbox new — get a new devbox
- devbox use — use a devbox that belongs to your colleague (sharing is caring)
- devbox pause — pause your devbox (scale everything to 0)
- devbox resume — resume your devbox (restore the scales before pausing)
These are the everyday commands that we use while working on the devboxes platform. In the following diagram, you can see how Devbox CLI interacts with different components and dependencies.
The devboxes platform also supports functional tests — a method of software testing where the service dependencies are simulated. All you need to do to deploy the service is run devbox functional develop or devbox functional deploy. Then, you can execute devbox functional run to run the functional tests against the service deployment.
One of the latest features we’ve added to the platform is devbox playbooks — a tiny framework that can automate repetitive tasks related to the development processes. This means anything from setting up a complex process for the company and user creation via the APIs to direct low-level database manipulations. The playbooks are written in TypeScript and run by a separate microservice, which is developed on the devboxes platform along with the playbooks. These playbooks can then be shared between different teams.
devbox playbook run [playbook] [play] [flags]Devbox CLI works on macOS (Intel and ARM64), Linux and Windows+WSL. With the new remote development environment, we’ve managed to unlock Apple M1 silicon-based Macs. Unfortunately, not all of our Docker images support ARM64 yet. Despite our attempts to find appropriate solutions, such as with QEMU, we’ve noticed the usage of KWS (the local development environment) on Apple M1 is impacted. With Devbox CLI, all you need is a local Docker instance — for building and pushing development images — coupled with a decent code editor and internet connection to deploy your services in a remote Kubernetes cluster. In turn, file updates trigger Devbox CLI to sync these files with the remote container. Hot reload would then trigger the service restart, and you’ll see changes in your service in a couple of seconds.
Demo
A video is worth a thousand pictures. Here is a little clip showing the basic usage of Devbox CLI.
Summary
Never stop experimenting and searching for the best solution. What works for one does not necessarily work for others. Pipedrive is almost 12 years old (founded on June 21, 2010) and has drastically changed its architecture over time. We started with a locally developed monolith service and shared development databases. Then, we started moving our architecture towards microservices and Vagrant-based devboxes, followed by Vagrant-based devboxes, in cooperation with Docker containers. We then moved away from Vagrant and towards building the development environment on top of Docker containers and Docker Desktop. After that, we moved the development environment closer to Kubernetes. We created KWS, were still unhappy and began switching to the remote development environment in Kubernetes, a.k.a the devboxes.
Our developer family likes the devboxes. It’s not uncommon to see someone praising the platform in Slack channels. It feels good and validates our decision. Having a dedicated person responsible for the developer experience makes a huge difference in our opinion.
It would have taken a lot of our time and resources if we had built a solution from scratch. With Devspace, we managed to accelerate the creation of our platform, for which we are grateful. Devspace is maintained by Loft Labs. This company offers other cool Kubernetes solutions like Virtual Kubernetes Clusters (vcluster). Check it out at https://loft.sh/.
Finally, I recommend that you listen to this DevOps Paradox podcast, where Pipedrive’s DX product manager, Diogo Correia, and I share our thoughts about the development environment in Kubernetes.
Stay tuned for more interesting content!
🚀🚀🚀
Signing off,
Ragnar Paide (proud DevOps engineer at Pipedrive)
Interested in working in Pipedrive?
We’re currently hiring for several different positions in several different countries/cities.
Take a look and see if something suits you
Positions include:
- Software Engineer in DevOps Tooling
- Backend, Full-Stack, iOS Engineers
- Junior Developers and Interns
- Infrastructure Engineer
- And several more…
