What do 50000 lines of code in a development environment do?

How do you accelerate revenue growth without hiring? You make it easier to develop new features.

In order to keep growing, Pipedrive puts significant effort into development environment research. To generate more revenue, a product needs both more robust offerings and new features. The ones that implement and architect these, are engineers. So, the more efficient your engineers are, the more your rate of growth will increase.

The semantics of development environments

Development environments are the software engineer’s setup. It is always easier to focus and get work done if yours is clean.

The efficiency of software development is largely influenced by the quality of development environments (dev envs) and the effectiveness of developer experience (dx) teams. Dev envs are the platforms where engineers create and deliver, while dx teams ensure that these platforms remain current. A poor environment can significantly hinder developer productivity so, by investing in an adequate solution, every engineer is empowered to reach their potential.

Why doesn’t every company have a dedicated dx team? Dev envs are hard. The concept itself is both broad and general since its semantics are loosely defined: an environment that allows for the development of something.

Looking at commercial offerings doesn’t help either, as there are very different projects that all call themselves “dev envs” or “dev platforms”: Github codespaces, JetBrains text editors and Qovery, to name a few. One is an on-demand VM, another is just a text editor and the last is a frontend to Terraform — with a bit of magic sprinkled on top.

Think of how different the flows of Pipedrive, a company with over 300 engineers, are from Google and your run-off-the-mill startup. Creating a product to suit the needs of this triad is a Herculean task: To provide a platform that can support every possible software development flow. Some of these end up as highly opinionated takes on the universe, therefore turning into micro-religions, such as text editors.

We propose to solve this problem, once and for all, by formalizing dev envs as the two-tier interface in which features are developed. The first tier is the set of highly arbitrary recommendations on how to set up one’s computer for “development,” such as which text editor (integrated development environment, or IDE), terminal emulator, or something else, to go for.

The second development environment tier is the software-specific setup, which is often a source of much-memed pain since it can get comically irreproducible, as developers forget that it too depends on the first tier, as well as just about everything else (including the position of Jupiter relative to Saturn).

From scripts to Devbox

At first things were quite primitive, with docker-workstation acting as a collection of scripts to spin up multiple docker containers. KWS came next — a significant evolution that abstracted away all scripts. Finally, we have Devbox, the contemporary take. In spite of there being an evolution in usability, they were all quite fancy and loaded with features.

Over years of development process experimentation at Pipedrive, we slowly went from microservice-specific scripts to attempting to get every developer to become part DevOps engineer by having their development environment be a local Kubernetes cluster, running almost three digits worth of containers. While this might seem like a not-so-lucid idea, it brought live and test environments closer and significant effort was put into standardizing all of our microservices, so that they could run in any cluster, be it live, or in a developer’s notebook.

A while later, the next and latest (and greatest) iteration of our dev env was born: Devbox. Building on the effort poured into our ahead-of-time-provisioned test environment, creatively named testbox, we attempted to solve everything that plagued its ancestor KWS — specifically, expecting developers to be part-time DevOps engineers and the unintended impact of first-tier problems.

Two years down the line, it has been perhaps the greatest success story of our internal development tools. The most relevant metric is time spent debugging dev env issues, which decreased by over 90% relative to the previous solution. Developers fight with their development environment for less than half an hour per week, on average.

This coup de grace was delivered by both eliminating first-tier problems altogether and by having a near-zero setup time. Whenever an engineer works, they do so by requesting a “devbox”, which is a pre-provisioned Kubernetes namespace that has the core microservices (out of 750 in total) needed for development. This is located off the developer’s computer and when requested is available immediately.

Given all devboxes are created equal, with the same hardware, software and everything else, it is not possible for “it runs on my machine but not on yours” to occur. Our developers say that this is pure bliss, all while being built in-house, costing ten times less than our live environments and giving developers enough confidence to successfully deploy to live thousands of times a month.

Focus

Context switching sucks. It takes on average 25 minutes to focus back after a distraction.

The key benefit that Devbox brings is focus. When there are no scripts, forgotten obscure env variables, docker, Kubernetes or anything left to fight, all that there’s left to do is to work and fuel our vision. While there are other similar-sounding offerings, as previously mentioned, we posit that in our case, none of them could offer better focus, as it is not possible to be as tightly coupled to our own core product and policies, being an external tool. These solutions are similar to Devbox’s ancestor and end up exposing developers to more variables than they need to.

To better demonstrate what Devbox is able to do, we’ll first show how it is used and then walk you through its 50000 lines of code.

The day of a Pipedrive engineer starts with them cd-ing into the microservice that they want to develop and then running devbox service develop on their terminal emulator of choice. We call the physical interface to one’s devbox, “Devbox.” It is a CLI.

A quick recap. The testbox environment contains, among other things, devboxes, which are pre-provisioned Kubernetes namespaces that have the core services necessary to develop any feature related to our main app, alongside multiple adjacent supporting resources.

After running the develop command, two things can happen: either they get assigned a fresh devbox, or if they already have one, it is unpaused, as devboxes can “go to sleep” after working hours. With one on hand, the developed service gets immediately deployed with file syncing, local port forwarding and hot reloading all configured. This is the very core functionality of Devbox. While it seems simple, take into account that this works seamlessly for all 750 microservices that we have, so there are a lot of edge cases that get caught under the hood.

The actual numbers

Around 20% of the codebase provides 80% of functionality.

The whole experience is made true with 55356 lines of Go code, out of which 30000 are either tests or mocks, amounting to 79.8% of code coverage. But just how many in total relate to core functionality (without tests or mocks)? That would be about 10500 lines of code.

Only something in the ballpark of 20% of the code delivers the most important functionality.

Less than 1% of the lines outline all logic that the service develop command follows. 2000 (5%) denote preflight checks that ensure developers have no local misconfigurations, such as a wrong service-level setup, or anything that could affect Devbox. The more assumptions about runtime there are, the less brittle it will be.

Most specifically, we run the following checks to ensure that the following first-tier problems cannot impact the second-tier experience:

• VPN — assures that the developer is under our VPN.
• Kubelogin — ensures that each developer can actually access their devbox’s cluster.
• Container Builder — asserts that there is a container builder available (a docker, in our case).
• VM Space — prevents the user from running any command that depends on a container builder, if there is no space to build new images.
• Shared Devbox — informs users whose devbox they are developing with. Yes, as long as developers have consent, they can use each other’s devboxes at will.
• Container Registry — ensures that developers are logged in to our private registries.
• Helm Template — updates the universal microservice helm template.
• Dockerignore — makes sure that developers have a dockerignore file to stop irrelevant or sensitive files from being synced to their container.

Configuration-related code equals 3% (1500). For a microservice to be supported by Devbox, it needs to have a set of files in its repository that will accurately describe its needs. 4000 (8%) other lines relate to important infrastructural minutiae, such as instrumentation, developer metrics, self-updating and the logic to “wake up” devboxes.

Sleep is important for devboxes as well. For the testbox environment to be cost-effective, we rely on cluster autoscaling. When a box has not been used for a while, it’s “paused” — i.e., all deployments are scaled to 0. Given that developers are actively programming for about 6 hours a day, this theoretically reduces the cost of infrastructure by 75%.

Activity is recorded by sending a heartbeat to the central testbox manager API with each command, alongside attempting to detect whether long-running commands are idling.

The latter half of the core contains source code that supports interactions with orchestrators, including Kubernetes. 1500 (3%) lines enable integration with testbox manager, while 600

(1.5%) are for custom logic related to Kubernetes. In spite of remotely developing on Kubernetes, developers have very little exposure to it and never have to directly interact with it. It is almost entirely transparent.

The remaining 2% are the APIs that communicate with product-related supporting services. Pipedrive is a sales CRM. Companies have people and people have accounts. A company can have multiple people. It often occurs that developers need to develop our web app from the standpoint of a specific user, or of a certain kind of company account. Or, perhaps they want to experiment with different settings. In the past, this used to happen via mostly informally-shared scripts or postman collections. These things are also part of the developer experience, so it should be exposed through Devbox.

The two subcommands that require communication with supporting product APIs are company, and user. These provide a comprehensive set of actions that go from mutating data, to opening pages and databases.

The remaining 80% of the source code is logic that belongs to the other 13 commands:

1. Company — company commands (create, delete, set, etc.).
2. destroy — destroys your current devbox.
3. functional — functional test commands (develop, run, open).
4. multidc — switches a devbox to “multidc” mode, a mode where the developer handles three devboxes at the same time, all in different regions.
5. new — gets a new devbox and destroys the current one.
6. open — open commands (shell, webapp, database, etc.).
7. pause — pauses the current devbox.
8. playbook — runs playbooks.
9. resume — resumes the currently paused devbox.
10. stack — commands to deploy multiple services at the same time that belong to some predefined group, or “stack”.
11. status — displays the active CLI’s devbox’s status.
12. use — switches the active CLI’s devbox to another developer’s devbox.
13. user — customer user commands (create, delete, set, etc.).

These commands range from running functional tests to executing “playbooks”, a novel system that allows for the ad-hoc enactment of “states” of some service, directly from one’s devbox. Yet again, it’s only convenient due to tight coupling.

Conclusion

In this blog post, we discussed the meaning of development and production environments, a term that ranges over a set of scripts, to bulky text editors, all the way to places where it shouldn’t have gone in the first place. We then recalled Pipedrive’s journey across multiple iterations of our in-house development environment, and how each was a strict improvement over its predecessor.

We introduced Devbox, the most recent take, with thorough examples of how it is used, what it can provide and, finally, what all of its 50000 lines of code do. Lastly, from the picture above, we see that since its inception, there’s been a steady decrease in the amount of hours wasted per developer.

As it can be seen from the graph, it has decreased by 70%.