Testbox: On-Demand Sandbox Environment For Developers
When I joined Pipedrive, I found a mature and robust platform for developers called Testbox. This caught my attention from the first moment I used it because of its excellent capabilities and potential. The more I understood its internal workings, the more I wanted to contribute to making it even better for our DevOps team and our engineers at Pipedrive. After participating in more than one opportunity to evolve the capabilities of Testboxes, I would like to share more about them, what lies beneath the surface, the advantages and disadvantages of this tool, and, ultimately, how developers can get the most out of Testboxes in their work.
I. Pipedrive Tech Stack (Status quo)
Pipedrive’s tech stack is in constant flux. The codebase, initially written in a monolithic PHP, has grown and evolved significantly over the past few years. Later, when Pipedrive adopted a microservices architecture, our development teams had to set up local environments on their machines using Docker Compose. This solution worked in the short term, but it became increasingly difficult to maintain it and keep up with the growing demands of resources on their machines.
The initially established development/testing solution that grew in adoption in Pipedrive was a workflow called “colorboxes” (formerly attributed to environments assigned to color-coded teams). It consisted of a remote, shared environment that developers could access for testing features, staging new features, and sharing changes amongst teams. However, this brought challenges as the company continued to grow. For example, testing a new feature sometimes broke the entire environment and compromised its availability for other developers. And oftentimes, automated Jenkins jobs that recreated or reprovisioned those machines had to be used in order to cope. It became a real challenge as Pipedrive expanded its capabilities. In particular:
- Recreating a test environment was complex and laborious. Sometimes, virtual machines became outdated within just a few minutes of their creation. As a result, we were burdened with the added task of regularly updating the virtual machines.
- Each team was required to manually update and fix the services. This reduced the efficiency of our engineering teams as updating the services required a significant amount of time and labor. The resources were underutilized and often wasted. Why? Because, in many cases, machines were sitting idle, not doing any processing. Also, the more teams we had, the more virtual machines we were maintaining, further reducing the overall efficiency.
When we migrated our stack to Kubernetes, the need for a scalable and easy-to-use platform for developers became apparent. We realized our engineers would require a flexible and more reliable sandbox environment to match Pipedrive’s development requirements. Similarly, we needed a platform that would allow the accelerated delivery of features/bug fixes to our end customers.
Essentially, our developers should be able to build, test, and deploy software not only on their local machines but also in controlled, production-like, virtual environments. This is where Testbox comes in.
II. What is Testbox?
As Pipedrive grows, its codebase becomes more complex to set up and manage. To this day, we have more than 1000 repositories developed by 300+ engineers across five locations. Our CI/CD system handles around 550 deployments per working day, roughly 15,000 deployments a month. Owing to this number, we offer developers two ways to develop their code: locally and remotely. Local development is convenient for services that require few dependencies. However, for larger projects, remote development environments offer significant advantages. Testbox was built to handle those requirements, and more.
Testbox is a fully automated, on-demand sandbox environment that enables developers to create on-demand, isolated, secure, ready-to-code development environments in the cloud. With Testbox, developers can focus on writing their code instead of trying to create a working environment to build and run the code. Each Testbox is configured to mirror production and provide developers with the most up-to-date version of the dependency services and database migrations.
Testboxes are pre-configured with all the services and settings developers need for their projects and tasks. And developers can obtain their Testbox whenever they need to test a feature branch quickly, fix a bug, experiment on a proof-of-concept, or just showcase a particular scenario without worrying about compatibility issues due to applications running in the background.
As mentioned, a Testbox can be used to build, test, and deploy changes to any service, on demand. To ensure everything runs smoothly in the software delivery process, we keep the number of boxes equal to the number of developers, thus providing an instance to each individual Pipedrive engineer. We maintain a ready-to-use pool of TestBoxes each day and adjust according to the usage metrics we gather. These environments can be customized to the last bit to match individual developer use cases. For example, developers can test performance improvements, push a UI change, enable remote development with hot-reloading, and run a database migration. Likewise, our DevOps team uses Testbox to develop new features for the platform itself. This and many other use cases are possible with Testbox.
By default, a Testbox lives in the pool for a short while until a developer requests it, or until it gets recycled and recreated. A developer can acquire a Testbox immediately once the need arises. When they do, they are asked if they want to configure it by choosing the service dependencies they want to deploy, and they can also set the lifetime (TTL) of the new environment, which defaults to four hours unless otherwise specified. When a developer needs more time to test their changes, they can easily extend the lifespan of their Testbox at any point before its automatic destruction. In the same way, they can manually initiate the deletion process once their work is completed. If no Testboxes are used from the pool by the end of the day, they will continue their lifecycle and be destroyed. Likewise, a Testbox has an additional TTL of two hours that expires when no network activity is detected. When this happens, Testbox enters a “pausing” state and deployments are scaled down to zero, preserving all the configurations and customization developers have made. If they want to continue using their Testbox, they can manually trigger it to resume its cycle.The following figure shows, at a glance, how developers interact with their Testbox(es).
III. What lies under the hood?
A Testbox is a copy of our production environment that lives on Kubernetes clusters. This is possible using Amazon’s Elastic Kubernetes Service (EKS), to be precise. Each instance hosts its own Pipedrive domain and subdomains for internal services. A Testbox is, in reality, just a Kubernetes namespace that allows the desired level of isolation of the network and resources for each developer. With each EKS cluster, we can hold a certain number of Testboxes (namespaces), depending on its capacity, but we can also scale up with ease. The Testboxes are orchestrated from beginning to end by an application service called Testbox Manager. This manager interacts with the Testbox clusters via Jenkins jobs for each step of its lifecycle: provisioning, deletion, pausing, resuming, etc.
Provisioning
The first step in the lifecycle of a Testbox is to create and configure the sandbox itself. This is also known as the “Provisioning” step. To provision a new Testbox, the Manager deploys a stack of “core” services crucial for our microservices. This list includes Kubernetes Deployments and StatefulSets with infrastructure services like Consul, MySQL, Rabbit, Elastic, Kafka, and others. Once completed, a second stack of microservices are deployed to assist Pipedrive’s main features. We constantly optimize the core stack of our Testboxes to avoid deploying many services to all Testboxes because this prevents them from overloading and cuts down on unnecessary costs. At this stage, we aim to provide developers with Testboxes that are bundled with all the essential requirements. At the same time, developers are free to add any necessary services or stacks that they need.
Customization And Utilization
Once developers are assigned a Testbox, they are free to add more features to match their requirements. All they need to do is deploy the required services together with the necessary dependencies for that specific functionality of Pipedrive, and that’s it; they have their Testbox ready and functional. This process of adding services to a Testbox is called “customization”. Fortunately, we have all that work automated, and developers need only choose a predefined template with services or choose a stack that belongs to a particular feature that requires services to exist.
Pause and Resume
To cut down infrastructure costs, we introduced a feature that puts a Testbox into a “pausing” state. In this state, all the Kubernetes Deployments and StatefulSets of a Testbox namespace are scaled down to zero. All the pods are removed, but we keep a snapshot of the replica state in a separate ConfigMap. This technique allows our Autoscaler to reduce the number of nodes in our AWS EKS clusters, reducing infrastructure costs.
The “Resuming” process reads the replica state from the ConfigMap and scales back those resources to match the previous Testbox configuration. Resuming takes an average of four minutes to reach full completion.
A developer can trigger the Pausing and Resuming transition manually. Likewise, a Testbox enters into the Pausing state when no network activity is found within a two-hour interval.
Deletion
When the Testbox is no longer needed, it can be disposed of or destroyed to free up resources. In this stage, the Kubernetes namespace is deleted, along with all associated services. Once completed, the lifecycle of a Testbox starts over again.
IV. How developers can use Testbox
Developers can interact with a Testbox using the Testbox Manager API. Apart from offering total sandbox control, the Testbox Manager also helps developers to take charge of the Testbox’s lifecycle. It allows them to assign Testboxes, trigger service deployments, destroy Testboxes, and access Testbox-related data using an comprehensive API interface. This API interface is leveraged with two tools: Testbox UI and Devbox.
Testbox UI allows developers to create, manage, and dispose of a Testbox through a web-based, easy-to-use UI interface. They can use the UI to monitor the status of their Testbox and make any necessary changes or updates. Additionally, the web UI provides a dashboard with complete visibility of other Testboxes. It displays how many of them are used and how many are available. With the UI, developers can also create custom-built, reusable templates to specify different tech stacks or functionalities they need in order to test specific features or parts of Pipedrive.
Devbox is the newest and most-popular interface for Testbox. In short, it’s a CLI tool that provides features that go beyond Testbox’s capabilities. It does not just offer a way to acquire and customize a Testbox, it also allows remote development of services, network proxy, automated testing, and other features that are out of the scope of this article. Everything that a developer needs to start writing code is just one Devbox command away. At a glance, you can interact with Devbox in the following ways:
$ devbox new # Acquires a new testbox
$ devbox service develop # Deploy a service in development mode
$ devbox functional run # Execute functional tests of a service
$ devbox stack deploy # Deploy services grouped as stacks
$ devbox open shell # Open a shell into a service container
$ devbox pause # Pause the current testbox manuallyIf you would like to learn more about Devboxes, we have written an extensive article about it that you can find here.
V. Key highlights
As of today, we have 300 Testboxes running in four Kubernetes clusters. Each Testbox has an average of around 80 microservices, 108 pods and 5344 pods per cluster. The provisioning time of a single one is around 10 minutes. When a Testbox is acquired by a developer, it takes a few minutes to customize it, depending on the size of the chosen stack. On average, we have 200 Testboxes in usage every working day. This is a growing trend that we have observed over the past twelve months.
VI. Conclusion
To be able to quickly deploy changes to production, our entire deployment technology stack had to be revamped and optimized for efficiency and failure resilience. Our development environment is the core of our efficient CI/CD pipeline. The automated testing, monitoring, and deployment process is possible with our service-tailored development workflow. It has proven that our solution works for our needs.
As the company keeps growing, our platform will develop its own set of challenges. Hopefully, our DevOps team comes to rescue. We are motivated to provide the best sandbox tool for our developers and ourselves. Testboxes that can work for any need of a Pipedrive engineer and team without worrying about the risks of sharing others’ testing environments. The best part about Testboxes is that they are easily scalable and can be interfaced in multiple ways; this has been proven with the trend of increased usage.
You can also learn from our experiences to build a sandbox environment for your team or you can choose to outsource a testing environment — whatever suits your organization’s needs.
