Dev environment evolution in Pipedrive from a developer’s perspective
In January 2020, when I joined Pipedrive, many things were new to me: from a 30-employees workplace to a company with more than 600 employees, from a Czech to English-speaking environment, from a few monolithic PHP apps to hundreds of (mostly) TypeScript microservices.
But I knew that certain things are the same everywhere. For example, when something works and migration to some other setup is too complex, you should just leave it as is. Boy, was I wrong.
Let me tell you a story about how our dev environment has evolved during my time at Pipedrive, about the changes and challenges it’s gone through, and how we reduced developer hours spent on troubleshooting dev env issues.
Docker workstation
When I joined Pipedrive, we’d use docker-workstation (a.k.a. dws) everywhere. Each service had its own docker-compose file(s) orchestrating what should run on which port in dev, prod or end-to-end tests mode, etc.
How did it (ideally) work whenever you needed to develop something? First, you would run something like “mode.sh your-service-name –mode dev” and, once you correctly set up all dependencies, your service would be up and running on the specified port with any change triggering a rebuild.
Sounds great, right? But there were some issues with that solution:
- Everything ran locally — Whenever you needed the whole app running (tens of or even 100+ services), your computer would start to sweat.
- Conflicting ports — Sometimes, one service wouldn’t start because something else was living on its port.
- When creating a new service, you had to know or guess the ports of all other services.
- Sometimes, when it got stuck, the only solution would be to restart the whole Pipedrive stack. Often, more than once.
- Sometimes, the entire computer would require a restart.
- The auto-rebuild function didn’t always work, either, although we found some workarounds around that. For example, if you “touched” a file, everything would get rebuilt.
- I felt that the user interface (a shell script) consisted of unintuitive command names and options.
There were good days — even weeks — when everything would work as it should. There were also bad ones, especially following docker or macOS updates. Then again, avoiding them and using updated versions was less than ideal.
Eventually, it led us to experiment with various different non-reproducible environments (OS version, Docker version, Dependency Versions, computer specs, etc.), making it very complex for the team responsible for the development environment to debug and fix issues quickly.
Developers were afraid to update dependencies (Docker, VPN, OS) or adopt new tools due to negative past experiences, where an update would break their development environment for several hours, if not days.
In the beginning, I kinda liked that I could delve into these issues, find out where the problem lies and understand how everything works. But when I was in a hurry and needed to do something important fast, it would become a nuisance.
Kubernetes workstation
Then, we decided to migrate from Docker Swarm to Kubernetes. We needed a solution that would allow developers to develop their services in the k8s context. And so, we started building our Kubernetes-workstation (also k8s-workstations or KWS). Some considerations we had were:
- Trying out remote dev environment as a proof of concept
- Gathering feedback from developers and solving dev env issues
- Assigning ownership and simplifying maintainability of dev environment
Though the first commit in KWS repository is from March 2020, we only started to migrate our tribe’s services at the beginning of 2021, probably due to some technical blockers and prerequisites from our infra/DevOps side.
Here is a description from our readme of what k8s-workstation is designed for:
- Provide a development environment where microservices can be built, keeping Kubernetes in mind from the very beginning (better parity between the dev, test and live environments)
- Simplify development environment setup
- Resolve dependency between Pipedrive’s different microservices
- Provide a method to offload some parts of the development environment away from developers’ workstations (remote development)
As service owners, developers only need to convert docker-compose files to Helm chart templates. We already had a decent step-by-step guide covering the most common use cases for that. If a service setup was somehow unusual or complicated, we would ask the dedicated DevOps engineer assigned to each team to help us with the migration.
KWS offered a way better CLI and did indeed solve some issues. But debugging it was (at least for me) even harder. Plus, it had its own issues. For example, running tens of services locally was unbearable.
Learn more about our KWS journey here.
Here comes Developer Experience PM
In the middle of June 2021, a post appeared in our #all-devs channel. Its main message read:
“We are opening a new position: Developer Experience PM. You will take a break from coding and focus on putting together the plan for the dev env. You will work closely with Devops and Infra to sync those plans. You will impact the daily life of 200+ engineers in Pipedrive. And you won’t be alone.”
And so we created a new Developer Experience PM role for someone who should consider us, Pipedrive’s developers, as his customers and try to make us happy.
Devbox
Pipedrive has been using testboxes for as long as I can remember. That is a sandboxed environment with pre-provisioned Pipedrive apps designed for manual/automated testing in an isolated environment running in a private cloud (Rackspace).
Whenever I needed some more complex setup with too many services, my laptop would heat up quickly, and I had to keep it connected to the charger to prevent it from shutting down, so I ended up using testbox frequently.
Everything works remotely: We have a sandboxed environment, UI and a friendlier CLI. Still, whenever you needed to test something, you had to commit, push, build and deploy it there. With no debugger available, any mistake meant starting anew.
Another issue was the pool we used to assign the tesboxes from: It was too small and, sometimes, we couldn’t get any free testbox.
These “disgruntled customers” demanded a new solution, leading teams to research and launch missions to find a suitable solution. One of our prerequisites was migrating the sandboxed environment to the public cloud (AWS) for a scalable and on-demand infrastructure that wouldn’t restrict the amount of boxes.
Devbox beta-testing was launched in November 2021 (I volunteered for it, and it was definitely worthwhile). Again, we needed some help with migration but only around three config files. Thankfully, everything has been working out-of-the-box (that is, in-the-devbox). The responsibles for devboxes are always there to help, guide us and debug whatever problems we encounter.
So, what do I do nowadays when I want to develop something?
And voila! After a few minutes (deploying the stacks is the most time-consuming and takes 2–3 minutes) I have Pipedrive’s complete setup running. The whole infrastructure (MySQL DBs, Redis DBs, Kafka cluster, ES, you name it) is available and completely isolated from other devboxes.
Today, we locally monitor the files I develop for any changes. We then sync the files with the devbox, triggering an automatic rebuild+restart process. Moreover, remote debugging is no longer an issue.
Put an intuitive CLI with either interactive inputs or passing values via arguments directly on top of it, and you got happy developers! :)
Some features are still missing, but we collect feedback and, based on it, come up with new cool features all the time.
Okay, I lied a little bit earlier… The script above has about 25 lines overall. And, apparently, some other people need complicated and more specific setups, too. So, we’ve added support for playbooks to help them:
- Populate Pipedrive with hundreds of thousands of leads
- Support the flow of randomly generated leads, persons, organizations and activities and create random links between them
Learn more about how devboxes work internally here
Conclusion
Is everything fixed and working flawlessly now? Nope, and it probably never will. There will always be new issues to tackle. But the most significant pain points have been resolved, and whenever something new appears, we have better and faster ways to deal with it.
Take care of your developers. Make sure that they’re happy and have the tools to work effectively. Time is money, and saving developers time allows the whole company to grow faster and make more money.
So, pretend every developer spends 2 hours a week debugging dev env issues. This adds up to a total of 8 hours (1 working day) a month. With 300 developers, you’d be throwing 300 working days a month down the drain!
And it’s not only about time. When your developers aren’t busy with debugging repeating, annoying and unnecessary issues, they’re more motivated and productive in general. So try to keep your developers happy. Trust me: it’s worth it.
Interested in working in Pipedrive?
We’re currently hiring for several different positions in several different countries/cities.
Take a look and see if something suits you
Positions include:
- Software Engineer in DevOps Tooling
- Backend, Full-Stack, iOS Engineers
- Junior Developers and Interns
- Infrastructure Engineer
- And several more…
