From shouting “Marek, your changes are going to production” to the first version of Continuous Deployment for all microservices at Docplanner
Docplanner? Haven’t heard.
Docplanner is the world’s biggest healthcare platform. We operate on 4 continents, our patients make over 2 million appointments to the best professionals in their neighborhoods — all this results in over 100 000 req/min.
A long time ago…
Late 2012 is the point in history where we start our tour.
Back then, there were just 7 or 8 of us in the IT dept working on one application. We had a System Administrator who was taking care of dedicated servers placed in OVH. To deploy changes, we had to SSH on some server and run a script that executed svn up on all servers in parallel; BUT, before doing so, we shouted to our entire room whose changes we will deploy in a moment. You know, for security reasons :D How often was that? Once, maybe twice per day.
Ok, but what was wrong with it?
The answer is really simple, everything.
The very first thing that was wrong was our mindset. We were, at times, committing changes that were not ready to go live or sometimes not even tested once on dev machine! (Nowadays we would call it YOLO mode.)
The second thing was that sometimes svn up failed on some servers, and we randomly delivered two versions of the application to our users.
Next, every modification of deployment flow required Admin’s attention and just wasn’t easy to perform.
Last but not least, sometimes the composer install failed, and the application was just broken on a particular machine.
Step forward
To minimize problems with failing deployment steps, we started to prepare the application on a single node and rsync all files to other machines — phew, half of our problems went away!
New kid on the block
Almost a year later, our IT grew bigger with 2 new people, all still working on a single application. The hype on Symfony 2 started to grow in these days, and as cool kids befit, we started to adopt it. To make it all h̶a̶r̶d̶e̶r̶ cooler, we decided to stop using SVN on behalf of Git at the same time.
As migration to Symfony2 was quite a big project, we wanted to deploy it next to our old application. Bash script that we have been using to deploy was hard to modify following the new requirements, so we decided to use Capifony, an open-source solution dedicated to deploying Symfony based apps.
Step back
Together with a step forward, which using Capifony undeniably was, we took a step back. We didn’t notice that, with its default settings, Capifony repeats its steps on each server that caused some confusion again. Fortunately, it was easy to change; there was a plugin to Capifony that performed all steps on a single node and rsync the result files on other machines.
Don’t yell at Git
Migration from SVN to Git itself was quite easy from a technical point of view. Together with a version control system, we decided to change our mindset too — since then, all changes pushed to the master branch had to be manually tested (before too, but you know…) and ready to deploy on production any time. It was the moment when we stopped shouting about the incoming deploy. Almost. This “tradition” was with us for a quite a long time, so it happened from time to time that someone forgot himself and started yelling :D
Bash doomsday
We were developing two completely different applications on two separate branches in the same repository. When the day of showing this new application to the world finally came, we had to put all the files of the old application to a directory in our new application — only to use the old application in those parts of our platform that were not rewritten to new framework yet. Sounds pretty easy, right? As we wanted to keep the history of changes in both applications, I made a dry “join” run with different branches, and it went well; we had the changes from both applications. When it came to replicating that on the main branches…. I messed it up, and we finished with a single commit named “legacy,” authored by me, and containing all files from the old application -_-” I don’t remember why, but we were in a hurry and we decided to leave it like that.
After joining the branches, we didn’t need our old bash deploy script anymore. I mean for a week since we decided to make a bash wrapper for Capifony to simplify deployment execution and automate some other stuff.
BTW, for the next year or two, the newcomers blamed me for the whole legacy code :D
Staging & Continuous Deployment
2014 was quite a busy year for us. Our IT dept grew to ~15 people, we started writing tests for our code, and the company began to operate on many new markets and acquired one of the competitors.
We wanted to bring in Continuous Deployment to deliver our changes faster in response to growing business needs. To do so, we decided to create an environment that was almost identical to production but also allowed us to test new features with production data before we introduce them to our users — we called it staging.
We defined staging as an environment using the same instances of database and services like Redis. Every commit pushed to repository needed to be deployed on staging as soon as tests on CI passed, and all our offices needed to use this version of the application.
To do so, we created 3 branches:
master — a default branch for the repository, developers merge their changes there
staging — a branch automatically deployed on a staging environment
production — a branch automatically deployed to production
To deploy changes, we were still using Capifony but with two steps added. The steps were simple; for staging, it was just a regular flow that we used before, but as a first step, we added a check if there are some commits on the staging branch that were not deployed yet — if so, then deploy them, if not, just exit. Deploys on staging were scheduled in crontab every 5 minutes.
Production had an additional step before checking if there are some changes to deploy. It moved all commits that are on the staging branch to the production branch. Deploys on production were scheduled in crontab 3 times per day.
Our approach to staging also had a drawback. Since we had the staging, all our changes had to be backward compatible in terms of using services like database or ElasticSearch. It soon caused some problems like migrations that crashed production despite staging working well, but we adopted quite fast to the new approach, and it stopped causing any problems.
Microservices
It’s 2015, there are ~20 people in the IT dept, and the company acquires another competitor.
Our application was already so big that it started to cause problems starting with complexity, through development comfort, and ending on performance issues related to the giant codebase. Taking that into consideration, we decided to cut out part of the application and create SSO as the first step of the transformation our application into microservices.
The additional application generated a new problem. How to deploy it? We were already using Ansible to manage our servers, and we wanted to develop the next microservices in PHP using Symfony. Hence, we decided to create an Ansible role that generates Capifony configuration files and other stuff required for each microservice that we need to deploy — and we did, based on the Capifony setup that we already had.
We took advantage of the opportunity, and we changed branch naming. Master became develop, and production was renamed to master — I don’t remember why, but we did it.
A bit later, we decided to keep commits on master only 30 minutes behind staging instead of moving them 3 times per day.
We were quite satisfied with this solution, but we already knew that due to the rapid growth of the company and plans on many microservices, this approach wouldn’t work well in the long run.
Then we decided to use Docker and Kubernetes, but that’s material for another story.
Are you curious about how the deployment flow at Docplanner works currently? Story coming soon!
