Engineering immutable infrastructure using Docker with minimal overhead at Venngage
Unnecessary complexity is the enemy of a good software system. Well-written code or a robust infrastructure with small maintenance work required tends to be simple in nature. Also, a good software system has fewer lines of code, minimal inter-dependencies, and solves a few well-defined problems very well.
Overview
At Venngage, we created an immutable staging environment that gets deployed automatically based on new commits to a branch in our git repository. A new Docker image is built entirely from the target repository ( except for credentials or sensitive information that does not belong in the source repository). Every new commit becomes a new Docker image tagged with commit SHA.
In order keep this system simple, we only used readily available tools that are mature, widely used, and well supported:
- Docker
- Makefile
- Git hooks
- Crontab
There are many deployment systems and services that work well in large and complex environments. But for our single server staging environment with Nginx, PHP, and MySQL, those existing solutions brought in too much overhead and complexity.
Dockerfile
Docker has been growing very fast. The awesome team/community around Docker added a lot of new features and improvements recently. However, many of the new features and tools are not necessary for small projects involving only a few servers and services.
Generally, it is a good idea to use one container for one process or service, but with our simple requirement, we decided to use single a Docker image for both Nginx and PHP(php-fpm). Another reason is that Nginx configurations and PHP configurations directly affect each other, so having their configurations and service in a single repository made sense.
Makefile
Instead of using heavy-duty build automation tools, we just need to execute a few shell commands in order to manage Docker images and containers.
Note that the pull command echoes the current date, because the output will be stored as a log.
Git hooks
Git hooks are simple to setup, and they can be used to automate many deployment tasks. We use them to trigger the build and run freshly built Docker images from latest commit. Using .git/hooks/post-merge hook script, we can be sure that Docker images will be rebuilt only when there are new commits. This is important because we’ll be using crontab to pull the code periodically.
As a bonus, we’ve added a quick call to Slack to notify us when the new deployment is complete.
Crontab
Using the Makefile from above, we can trigger a git pull periodically. Every 5 minutes, Git will check with the default remote branch, fetching/merging new code if available. Only when there are changes is the post-merge script executed, building a new Docker image and replacing the existing container.
One added bonus to this setup is that we can pretty much update all aspects of the setup remotely. We can update the Makefile to modify Docker commands, git hook scripts, and even crontab itself can be updated by adding appropriate commands in Makefile. Of course, it’s a bit dangerous in the sense that we are allowing remote command execution. However, we can put more protective measures in the hook script using signed git commits. This is where we can improve this setup in future iterations.
We could have used Github’s web hooks for triggering the build. Unnecessary git pulls we do every 5 minutes could have been avoided. However, for that to happen, we need to run a web server for it — complexity we didn’t want to add. And, having a log of what happened every 5 minutes isn’t too bad. With cheap storage cost, extra 50 bytes per 5 minutes is small price we pay for potentially useful information.
Conclusion
With very few lines of code with widely available and stable tools, we have built a simple but quite robust staging environment which is immutable (sort of). This is the very first iteration of our environment and we are sure there is tons of room for improvement. For the next iteration, we are planning to add more safety mechanisms, automated testing, and logging. Stay tuned!
