Release 4.x of ce-vm, a digression about Docker, VirtualBox, Vagrant and docker-compose.

A new version of ce-vm, our local development stack, came out last week. It comes with a lot of changes and improvements, amongst which notably:

• Better default handling of Docker networking and port forwarding: on Mac OS X, ports are now only forwarded to loopback aliases (created automatically), preventing conflicts with ports on the host. On Linux and Windows, ports are not forwarded at all anymore by default, as they are reachable directly from the host.
• Removes the dependency on the “vagrant-triggers” plugin. This was not in the plans, really, as it means resorting to more “crude” ways of doing. But because the plugin was not maintained anymore, it was locking us to Vagrant 1.9. The plus side it that means one dependency less, so one less chance of things breaking due to version mismatch.
• Introduces a basic “dashboard-like” landing page. While it is still fairly minimal, it does give a nice overview of the various components.
• Last but not least, drops support for the VirtualBox provider !

Until now, you could choose between either Docker or VirtualBox as the provider with a simple variable change in ce-vm’s config file. A lot of “magic” was then performed at the Vagrant level to ensure everything would behave consistently whatever provider you choose.

Sounds fun, no ? However, it also meant a lot of work to try and maintain those two systems, that are profoundly different, while keeping the process as painless and automated as possible for the users:

Platforms (Mac+Linux+Windows) * providers (Docker+VirtualBox) * mounts/fileshare (vboxfs+NFS+SMB+rsync+native+osxfs) * networking (vboxnet+docker bridge+port forwarding) = far too many combinations to support.

Taking one provider out of the equation should greatly reduce the numbers of possible use cases to take into accounts, thus I had to make a choice. And it was not such an obvious choice as one might think, as I still consider VirtualBox has a lot of advantages over Docker for cross-platform local development stacks.

So, should I switch from VirtualBox to Docker?

If you are yourself maintaining your own stack/solution, and are wondering whether you should be moving away from VirtualBox to Docker, here are a few things you should consider:

Snapshots: VirtualBox lets you take hot snapshots of a running VM in a matter of mouse clicks [1], that you can restore in seconds (minutes for bigger projects). Handy when you screw up your database by testing your migration script. While you can “commit” your containers as images, it’s far less convenient as a quick “back in time” local backup tool.

Network consistency: As soon as you add a “host-only” network, you get consistent networking on all host platforms, with proper IP assigning and access from and to hosts/guests. Docker on Mac relies on port forwarding for host to guest access, and a clunky docker.for.mac.host.internal for guest to host access.

Performance: contrary to popular belief, Docker isn’t necessarily “faster” than VirtualBox. It is damn faster on Linux, no discussion possible. On Windows and even more on Mac, you get at best on-par performances[2] with what you would get with VirtualBox using NFS/SMB (or rsync, of course)[3].

Ease of use: Virtual Machines are operating in total isolation, and the concept is easy to grasp. Once you have sorted the file sharing, you’re basically done. Docker was not meant to be a “local dev” stack in the first place, and the ports to Mac/Windows still feel “alien”. You need to figure out networking (again), tons of options ( — privileged vs — cap-add) varying between platforms, and the commands can be cryptic for newcomers (docker run --name example -d -p "192.168.56.2:80:80" -p "192.168.56.2:443:443" -v "/Users/example/Projects/example/:/var/www/example:delegated", -h "example" --network=mynet --ip "192.168.56.2" --privileged=false pmce/jessie64:4.0.0 ?).

So, all that said, why did I pick Docker in the end ? For a variety of reasons, but in short:

• Performance on Linux: we do use a mix of Linux and Mac OS X internally, so the gain is immediate for part of the team already: no VDI hard-drive limit and resizing, RAM and CPU are not capped, native I/O…
• Flexibility: the ability to split the architecture into separate services is a game changer. While ce-vm still consists of 2 main containers (app/db) because of its “historical” background, creating dedicated containers is next on the roadmap. It’s also easier to share and re-use existing images.

Whether you should pick one or the other depends of a number of factors, like your audience (single dev ? multi-OS ? Linux nerds only ?), your deployment processes and your architecture (monolithic server ? services ?) and your motivation to spend time finding the perfect setup. But basically, don’t trust the hype and think Docker is the silver bullet that will solve all your problems.

What are you doing with Vagrant then? Haven’t you heard of docker-compose?

With the removal of VirtualBox support, it might seem logical to also drop the use of Vagrant enterly, and instead switch to a docker-compose approach.

Besides the fact that we have been using Vagrant for years, so everyone internally is familiar with vagrant up, ssh, halt, destroy, there are a few compelling reasons to stick to Vagrant.

• Advanced configuration: there are still a few areas that you cannot manage from docker-compose.yml. Typically, creating a named custom bridged network (that soon should be fixed by this MR, though).
• Preference parsing: ce-vm uses YAML configuration files that are parsed at run time (per project, per user, etc) for fine-grained control. Vagrant makes it easy to selectively parse them in order of precedence and react accordingly. With docker-compose, you end up forcing users to either maintain their own version of a docker-compose.yml file, or to pass variables at run time. This totally defeats the simplicity of having to just up.
• Host awareness: if you only care about a single platform, everything is fine, in that you can share a unique docker-compose.yml within your team. As soon as you go multiple-platform, you end up having either to maintain multiple variants, or to instruct people to manually amend some parts. Vagrant is aware of the host platform, letting you do react on this in Ruby from within the Vagrantfile and automatically perform changes accordingly.

In short, you can do a lot of ‘clever’ things from within Vagrant. On a simple vagrant up, we actually take care of git pulling the right branch matching your project version of ce-vm, creating loopback interface aliases if you’re on Mac OS, setting up sensible port forwarding for your host platform, switching on the --privileged flag if ever needed, computing various settings to pass to Ansible, and so on. All of which would require manual setup with docker-compose.

There are other less crucial points that are worth noting too: the handy vagrant share feature, having an easy ssh access, which means you don’t have to wrap each and every command in docker-exec or whatever layer (plus you can customize your shell session to your liking, eg. oh-my-zsh).

With that in mind, though, nothing is set in stone. Docker and docker-compose are in constant evolution — the recurring issue of accessing containers by IP on Mac will presumably be fixed someday — and I’m as averse as anyone to extra layers of dependencies when I can get rid of them…

1. See https://github.com/pm98zz-c/vbox-snapshots for automating the process and having an ongoing safety net.
2. Just search for “performance Docker Mac”, in case you think I’m making that up. It is notoriously slow, even though the introduction of “cached” volumes help a bit.
3. Yes. I know about docker-sync.