The way I use Docker as interactive environment for tries and experiments
TL;DR To protect my environment from from changes when doing experiments and playing around with new stuff, I made this docker image based on Ubuntu 18.04 for convenient interactive work inside the running docker container
The Problem
We experiment a lot. We often have no other choice. For good or for bad, the the pace at which software ecosystem is growing is accelerating. New versions of packages, modules, frameworks and even programming languages are released, before we even finish the first working version of a software product.
If you work with such widespread and extremely popular language as Javascript, you know about its very dynamic ecosystem, and you might have most certainly seen or even used a HUGE variety of packages, libraries and frameworks coming out and fading away. The things are changing very fast. Frontend developers very well know what is a Framework Hell. Every now and then there is a new shiny new shiny thing you’d like to try, unless you want to look outdated when talking to colleagues, and get yourself a feeling that you’ve got “behind the market trends”
Nearly every engineer experienced a frustrating Dependency Hell too. I am sure most Javascript developers had a tough night, after a minor change in some package (that’s supposed to be backwards-compatible) made the build failing.
But don’t worry you Javascript developer! You are not alone :-)
Your problems are well familiar to other engineers too. You might not believe it, but other programming languages have even crazier ecosystems. The dependency hell in many Python projects is not simpler than what you have in the front-end, believe it or not. At least you have a programming language that was designed to be async from the start. And you don’t need to care if the package is compatible with the event loop.
When it comes to Data Science and Machine Learning the complexity goes to the whole another level. The thing is that Python is usually just a wrapper over more performant C, C++, Java or Fortran. For example, in order to use Stanford CoreNLP, you’ll need to install Java. If you work with computer vision you’d need to install opencv. Often it even comes to building those dependencies from source. And of course, since we are working with “Big Data” we would like to have Spark, Dask or Ray on the local environment too.
Frankly, we often over-engineer, and use tools just to try, to learn, or simply because those tools look more sophisticated and make us look smarter. But this is something we cannot escape — it is an inevitable part of every engineer’s professional growth.
Okay, too many words
TL;DR Merge the complexity of the development environments with the speed changes, and you get the recipe for a huge headache.
The headache comes at times of changes - when we need to try a new version of something, to add new dependency, install additional software, upgrade dependency trees etc.
Simply being angry on the progress, and trying to convince everyone around that software engineering is a mess, and we need get back to vanilla Javascript, or even Java — is not an option.
No, it is not possible to ignore changes and freeze your environment. In 6 months you will realise the things have moved so far ahead, that the new library you want to use is not compatible with your application at all. And this new “thing” can easily be something groundbreaking that your project obsolete if not getting it! Like when word embeddings came out, that you simply can’t ignore. So it is better to keep up with the things changing.
Let’s start experimenting
But before an update happens, we need to try, evaluate and test. This process begins on the local developer’s machine —our laptops we use for coding. And not always the things go smoothly. Often it takes a lot of time reading docs, searching in for an answer in all possible platforms, groups and channels. And a lot of trying and trying hoping for the luck.
There is an issue though, we are already using our laptops to actually develop the production code. And in a haste to update the package/framework/language version, we can easily affect a main environment.
Thankfully we’ve got a lot of tools to isolate environments — the virtual environments. There is a plethora of them, for every programming language. These are only the few in the Javascript world: nvm, n, fnm, nodenv, nodeenv… Every couple of years there emerges another one like a Phoenix. Some teams jump to it ASAP, others keep using an old proven tool as soon as it does the work done.
And they do! Virtual environments helped us a lot in the ever changing world of software engineering. Today I can’t imagine having a project without virtual environment or another similar tool, that manages the tree of dependencies. We don’t use packages that are installed globally for serious development. So we can have several projects with the different package versions, and we use them to do experiments — to try new stuff.
But what if your project is beyond one language? Or has system dependencies that you can only install globally? The things are becoming tricky.
Let’s really start experimenting
Luckily there is a tool that we all love and use, and it perfectly suits the goal of isolating experiments and tests from the global environment without risking to affect the latter in any way. Sure thing — it is docker, what else could it be?
If in your project you need to activate more than one environment, or install any global dependencies, why not just using containers? Not only we will get the complete isolation, but we will also get a good part of a Dockerfile, that we will later use to package our application.
Docker is great for containerization - packaging code with all the dependencies. So if we use docker containers as isolated environments, why not using them as isolated development and experimentation environments? Seriously, this idea is not new. There are lots of attempts and publications on this regard.
There are very good reasons why we should use docker exactly for this purpose. With docker, any experimentation environment gets:
- isolation
- layered versions and backups
- ability to push and pull to container registry
- ability to run anywhere
So why aren’t we already doing everything in Docker yet? Maybe, because there is no suitable docker image.
It is actually not easy to set up and configure environment ready for experiments. Developer’s laptop suits best this purpose, because all the required and familiar tooling is already there. Setting it up in a docker will be tedious and not worth it for just one experiment. But if we do them often, investing some time becomes worth it.
Suitable Docker image
Most docker images we are used to, are either with runtimes (python, node, java etc.), with applications or services. Despite docker allows you to work with containers interactively, in my observation (quite limited, honestly, so I might be wrong), this feature is not used as often as it deserves to be.
Docker containers usually run only one process, don’t have docker inside docker, crontab is not running. To use it as test environment one would need each time to install lots of stuff - vim, git, wget, curl… even favourite Z-shell.
Actually there are lots of Dockerfiles out there to build images, that do exactly what’s needed: have cron, zsh, docker-in-docker and allow multiple processes. But each of them has just one of these features. Someone made dockerfile for Ubuntu with crontab. Another engineer created dockerfile with z-shell, and so on.
It was quite an easy job to put them all together, in one docker image that would have it all. I made it for myself, and enjoyed using as light-weight isolated environments — workspaces. They really helped me to iterate faster, try new things, versions and updates without being afraid to break something important.
I could start and stop dockerized ubuntu environments with a single docker command. I could push workspace images to the container registry, and pull on the server in the cloud. I was able to share them with peers, have backups and versions.
I made this docker image based on Ubuntu 18.04 with the minimal set of tools, that I usually need when trying something new:
- docker in docker — to be able to work with docker inside docker
- crontab — to schedule
- supervisord — to run multiple processes
- Python, pip
- Node, nodeenv, npm, yarn
- Z-shell — to makeconsole more user-friendly
- nano, vim, mcedit —text editors
- MC, ncdu — to explore files and sizes
- htop — to monitor resources
- git, git-flow, curl, wget, telnet, jq, apache2-utils
I included several applications, such as Midnight Commander, htop or ncdu — the set of minimally needed tools to work conveniently with dockerized environments, to make a feeling of working in VM. Midnight Commander will be familiar for those who worked with Norton Commander or its alternatives. Ncdu lets you analyze disk space usage, and htop lets you examine running processes and see resource utilisation.
This environment served me a great deal in projects based on Ubuntu, and I decided to share it with the community.
