Testing Meteor Up — It’s Harder Than We Thought

As you may already know, we here at Kadira are busy building a modular version of Meteor Up. We are planning to release it in February 2016. It’s based on mupx with some fixes for common issues. Most importantly, we will be able to get new features and bug fixes very quickly.

Have a look at the source code and the roadmap.

In the previous version of Meteor Up, there were no unit or integration tests. Actually, Arunoda has to check for all the situations manually. That’s a major barrier to accepting new PRs. We could get around this with a proper test setup, but building an automated test setup for Meteor Up is very hard, because you need to work with real servers.

Test setup for Meteor Up 1.0

For Meteor Up 1.0, we are using a combination of Mocha and shelljs to write integration tests. Shelljs invokes the Meteor Up binary just like any other command-line tool and checks the output for certain conditions.

Instead of mocking, we decided to actually deploy a Meteor app on a real server when testing Meteor Up. Our initial thoughts were to spawn small aws/gcloud vm instances on the fly per test and destroy them afterwards. But we realized this approach might make it hard for other people to contribute to if they don’t have access to our selected cloud provider. So to make things a little easier, we thought of using a Docker-in-Docker approach.

Docker-in-Docker

To test Meteor Up, we need a server with SSH access. In this approach, we launch a separate Docker container for each of our test cases and run the test selecting that Docker instance as the server.

Interestingly, Meteor Up also uses Docker inside the server. Actually, it installs Docker inside the server. Then it uses MeteorD to run Meteor apps.

This is a case of Docker-in-Docker.

There are a few things we need to consider when creating a Docker-in-Docker setup. Some of them are listed in this post.

Other than these, the most important things we noticed are:

• You should change the Docker storage driver from Aufs (the default) to devicemapper since Aufs does not support nested mount points.
• You should run your outer Docker container with the `- privileged=true` flag to allow it to access host resources like devices (which is required to run a Docker daemon).

Running tests in parallel

Mocha runs tests in series. There is no option to run tests concurrently even if you have the resources. But since each of our tests uses its own Docker container, we can speed up the testing if we can run them in parallel, if we have test boxes with enough CPU and RAM.

Most of the tests were already written in Mocha so it was hard to switch to another test runner that supports parallel testing. So we decide to write a custom shell script which uses GNU Parallel to run tests in parallel.

Basically it sets up the Docker instances as servers for each test, and then runs the Mocha binary with the `-g` (grep) flag to select just one test per Docker instance from the whole test suite.

Then it uses GNU Parallel to schedule tests and manage parallelism. GNU Parallel can schedule a given number of processes to a limited number of CPU cores easily.

Continuous integration

We tried several CI providers, but none of them supported privileged Docker containers, for obvious reasons. In the future, we are hoping to set up our own CI server like Jenkins to run tests automatically on pull requests, but for right now, all the tests have to be run manually by running a single shell script.

Have a look at Meteor Up’s contribution guide for more information.

We are getting so close to releasing Meteor Up 1.0.
Have a look at the roadmap, since we would really like your help.

Follow KADIRA VOICE to receive updates on Meteor Up.