Running Meteor tests in Heroku CI

We have a culture of using Continuous Integration (“CI tests”) at Leaf in order to keep the quality of production high as our team scales.

Heroku recently announced public beta of Heroku CI, which is what we use to conduct our CI tests.

What are tests?

Tests in this context are basically code to check the functionality of the app’s code. Put simply, it runs some parts of the code and checks the execution result to see if it works as expected. If you change the target code that doesn’t follow the spec, the test code would fail, and we would be alerted. This means that our production code for the Leaf app has a higher chance of being error free for our users without the burden of continuous manual testing.

What are CI tests?

CI tests are the tests that run continuously and automatically whenever developers submit new code to add new functionality. At Leaf, we require a GitHub pull request to pass a CI test, and reviewers check the CI test result before they approve the change in the pull request. You could even configure GitHub to avoid accepting the change if the CI test does not pass, which is something I may implement as our team scales further.

Running tests in Heroku CI

Heroku CI is built on the same infrastructure of Heroku, a platform-as-a-serivce, where you can deploy code to host a web application or such. We use Heroku to host several Leaf backend services and applications.

To run tests in Heroku CI is as easy as if it’s normal Node.js code. It will automatically detect the code being Node.js and will run the test by npm test which is a typical way of running tests in Node.js.

How about Meteor tests?

Meteor is a JavaScript framework that allow us to code a Web application pretty quickly with well-organized application structure and lots of third-party libraries. We use Meteor at Leaf because it allows our team to develop a product very efficiently and it also helps introducing the bleeding edge Web technologies.

Meteor also provides a test framework, however to run Meteor tests you need to install Meteor.

Running Meteor tests in Heroku CI

Fortunately, Heroku CI provides a way to customize the behavior, so all you need is to put a file app.json with the following content.

"environments": {
"test": {
"scripts": {
"test-setup": "mkdir /tmp/meteor-home; curl | HOME=/tmp/meteor-home /bin/sh; HOME=/tmp/meteor-home PATH=$PATH:/tmp/meteor-home/.meteor meteor add dispatch:phantomjs-tests",
"test": "HOME=/tmp/meteor-home PATH=$PATH:/tmp/meteor-home/.meteor npm test"

This simply installs Meteor before running a test. We also install the PhantomJS library in this case. For more information how to write the app.json file, please refer the original document in Heroku.

Next step

The above example works well, but it is not ideal in terms of resource efficiency. It should be better to cache the Meteor installation and reuse it for the next execution of tests. Heroku provides another customization method called Buildpack, and by using it we should be able to have more control on installation and caching. We look forward to seeing a better solution with it.

As this technology improves, we hope to be able to build more sophisticated and resource-efficient tests, which would be critical as our engineering team at Leaf grows.