Configuring Travis CI to enable devDependencies incompatible with Node versions your application supports

“Wishes” (Glass Feathers with 24k Gold Leaf) by Kiara Pelissier

As part of the current Outreachy cohort, I am contributing to the Node.js client library for Jaeger, an open source distributed tracing system.

Recently, we ran into the problem of wanting to update our tooling in a way that incorporated dependencies that were incompatible with all the Node versions we support. Note: the Node.js client library for Jaeger is committed to supporting Node v0.10 and up.

Yes, Node v0.10 and v0.12 have end-of-life status. However, surveys suggest that 30 to 50 percent of Node.js commercial projects are using Node v0.10 or v0.12 in production. Jaeger, as a distributed tracing system, provides the ability to monitor and troubleshoot microservices-based applications. Jaeger’s Node.js client supports versions 0.10 and up so that projects using those Node versions in production can have the benefit of observability into their microservices-based distributed systems.

However, Jaeger’s current commitment to supporting Node v0.10 had a limiting effect on the dependencies that we were using in the Node.js client library.¹

To recap, the most common types of dependencies in every project are dependencies and devDependencies. Your package.json specifies your project’s dependencies:

{
   "name": "my-project",
   "version": "1.0.1",
   "dependencies": {
      "package-a": "^1.1.2"
   },
   "devDependencies": {
      "package-b": "^1.5.0"
   }
}

The packages specified in dependencies are the dependencies needed when running your code.

The packages in devDependencies are your development dependencies. These are the dependencies that you need at some point during your development workflow but not when running your code (e.g. a test framework or a transpiler).

Previously, we had Travis CI configured to build and test each version of Node that Jaeger’s Node.js client supports.² In practice, this meant that our dependencies and devDependencies had to be compatible with every version of Node the Jaeger Node.js client supports, including Node v0.10.

Then we decided to upgrade from babel-preset-es2015 to babel-preset-env.

Upgrading to babel-preset-env

For transpiling ES2015+ to ES5, Babel recommends using babel-preset-env. Without any configuration options, babel-preset-env behaves exactly the same as babel-preset-latest (or babel-preset-es2015, babel-preset-es2016, and babel-preset-es2017 together). You can also configure it to target certain browsers or runtime environments, and this will make your bundles smaller.

There were enough benefits that we decided to upgrade from babel-preset-es2015 to babel-preset-env. However, babel-preset-env targets node 4.

Our .travis.yml was configured so that Travis would build and test for each Node version the Jaeger Node.js client supported. The start of our .travis.yml looked similar to this:

language: node_js
node_js:
- 'node'
- '8'
- '6'
- '4'
- '0.12'
- '0.10'

Then we had a script that both built and tested each version of Node listed under node_js. The result was that upgrading our devDependencies to include babel-preset-env broke the Travis build for Node v0.10 and v0.12.

We needed to reconfigure our .travis.yml file so that we would be able to upgrade to newer versions of our tooling, and then, once the transpiling was done, run our tests in Travis with different versions of Node. In order to do so, we utilised the Travis build matrix:

language: node_js
node_js:
- '6'
matrix:
  include:
  - env: TEST_NODE_VERSION=0.10
  - env: TEST_NODE_VERSION=0.12
  - env: TEST_NODE_VERSION=4
  - env: TEST_NODE_VERSION=6
  - env: TEST_NODE_VERSION=node

If there is one version of Node listed under node_js Travis will interpret that Node version as it’s default version. We can then use the Travis default Node version as our build version for every set of tests, thereby enabling the use of devDependencies irrespective of whether they are compatible with every Node version we support. We also customise our tests to run against every version of Node included as an env within the Travis matrix. To do so, first we install the Node version we will test against and then switch to the Travis default version:

before_install:
# Install the Node version to test against
- nvm install $TEST_NODE_VERSION
# Switch back to build-time Node version
- nvm use $TRAVIS_NODE_VERSION

Then we build with the Travis default version of Node, and afterwards switch to the version of Node we are testing within our Travis matrix:

before_script:
- make build-node
- nvm use $TEST_NODE_VERSION
- node --version
- rm -rf ./node_modules package-lock.json
- npm install

Notice that we print out to our Travis CI logs the Node version that we’ll be testing. We then remove all the node_modules that had been installed during the initial build and, importantly, the package-lock.json that had been installed during that build. We then initiate another npm install while using our TEST_NODE_VERSION.

This worked great while updating from babel-preset-es2015 to babel-preset-env! However, soon we thought to add Prettier to the project.

Adding Prettier, Husky, and Lint-staged

Prettier: An Opinionated Code Formatter, Excellent for Open Source

I’ve written about Prettier and why we decided to add Prettier to Jaeger’s Node.js client here. To facilitate smoother collaboration, we decided to add a pre-commit hook using husky and lint-staged, which would ensure that all PRs would be formatted by Prettier using our chosen configuration.

Unfortunately, husky could not be installed on Node v0.10/v0.12. The Travis CI build status page is great for seeing at a glance if the build passes or fails and, in our case, for which versions of Node. When failures occur, reading the logs in Travis CI is fantastic to find out more information. In our scenario, we could see that our final npm install in our before_script would error out when trying to install husky on Node v0.10/v0.12. My first attempt to solve this problem was to place husky as an optionalDependency within our package.json:

"devDependencies": {
...
},
"optionalDependencies": {
  "husky": "^0.14.3"
},
"scripts": {
...

Optional dependencies are just that: optional. If they fail to install, npm will continue with the installation process and, once completed, will consider the installation successful.

This is useful for dependencies that won’t necessarily work on every machine, or every version of Node your application supports. However, you do need to have a fallback plan in case the optionalDependencies are not installed. For Jaeger’s Node.js client library, this is not a problem as husky is a devDependency not necessary when running the application in production and our recommended Node version for development is Node 6.

To ensure we had no problem with our builds in Travis CI, for our final npm install in our before_script, we used a --no-optional flag, like so:

- npm install --no-optional

Our Travis CI tests all passed! Excellent.

Except, optionalDependencies are by default installed as dependencies not devDependencies. In other words, optionalDependencies are also production dependencies. Unfortunately, this meant that husky was being installed for consumers of the Node.js client library. What we needed was an optional devDependency.

More broadly speaking, for any package downloaded from npm, all production dependencies are installed, including optionalDependencies. Unsurprisingly, we weren’t the only ones wishing for optional devDependencies. The problem and possible solutions are discussed here.

Happily, Joe Farro, came up with an elegant, simple solution that works for Jaeger’s Node.js client. Husky and lint-staged are now both listed as devDependencies in the package.json; however, before the final npm install in the before_script, we remove husky and lint-staged from the package.json :

- npm uninstall -D husky lint-staged

For clarity, I’ll explain the components of the above command.-D is an alias for --save-dev, which when used with npm install would ensure that the listed packages would both be installed in the local node_modules folder and be listed as devDependencies in your package.json. As we are using it with npm uninstall it does the opposite, removing the packages from within your devDependencies in your package.json. When we run, in the next step, the npm install command, those packages are no longer listed in the package.json and so there will be no attempt to install them. The before_script in the .travis.yml file now looks similar to:

before_script:
- make build-node
- nvm use $TEST_NODE_VERSION
- node --version
- rm -rf ./node_modules package-lock.json
- npm uninstall -D husky lint-staged
- npm install

In the end, I’m really happy with the solutions we’ve come up with for enabling us to update the devDependencies for Jaeger’s Node.js client library, and hope that this explanation is useful for others!

Note: In an attempt to make this explanation of how we reconfigured our .travis.yml file more helpful to others, I have simplified the descriptions of Jaeger’s Node.js client library’s .travis.yml file. The Node.js client library for Jaeger is open source and the current .travis.yml file can be viewed here.

[1] The Node.js client’s current support for Node v0.10 and up also has an effect on which package manager we use. The Jaeger Node.js client uses npm, as yarn supports Node 4 and up.

[2] Travis CI is a continuous integration service used to build and test projects hosted on GitHub. For Jaeger’s Node.js client we have Travis CI run our tests on every pull request made to the GitHub repo. The .travis.yml file tells Travis the programming language for your project and how to build it. Any step of the build can be customized.

Travis CI is free for open source projects on GitHub. ❤️