Configuring Travis CI to enable devDependencies incompatible with Node versions your application supports
As part of the current Outreachy cohort, I am contributing to the Node.js client library for Jaeger, an open source distributed tracing system.
Recently, we ran into the problem of wanting to update our tooling in a way that incorporated dependencies that were incompatible with all the Node versions we support. Note: the Node.js client library for Jaeger is committed to supporting Node v0.10 and up.
Yes, Node v0.10 and v0.12 have end-of-life status. However, surveys suggest that 30 to 50 percent of Node.js commercial projects are using Node v0.10 or v0.12 in production. Jaeger, as a distributed tracing system, provides the ability to monitor and troubleshoot microservices-based applications. Jaeger’s Node.js client supports versions 0.10 and up so that projects using those Node versions in production can have the benefit of observability into their microservices-based distributed systems.
However, Jaeger’s current commitment to supporting Node v0.10 had a limiting effect on the dependencies that we were using in the Node.js client library.¹
To recap, the most common types of dependencies in every project are dependencies
and devDependencies
. Your package.json
specifies your project’s dependencies:
{
"name": "my-project",
"version": "1.0.1",
"dependencies": {
"package-a": "^1.1.2"
},
"devDependencies": {
"package-b": "^1.5.0"
}
}
The packages specified in dependencies
are the dependencies needed when running your code.
The packages in devDependencies
are your development dependencies. These are the dependencies that you need at some point during your development workflow but not when running your code (e.g. a test framework or a transpiler).
Previously, we had Travis CI configured to build and test each version of Node that Jaeger’s Node.js client supports.² In practice, this meant that our dependencies
and devDependencies
had to be compatible with every version of Node the Jaeger Node.js client supports, including Node v0.10.
Then we decided to upgrade from babel-preset-es2015 to babel-preset-env.
Upgrading to babel-preset-env
For transpiling ES2015+ to ES5, Babel recommends using babel-preset-env. Without any configuration options, babel-preset-env behaves exactly the same as babel-preset-latest (or babel-preset-es2015, babel-preset-es2016, and babel-preset-es2017 together). You can also configure it to target certain browsers or runtime environments, and this will make your bundles smaller.
There were enough benefits that we decided to upgrade from babel-preset-es2015 to babel-preset-env. However, babel-preset-env targets node 4.
Our .travis.yml
was configured so that Travis would build and test for each Node version the Jaeger Node.js client supported. The start of our .travis.yml
looked similar to this:
language: node_jsnode_js:
- 'node'
- '8'
- '6'
- '4'
- '0.12'
- '0.10'
Then we had a script that both built and tested each version of Node listed under node_js
. The result was that upgrading our devDependencies
to include babel-preset-env broke the Travis build for Node v0.10 and v0.12.
We needed to reconfigure our .travis.yml
file so that we would be able to upgrade to newer versions of our tooling, and then, once the transpiling was done, run our tests in Travis with different versions of Node. In order to do so, we utilised the Travis build matrix:
language: node_jsnode_js:
- '6'matrix:
include:
- env: TEST_NODE_VERSION=0.10
- env: TEST_NODE_VERSION=0.12
- env: TEST_NODE_VERSION=4
- env: TEST_NODE_VERSION=6
- env: TEST_NODE_VERSION=node
If there is one version of Node listed under node_js
Travis will interpret that Node version as it’s default version. We can then use the Travis default Node version as our build version for every set of tests, thereby enabling the use of devDependencies
irrespective of whether they are compatible with every Node version we support. We also customise our tests to run against every version of Node included as an env
within the Travis matrix
. To do so, first we install the Node version we will test against and then switch to the Travis default version:
before_install:
# Install the Node version to test against
- nvm install $TEST_NODE_VERSION
# Switch back to build-time Node version
- nvm use $TRAVIS_NODE_VERSION
Then we build with the Travis default version of Node, and afterwards switch to the version of Node we are testing within our Travis matrix:
before_script:
- make build-node
- nvm use $TEST_NODE_VERSION
- node --version
- rm -rf ./node_modules package-lock.json
- npm install
Notice that we print out to our Travis CI logs the Node version that we’ll be testing. We then remove all the node_modules
that had been installed during the initial build and, importantly, the package-lock.json
that had been installed during that build. We then initiate another npm install
while using our TEST_NODE_VERSION
.
This worked great while updating from babel-preset-es2015 to babel-preset-env! However, soon we thought to add Prettier to the project.
Adding Prettier, Husky, and Lint-staged
I’ve written about Prettier and why we decided to add Prettier to Jaeger’s Node.js client here. To facilitate smoother collaboration, we decided to add a pre-commit hook using husky and lint-staged, which would ensure that all PRs would be formatted by Prettier using our chosen configuration.
Unfortunately, husky could not be installed on Node v0.10/v0.12. The Travis CI build status page is great for seeing at a glance if the build passes or fails and, in our case, for which versions of Node. When failures occur, reading the logs in Travis CI is fantastic to find out more information. In our scenario, we could see that our final npm install
in our before_script
would error out when trying to install husky on Node v0.10/v0.12. My first attempt to solve this problem was to place husky as an optionalDependency
within our package.json:
"devDependencies": {
...
},
"optionalDependencies": {
"husky": "^0.14.3"
},
"scripts": {
...
Optional dependencies are just that: optional. If they fail to install, npm will continue with the installation process and, once completed, will consider the installation successful.
This is useful for dependencies that won’t necessarily work on every machine, or every version of Node your application supports. However, you do need to have a fallback plan in case the optionalDependencies
are not installed. For Jaeger’s Node.js client library, this is not a problem as husky is a devDependency
not necessary when running the application in production and our recommended Node version for development is Node 6.
To ensure we had no problem with our builds in Travis CI, for our final npm install
in our before_script
, we used a --no-optional
flag, like so:
- npm install --no-optional
Our Travis CI tests all passed! Excellent.
Except, optionalDependencies
are by default installed as dependencies
not devDependencies
. In other words, optionalDependencies
are also production dependencies. Unfortunately, this meant that husky was being installed for consumers of the Node.js client library. What we needed was an optional devDependency
.
More broadly speaking, for any package downloaded from npm, all production dependencies are installed, including optionalDependencies
. Unsurprisingly, we weren’t the only ones wishing for optional devDependencies
. The problem and possible solutions are discussed here.
Happily, Joe Farro, came up with an elegant, simple solution that works for Jaeger’s Node.js client. Husky and lint-staged are now both listed as devDependencies
in the package.json
; however, before the final npm install
in the before_script
, we remove husky and lint-staged from the package.json
:
- npm uninstall -D husky lint-staged
For clarity, I’ll explain the components of the above command.-D
is an alias for --save-dev
, which when used with npm install
would ensure that the listed packages would both be installed in the local node_modules
folder and be listed as devDependencies
in your package.json
. As we are using it with npm uninstall
it does the opposite, removing the packages from within your devDependencies
in your package.json
. When we run, in the next step, the npm install
command, those packages are no longer listed in the package.json
and so there will be no attempt to install them. The before_script
in the .travis.yml
file now looks similar to:
before_script:
- make build-node
- nvm use $TEST_NODE_VERSION
- node --version
- rm -rf ./node_modules package-lock.json
- npm uninstall -D husky lint-staged
- npm install
In the end, I’m really happy with the solutions we’ve come up with for enabling us to update the devDependencies
for Jaeger’s Node.js client library, and hope that this explanation is useful for others!
Note: In an attempt to make this explanation of how we reconfigured our .travis.yml
file more helpful to others, I have simplified the descriptions of Jaeger’s Node.js client library’s .travis.yml
file. The Node.js client library for Jaeger is open source and the current .travis.yml
file can be viewed here.
[1] The Node.js client’s current support for Node v0.10 and up also has an effect on which package manager we use. The Jaeger Node.js client uses npm, as yarn supports Node 4 and up.
[2] Travis CI is a continuous integration service used to build and test projects hosted on GitHub. For Jaeger’s Node.js client we have Travis CI run our tests on every pull request made to the GitHub repo. The .travis.yml
file tells Travis the programming language for your project and how to build it. Any step of the build can be customized.
Travis CI is free for open source projects on GitHub. ❤️