Moving from Karma to Jest with a React-based UI

In the throes of Karma woes

For the last two-ish years we’ve been using Karma as our Javascript test runner. On paper, testing the frontend in a real browser is a great idea. Karma actually runs the tests against the production build in a real browser and would potentially find discrepancies across different browsers. At first it worked ok, but then we had all sorts of problems.

In order to use Karma we needed Webpack because the codebase used ES6 and JSX which needed to be transpiled to run in browsers. Initially, the build times were a few seconds. Unit tests were usually no problem because the builds only included a few files. But integration tests that would load the whole page would take 1–2 mins. Thankfully, Webpack can build and test bundles incrementally preventing Karma from being completely unusable.

After writing hundreds of tests, including ones with functional integration tests, the Node.js process invoking Karma started running out of memory. How is this possible you say? Well Karma and Webpack were building bundles for each test. Each test would have 1, 2, 500 dependencies potentially. We could have increased the max RAM usage (~1.7GB) for a single Node process by having a wrapper script increase the limit via some flags. But sooner or later we would keep hitting the ceiling and we’d have to address optimization issues.

One way to reduce the build sizes was to disable source maps, which provide mappings from the build code to the original source code. Source maps are very expensive and they heavily increase the bundle size and load down the test execution time. The big downer to disabling source maps was it prevented us from getting a nice stack-trace when errors were thrown. Obviously, this sacrifices the ability to quickly debug any problems in the test.

Another way we reduced build size was to use the require.context feature in Webpack when configuring Karma on which tests to run. The way I understand it, using require.context builds a single bundle that imports all the individual tests, whereas the previous method built many bundles with a single entry-point each. This issue might help explain better. But building the test bundles using require.context created an alternate bundle that deviated from production. We even had issues of circular dependencies only manifesting in the tests, but not in the real UI.

Debugging tests was frustrating with Karma. Very often the full browser console output where the code was running did not make it back to the Node process where Karma initiated everything. Source maps seemed to break regularly. And if runtime exceptions occurred the logs would show the exception (maybe) and the test process would exit prior to logging the test that was actually running. So every time an error like this occurred it was annoyingly hard just to figure out which test failed.

Debugging through the spawned browser process also had issues, but it was the only way to get better logs. So I would click this “DEBUG” button and then open the console in the dev tools. Literally half the time I did this the browser would get totally locked up and crash. (Yay).

Since the test suites on Firefox took a few minutes we never could justify testing in other browsers as part of continuous integration. And to be honest, I’ve never noticed a single browser related bug get caught anyway, YMMV. I realize that Karma can be configured to run against jsdom running in Node, but when we tried it, we experienced a lot of the same issues.

I don’t want to paint an all bad picture of Karma. It was written relatively early in the Javascript framework explosion when module building was an after-thought. Karma runs much better when it can bootstrap native ES5 code without dealing with many large bundles. I don’t remember having these issues when writing tests for an Angular 1 app. But things have changed. Most apps need a build process just to run in any environment nowadays.

Meet Jest

Since we use React heavily, Jest seemed like a good substitute. Early Jest was not so great. But then the React team totally revamped it with some awesome features. The biggest thing is its efficiency and stability. Since all the extraneous code like the CSS modules, fonts, and images were being mocked, only the Javascript was really being compiled and run.

Migrating to Jest was relatively painless. Our test code used Jasmine for the assertion and test block syntax and was mostly compatible with the Jest’s syntax.

Setting up the configuration was more time-consuming because we had to configure Jest to be able to compile everything with Babel. Jest doesn’t support direct integration with Webpack, but it turns out this is a serious advantage because Jest just focuses on the Javascript and not the static assets or CSS. Because of this, we also had to tell Jest to mock all of the static assets and to provide a mapping for all the Webpack aliases in the module imports. Jest also supports to ability to write custom transformers that can specify how assets are transformed during the build process like creating empty classname lookups for the CSS modules. Jest has a helpful guide on their site for apps that use Webpack.

Another fantastic Jest feature is the test change detection and parallelism. If I change only a single file, as I often do when debugging a test, Jest is smart enough to only rerun the file affected by the change. If multiple files change then multiple parallel test executions occur. We were able to shave about 5 minutes off our CI build time.

Best of all is Jest’s solid logging and stack trace reporting. And the errors happen after the current test name is logged, not before. It even includes some CLI graphics (oxymoron?) with pass/fails that clearly stand out.

Snapshotting is another big feature of Jest. It can spit out the rendered HTML of a given test and verify it matches a known-good snapshot div-per-div. However, the minute someone changes anything about the component that affects the rendered markup a new snapshot has be to made and committed. Jest makes this process quick and easy, but brittle tests usually cause more harm than good in my experience. Plus, style changes within the CSS classes won’t get caught because they’re mocked out. Though, I can imagine projects that live and die by small UI changes, like a checkout page on an e-commerce site, could really receive the benefits of snapshotting.

I’m sure lots of developers have been able to get Karma and other test runners out there like Mocha working just fine for their organization. And due to time and resource constraints, I haven’t been able to take a super scientific approach to measuring, comparing, and optimizing them. But I can tell you Jest is a top contender worth considering.