From Karma to Mocha, with a taste of jsdom
This is the story of how frontend engineers at Podio improved productivity, developer experience and reduced technical debt simply by ditching Karma and Jasmine in favour of Mocha and jsdom.
Read on for the full story…
Testing is hard — I’m not saying anything new here. Tons of people have written about it. Explaining the ins an outs of testing, why it’s hard and what you can do about is not what I want to talk about.
What I do want to talk about is how hard testing gets when your test suite takes more than 10 seconds to bootstrap, only to run a 10 ms unit test.
It starts off as an annoyance, it evolves as a huge amount of wasted productivity, and in the end it becomes downright impossible to do some decent Test Driven Development without going crazy.
This was our situation at Podio, and it wasn’t fun;
so we decided to change it.
The Problem
Podio is a huge codebase, and the frontend is no exception: the first of 38,000+ commits date back to Dec. 7 2010.
Our testing setup was based on Karma as test runner, Jasmine as BDD and assertion library and PhantomJS as a headless browser for testing, and a sprinkle of Grunt magic on top of it.
In order to be able to run the first test on this setup we needed to bundle all our tests and prepare them for Karma, load grunt-karma task, spin up Karma, it’s launcher for PhantomJS, and of course, PhantomJS itself.
Look at this screenshot of a single run.
What do you see?
Yes, you’ve read that picture correctly.
A whooping total of 10.861 seconds, in order to run a 0.007 second test.
Sure, we could have used a task like grunt-contrib-watch to avoid restarting the entire setup every time, and we tried, but for one reason or another it was never stable enough for us to use it. I’m talking about watchers dying after a couple of runs, multiple instances of PhantomJS being spawned and random crashes.
We wanted to go truly TDD, writing tests before or along side the development, but this was totally impossible with this kinds of issues.
Can you imagine waiting 10 seconds every time you change a line of code?
Dependency leaks and technical debt
Another big problem with this process was the intrinsic leakage of our dependencies and globals all over the place, which, among other things, over time had a significant impact on the growth of our technical debt.
Due to the fact that all our tests were meant to run in one browser instance, the bundling of those tests was causing the dependencies to be shared, which caused all sorts of headaches when trying to fix our dependency graph.
Way too many parts of legacy code were assuming some libraries as global (like jQuery, moment or underscore), and since everything was bundled together, PhantomJS never complained about unresolved dependencies.
This was a side effect that was effectively preventing us from exploring different bundling strategies in production.
The Solution
We love Mocha and Node.js, and we wanted to have the same amazing experience on our test suite. However, the question we struggled to answer was this: how we could ever run tests for components that were conceived for the browser, in a Node.js environment, while reusing our RequireJS configuration and module loader?
jsdom: A JavaScript implementation of the WHATWG DOM and HTML standards, for use with Node.js.
We had to come up with a small module called node-requirejs which loads the modules’ configuration, replaces Node’s require, and allows us to reuse the same exact configuration that our web app is using. Of course it adds a little bit of extra time, as we will see later on, but it gives us the consistency we need between tests and production.
The last step needed to get rid of PhantomJS was an easy way to create, reset and destroy DOM instances that were isolated, encapsulated and running on NodeJS.
jsdomify was created, and it was doing exactly this.
With a very simple API, jsdomify lets you create, reset and destroy a DOM instance that will be exposed as a global variable using jsdom and behave exactly like a real browser. Only much faster.
A simple implementation of testing a React.js component now goes like this:
The key for isolation here is that the DOM instance gets created only at evaluation time, and before React gets included in the “page”.
It then gets destroyed after all the tests have been completed.
If we want complete isolation during tests, we could reset the instance before each new test, for a clean start.
beforeEach(() => {
jsdomify.clear();
})The Outcome
a.k.a. showMeTheNumbers
It took us almost 5 months of work as a side project to migrate our entire test suite and fix all the mistakes we’ve made in the past.
At first glance we underestimated the necessary effort, as none of us had a clear picture of how much technical debt we had to dig up to reach the finish line, and boy it was a lot!
The final results were impressive though, and definitely worth the effort!
For brevity, I will call the combo Karma/Jasmine/PhantomJS, Karma, and the Mocha/jsdom one, simply Mocha.
Isolation
- Mocha: every test is run in isolation. Every unresolved dependency is throwing an error. No two components are tested in the same DOM, unless specified.
- Karma: no isolation here. Everything is bundled in the same page and everything leaks all over the place
There’s no match here, one has it and the other doesn’t.
Full test suite run
- Mocha: 1131 tests run in 30.017s
- Karma: 842 tests run in 20.334s (1131 tests in approx 27.313s)
At first glance Mocha is slightly worst here, but the catch is that Mocha is destroying and creating a new DOM instance for every single test file, and we have 150 of them! Pretty impressive, huh?
Single test run (speed)
- Mocha: a single test is running in 2.436s, 2.009 of which are used to setup the RequireJS environment.
- Karma: we’ve already seen this, but to recap, a single test is running in 10.861s, while the unit test itself only takes 0.007s
Well, if we wanted to play with numbers here, we could say that with Mocha we had a performance increase of 445% on a single test run.
Which is not that bad.
What I also want to highlight is that creating a new DOM instance for every test file does have a cost, which can be quantified in roughly 3-400 ms.
This delay is hardly perceivable in any real use case scenario though.
Single test run (ease of use)
- Mocha: mocha has an unbeatable feature here that lets us pass grep parameters to only run specific test(s).
Karma style [describe/it].[only/skip] are of course available, but only come in handy when the selection is final (e.g. for skipping tests).
- Karma: the only way of achieving this in Karma (Jasmine) is by using the iit (ddescribe) or xit (xdescribe) notation, which requires us to actually change the source code.
Mocha’s ease of use when it comes to running single tests or specs is unmatched, and it adds so much to the developer experience that it’s a major selling point.
Watchers
- Mocha: mocha has a built-in blazing fast watcher that can be added to any other parameter, making life very easy when it comes to TDD.
- Karma: on the other hand, Karma doesn’t come with this functionality. The only way to add it is by using some grunt/gulp/broccoli plugin on top of it, which may or may not work as expected. In our case it was grunt-contrib-watch, and it wasn’t performing very well at all.
Conclusions
We are extremely happy of the new Mocha/jsdom combo, for all the aforementioned, the tech debt we’ve cleared out, and all the learnings we made throughout the entire process.
If your codebase is big and “old” this could be a daunting and time consuming process, but in the end you and your team will enjoy the huge benefits of it. Don’t be afraid of your technical debt, pay it back, enjoy renovations, and fight for your code.
Don’t be swallowed by it.
If you liked this story, hit that little heart down there and don’t forget to tweet about it. While you’re at it, why not follow me on Twitter?
I might have something more to say!
