Don’t be afraid of headless Chrome! Learn why and how to use it for Ember testing
Last updated 9/1/17, Ember CLI 2.15. Special thanks to Scott Newcomer and Ben Demboski for helping me to debug, Karl Becker for editing, and Tobias Bieniek for some CLI tips!
After a few hours debugging my EmberJS testing suite that refused to run from the command line, I ditched PhantomJS and switched to Headless Chrome. Let’s break down what it all means, how I did it, and what the impacts are.
What even is Headless Chrome?
Ember developers have options for which browsers they use to run their testing suite, and Headless Chrome is one of them. But what is it? The Google blog says:
It’s a way to run the Chrome browser in a headless environment. Essentially, running Chrome without chrome!
Right. Here’s a better definition from Wikipedia:
A headless browser is a web browser without a graphical user interface. Headless browsers provide automated control of a web page in an environment similar to popular web browsers, but are executed via a command-line interface or using network communication.
Starting with Ember CLI 2.15, headless Chrome is the default for testing in EmberJS. If you’re working with an older Ember app, I have good news - you don’t need to upgrade your app in order to try out headless Chrome. As a matter of fact, you can use the latest CLI release with almost any older Ember app version.
Why is a “headless” environment needed?
Just like a regular browser, a headless browser understands HTML and CSS. It can execute JavaScript like AJAX requests. Think of acceptance tests in Ember. If a test tries to click a button that is hidden, it shouldn’t be clickable, and the test should fail. But how is that known? Because the browser does the heavy lifting of combining all the HTML, CSS, and JavaScript into something useful. And, since there are no images to be displayed, testing is faster in a headless environment. There are many different kinds of headless browsers. Chrome and PhantomJS are just two examples.
So, why not just run the tests directly in a normal Chrome browser? If you have an Ember app, run ember serve, and visit http://localhost:4200/tests, you can actually watch your tests run in real time, or pause them and look at the app state visually. However, where headless browsers really shine is when they are used for Continuous Integration testing, commonly referred to as CI. In production apps, it is common to use a service that automatically runs your test suite as code is committed. And for the most part, those tests run on a server, not in “normal” browsers. For example, check out the open Pull Requests for part of the Ember website, especially any with a red x next to them. Whenever a pull request gets opened on GitHub, you can see whether it passes the tests.
Why not use PhantomJS?
PhantomJS is another example of a headless browser. Creating and maintaining it was a Herculean task, and its success is why we have nice things. Building apps is hard… can you imagine building a whole browser??? But it would seem that it’s on its way out. One maintainer stepped down in April 2017, saying:
Headless Chrome is coming. I think people will switch to it, eventually. Chrome is faster and more stable than PhantomJS. And it doesn’t eat memory like crazy. I don’t see any future in developing PhantomJS.
As the maintainer noted, PhantomJS has some problems. I had one of my own: none of my tests would run. Everything was fine in a fresh app, but some unknown part of my real app wasn’t compatible after introducing some well-established dependencies. I ran ember test, but before any tests had even begun, I was greeted with these errors:
not ok 1 PhantomJS 2.1 — Global error: SyntaxError: Unexpected token ‘}’ at http://localhost:7357/assets/vendor.js, line 120177not ok 2 PhantomJS 2.1 — Global error: Error: Could not find module ember-metal required by: ember-testing/support at http://localhost:7357/assets/test-support.js, line 58not ok 3 PhantomJS 2.1 — Global error: ReferenceError: Can’t find variable: define at http://localhost:7357/assets/ember-bio-bright.js, line 5not ok 4 PhantomJS 2.1 — Global error: ReferenceError: Can’t find variable: define at http://localhost:7357/assets/tests.js, line 3not ok 5 PhantomJS 2.1 — Global error: ReferenceError: Can’t find variable: EmberENV at http://localhost:7357/4215/tests/index.html?hidepassed, line 38
I threw everything at this error. Blowing up node modules, removing all but the simplest test, reinstalling the EmberCLI, installing/uninstalling PhantomJS, digging into the vendor bundle, sharing angry cat GIFs, lighting some incense… nothing.
After a little bit of Q&A with a couple other developers, it was suggested that I try out Headless Chrome to see if the errors became easier to debug.
The errors did not become easier to debug.
The errors simply disappeared.
How to make the switch
There’s a file in Ember apps called testem.js, and that’s where you configure which testing tools to use when you type ember test or ember test —-server. Here’s a link to the testem.js content that I ended up using, copied and pasted from the article by Ryan Toronto. You can view the testem file shipped with EmberCLI at this link on the EmberCLI GitHub.
What is Testem?
Testem is a test runner, meaning that it loads and runs your app’s tests, using the configuration you specified in testem.js. You also get a friendly interface to see the results of your tests from the command line. I was surprised to learn that Testem isn’t unique to Ember. It works with many JavaScript frameworks, testing tools (like QUnit, Mocha, and Jasmine), and browser environments.
What negative impacts could Headless Chrome have?
Well, for one thing, Headless Chrome isn’t open source like PhantomJS. The pros and cons of that could be their own article.
It’s also new-ish. It was shipped with Chrome 59, but developers have had ways to use Chrome before that. The big question is, if you’re already doing CI testing, how well does your vendor support it? A lot of the big players have been quick to implement it, but it’s possible you might run into some issues.
Finally, you might have some tests that pass in PhantomJS and fail in Headless Chrome, which means it’s time to do some debugging. Of course, I experienced the reverse, where all my problems magically went away when I switched headless browsers, but problems are not uncommon. This is a pain point akin to things looking great in Chrome but they explode in Firefox. For this reason and to make their tests more robust, some organizations run their tests in multiple headless tools.
When to switch
If you’re feeling stuck, or simply want to know what the latest and greatest is like for Ember testing, give it a try!
Happy Headless Testing!
