React Full Stack Test Automation with Saucelabs and Circle CI

Plenty of great ink has been spilled over unit testing React components and Redux reducers et al, but we had a lot of difficulty finding meaty posts about settings up proper full-stack integration tests for the modern JS stack.

This is article 3 in a 3-part series:

• Nightwatch and API Calls for Clean, Tight, Isolated Full Stack Tests
• Gulp, AWS, and Terraform for Effortless Continuous Delivery
• Circle CI and Saucelabs to Automate All the Things

Our setup was particularly tricky because we have a handful of fully decoupled backend services and two React apps sharing components. If you’re going down a similar road, hopefully this series saves you a couple steps.

Saucelabs for Manual & Automated Cross-Browser Selenium Testing

At Parsable, we are building software for enterprise users, so we need to know our stuff works on a range of browser, OS, and screen size combinations. This means selenium rather than headless testing, and Saucelabs is our best friend both for manual and automated testing.

Fire up SauceConnect for Local Dev and CI Automation

Rather than maintaining a mess of VMs for local development, I can use SauceConnect to start a session with any combination of browser, OS, and screen size, but still pipe it right through to localhost on my machine.

We captured the connection script in package.json, so npm run sauceconnect is all it takes.

Similarly, we use SauceConnect to run automated tests from Circle CI. We had some DNS dragons to slay, but I will touch on that below.

Out of the box, Saucelabs allows you to playback the automated tests that were run and inspect the array of commands that were attempted. But as you write more tests, it becomes cumbersome to dig through all the test results. Thankfully, Aaron found a way to wrap the nightwatch tests in a way that posts the metadata we need, including pass/fail status.

Now we know what’s going on:

We can search by CI build and test id, search for test failures by platform, you name it.

Circle CI to Automate All the Things

I was using Circle CI the week they came out. They have a killer product…and their front end is written in ClojureScript no less! If you’re slogging it out with Jenkins, give these guys a shot. You will not regret it.

As usual, unit tests in Mocha worked out of the box. But getting full stack integration tests running required a little more finesse.

Spin Up Backend Services and Webpack

In order to run full stack integration tests, we need our API server running and connected to a seeded DB. We need our websocket signaling server running, and we need the front end running on a web server.

To facilitate this, we have a script responsible for each service. Circle allows you to specify directories to cache across runs, which saves a ton of setup time, since you do not have to clone everything fresh each time.

With scripts like the one above, we can reliably start background processes that run the supporting backend services we require. If we need a specific rev we can specify it as an environment variable. These scripts are then called in circle.yml thusly:

Run Tests in Parallel

Naturally, we want to take advantage of Circle’s parallel test-running, but Nightwatch and Circle cannot divide up the tests on their own (as they would with RSpec or Mocha, for example). So we need to get our bash on:

Basically, if you only have 1 circle container, run everything yourself. If you have multiple nodes, use the first one to run ESLint and Mocha. Then divide all your integration test files among the remaining containers. By piping the results to the appropriate location, Circle can aggregate the results when all is done.

For this to work, though, we need to be confident that we are not starting Nightwatch before the other services are ready. The wait-for-dependencies script achieves this:

For the background services, the job is easy. We just listen for a response on the expected ports. Webpack, on the other hand, is a bit trickier because the web server begins serving instantly, but it holds incoming requests until the assets are bundled. Tests will timeout while compilation is happening.

To get around this, we added a Webpack plugin to touch a temp file when the initial compilation is complete, and our wait script just looks for them.

We also ran into an absolutely devastating hazard where Nightwatch would periodically fail to create a session with Selenium. When we only had a few dozen tests, it didn’t happen often enough for us to worry about, but now that we’re running hundreds of tests each push, it’s a thing. Any time a session fails to initiate, it just hangs, and after 10 minutes without output, Circle kills it and fails. That means you discover the problem 10 long minutes after it happens.

This little bit of bash skullduggery wraps our call to Nightwatch, piping the logs to a temp file. If it doesn’t see “Running” in the log in the first thirty seconds, it kills the process and tries again. Problem solved.

Handling DNS with dnsmasq

Our users get their own subdomain for each of their teams, so during the integration test process we will literally need to create thousands of randomly-generated teams. We cannot add each subdomain to a DNS registry, so we use dnsmasq to map *.wi.dev to localhost. Thankfully, Circle CI has dnsmasq preloaded in their environment, so we just need to add the configuration to circle.yml and restart:

That’s just about it. I hope you found this useful. If you have any ideas about how we can make this even better, please don’t hesitate to drop us a comment.