Automating browser tests in CircleCI
At Haus, our main product is a web application with a single-page (SPA) frontend and RESTful backend. We develop and deploy them from separate repositories and in different ways, but of course they’re highly related so we test them using end-to-end (E2E), browser-driven tests (in addition to unit and integration tests within each repo). We wanted to share how we’re doing this testing in CircleCI in case it helps anyone else out (and to get some tips from the community if you have a different approach!).
Repository structure and CircleCI builds
Our repository structure is fairly flat, but we’re not monorepo and we don’t yet have microservices. We broke out our applications into:
- Backend service
- Frontend client
- E2E test suite
- A couple assorted NPM modules for shared code
All of these are in private github repos, including the modules.
If you’re familiar with CircleCI’s project model, you know that they structure builds around repositories. The structure above gives us the nice quality that we can have unit/functional testing for each individual repo, but drive our E2E test builds from the test suite repo itself.
E2E build setup
Our automation build pulls down the other repositories and runs a test against the service and client set up locally, much like developers do on their own machines. There are definitely other approaches, but this one mimics and, more importantly, automates something that takes a while locally.
CircleCI doesn’t really have a native concept of pulling in multiple repos into a build, but they certainly give you the tools to do so should you need to. Most importantly, they inject sufficient Github credentials so that you can pull in all the repos you need, if you set up a user key for the project (in this case, our test suite project).
The main flow of the dependencies phase, with respect to the service and client repos is:
- Clone each repo.
- Run npm install for each repo.
- Do some project-specific setup (database migrations, Webpack build, etc.)
- Run a server for each.
A note on caching
Before we move on to other concerns, let’s just pause on step 2 above, running npm install. CircleCI has a cache in place for the main repository’s node_modules to save time from run-to-run, but it doesn’t know anything about the other two projects that we pulled down and configured ourselves.
However, we can tell CircleCI how to cache them! Namely, CircleCI lets you specify extra cache directories in your dependencies section. It’s a little tricky however because we need to clone our repos. Thus, we can’t use node_modules in our cloned subdirectories directly because it would interfere with cloning on subsequent runs, like so:
Run 1:
- Clone service repo
- npm install
- service/node_modules is created
- Cache service/node_modules
Run 2
- Cached service/node_modules is injected into the build environment
- Clone service repo…
- ❗❗❗ Collision on service/ directory already being there!
To avoid this, we simply create cache directories outside of the service/ and client/ and symlink them to (service|client)/node_modules after they are cloned.
Alternatives include specifying a different directory to npm on install or caching the whole repo directory and just doing a pull followed by an npm install. We went with the symlink approach.
Running background servers
Once we get our repos cloned, modules installed, and setup run, we need to run servers for our automated tests to interact with. These services take time to start however and we need to know when they’re ready before proceeding.
Luckily, we have simple, GET-able endpoints that we use for health checks in production to indicate the service is up! Thus all we have to do to make sure the servers are running is to poll these with curl.
This is what our circle.yml dependencies section looks like:
The run-client.sh and run-service.sh encapsulate the entire (clone ➡️ symlink ➡️ cache ➡️ npm install ➡️ setup ➡️ run server) steps for both.
Using the handy-dandy “background” tag lets us do an interesting pattern with the dependency post phase:
- We grab the latest version of Chrome first (blocking operation)
- Fan out, running the client, service, and webdriver in the background (non-blocking)
- Do a “barrier” operation, waiting for all the services to have a successful health check (blocks start of the next phase, which is test)
If you’re curious about the barrier script, it’s simple and ugly, taking advantage of curl’s -f option, which sets the return code based on the HTTP status returned in the call:
Why install Chrome?
As you may know, CircleCI has Chrome already installed in its build environment, so why install again? Because they have historically had a fairly old version and it appears to be somewhat difficult to upgrade since it’s in their base image. Thus we use a script to apt-get the latest Chrome before each build, ensuring we’re always testing against the latest evergreen release.
A quick note on headless testing on CircleCI
Just in case you’ve never done headless testing, or not done it on CircleCI, here’s just a bit of background on how that works. Whether you’re using the preinstalled Chrome or installing an update, CircleCI’s Linux image runs a framebuffer X server (xvfb) where Chrome just renders to memory. As we discuss below, we use a WebDriver-based test framework, called Protractor, which has utility called webdriver-manager. webdriver-manager is responsible for starting a Chrome instance and communicating to and from the instance to drive the tests.
This setup is really great for validating that the code works in a browser that human users will actually use, which is an advantage over PhantomJS and other purely headless browsers. That being said, it comes at the expense of higher setup costs (i.e. much of this article’s content!), so this is tradeoff we’ve chosen to make rather than a criticism of PhantomJS.
Test fixtures and database snapshots 📷
Our product has many different states and requires set up to get to many of them for end-to-end tests. It’s fairly straightforward in our unit tests, but it would be expensive and time consuming to run through the entire setup and explore all the possible branches, so we added some snapshotting features to our backend that allow us to save and restore database states.
To create a snapshot, the backend API has a POST endpoint /snapshots/:name (no body required) which runs a pg_dump script, the output of which is stored locally. To restore, the API has a /snapshots/:name/restore endpoint. We can create semantically named snapshots of the database at any point so the tests can reuse that state as fixtures. Of course these endpoints are only enabled during these tests… 😉
To get started, we have only one database seed on the backend itself that creates a single, admin user. The rest of the seeds are created by running browser automated scripts to generate data for the tests (and for our manual QA!).
Ok, we can, but should we?
The advantage of the browser-driven seeding approach is that you know that your seeds will be something that can be exactly produced by your system. In other words, if you’re missing an entry in a join table or something else in database-driven seeds, you might be creating a false positive or false negative in terms of overall system behavior.
The downside is that we’re running automation against a system before we’re testing it, so it may well make it harder to debug and diagnose problems if they show up in the seeding phase.
We’ve made this choice, but certainly understand if other folks disagree on philosophical grounds.
Running Tests
We use Protractor with mocha for driving our tests. This choice is a fairly non-obvious one, mainly because our frontend is written in React, not Angular, but we like it because it has a nice interface atop webdriver. Nightwatch would be another good choice for us, but we simply got started on Protractor first.
There’s not a lot to say about our testing itself, but what’s really important when running tests on CircleCI or via any other headless method is that you be able to see what was happening when a test fails. Thus we automatically take a screenshot and dump browser logs on every failure.
First we’ll need some utilities for saving screenshots and browser logs to disk:
Next we create a global Mocha “afterEach.” Mocha calls afterEach callbacks bound to an object with the current test state, so if we check that and the test failed, we take a screenshot and dump the logs using the test name (sanitized so it will actually write in case we described the test using non-filesystem safe characters).
What’s different?
So we do these tests to validate that the behavior of the system as a whole is operating as expected. We tend not to develop these tests using TDD/test-first methodology, so they end up being treated more as a regression suite than anything else.
All that being said, there are some differences between how these tests run and our actual deployment, namely:
- We use a fake S3 implementation that writes to disk rather than the real S3. There are some real concerns here about timing, i.e. writing to local disk takes far less time than to S3 and may affect browser test timing.
- Some of our user onboarding uses verification “challenges,” e.g. SMS tokens and email tokens. We implement these via generators and replace them in this test mode with deterministic implementations rather than the (hopefully!) random ones we use on the production site.
- There are some differences in the HTTP stack such as TLS (not used for tests), having a load balancer in front of the server, our static frontend server setup, file upload limits, CORS, etc. These are fairly non-trivial concerns that have bitten us in the past, so we’re careful about any changes that may affect those.
- We use Chrome exclusively at the moment as Firefox has issues with file uploads via webdriver, Safari would require us to set up an OSX/macOS build (which CircleCI supports, but we haven’t gotten around to it yet), and IE/Edge would require Windows. These get validated manually at the moment. 😦
With respect to the first two bullets, which replace one implementation for another, we lean heavily on our use of dependency injection. For more information on that, we’ve written about it previously and given a talk at our local SFNode meetup.
Wrap up
We know there must be better ways out there to do this, but we didn’t see a lot of in-depth articles describing the full setup, so hopefully this helps/inspires some of you. For everyone else, let us know how we can improve it! Thanks! 💜💜💜
If you’re interested in working with us, discussing issues like these and others, please check out our jobs page and get in touch!
