Level Up CSS Regression Testing

Source: Photo by Samuel Zeller on Unsplash

Docker + Gemini

Last April, I went to the Docker for Angular developers talk at NG-CONF. Having previously worked on a team that used Docker to put our app servers and databases into containers it was a nice review of one of the ways that containers are changing things for all engineers, no matter where you hang out in the stack. As the talk progressed my mind wandered to “What if …” and “How else …” questions for other uses of Docker for engineers who only focus on the front end.

Don’t get me wrong, using Docker to make it dead simple and easy to onboard a new front end engineer is great. But with my work on Clarity I wasn’t focused on the middle tier or back end any more. After I left the conference, the Docker talk stuck with me but I didn’t have a practical use case to work with. Then we made a goal to optimize our CSS regression testing and an opportunity presented itself.

Testing Issues

Source: Photo by chuttersnap on Unsplash

In the code, building the code, testing the code … everyone’s software has issues. Things that you would love to improve but probably not today while fixing that bug or finishing the latest shiny new feature. We felt that way about our initial Gemini setup. It worked well but there were two things that caused us pain:

1. Travis: we had set our project up in such a way that we needed to use Travis to run our tests and generate our reference images.
2. Our setup made it difficult for external contributors to make changes that required updating or creating new reference images.

The main issues that cropped up were false positives resulting in failed pull request builds (and occasionally on master 😱). And one of the core team members had to use an external pull request to run a job on Travis so that we could give an external contributor any new images they needed to turn their pull request into a green build.

Thinking about these issues, my mind kept returning to the Docker talk at NG-CONF and my “How else …” questions. This seemed like a good place to test my theory that Docker could be used to make the development experience better for engineers focused only on the front-end of the stack.

A Clear Solution

Source: Photo by elizabeth lies on Unsplash

The solution I settled on was to use the selenium/standalone-chrome image available on Docker Hub. Because of the awesome docker-selenium project and the amazing volunteer contributors I didn’t have to create my own Docker image with selenium-standalone installed and ready to use. What I did have to do is wire up the use of Docker with our build scripts and make it easy for a front end engineer who had never used Docker before to run our Gemini CSS regression suites on their own host.

First, I modified our build that already ran the regression tests to use the hosts IP address when it started Gemini. Next, I had to make it easy to start the Docker image, build Clarity, run our tests and then stop Docker. A simple node script made it easy to leverage the small change to our Gemini build task above and wrap it into Clarity Docker control script. Finally, to make it easy for folks who just wanted to run the Gemini suite but were not familiar with Docker or Gemini, I added npm scripts that would run the appropriate command for testing or updating.

Now anyone who needs to work with Clarity’s CSS Regression testing can run one of a few simple npm commands which I outlined here.

Lessons Learned

Source: Photo by The Climate Reality Project on Unsplash

Not everything was roses. There were two big take-away’s from my experience:

1. Spend time on your Gemini tests to capture the smallest possible image. This will go a long way towards eliminating false positives and making your test suite more robust.
2. Lock down the Docker image version you are using.

Some context for item number two.

My first attempt used

$ docker run (:dash::dash:)rm (:dash::dash:) name=clarity_chrome -d -p 4444:4444 selenium/standalone-chrome

to start the docker image for testing. After this was merged to master we quickly ran into problems with our reference images because this used the latest image on Docker Hub. So the reference images I generated were different from the reference images my teammates were generating and testing against. Investigation revealed that Docker image we run is occasionally updated with a newer version of Chrome. This meant that when someone used an image with a newer version of Chrome there was a high probability that some of their test images would have subtle differences and fail for seemingly no reason.

Once I figured out what was going on, it was easy to lock down the specific version of Docker that you use to run for testing (note the sha256):

$ docker run (:dash::dash:) rm (:dash::dash:)name=clarity_chrome -d -p 4444:4444 selenium/standalone-chrome@sha256:b899f16b6d963600ef6da8a8dd49e311146033ed66cb5af71eccb78ab378e19a

The sha256 value is a computed SHA256 hash of the image configuration object, which contains the digests of the layers that contribute to the image’s filesystem definition.

What that means is that its unique for that set of changed. By specifying the image sha, we can lock down the exact docker image that all environments will use when running these scripts. It doesn’t matter when you run it or where you run it. The exact same image and thus the exact same browser will be used to run the gemini suite of tests.

Optimize Continually

Source: Photo by Greg Willson on Unsplash

Of course, it could be better. While figuring out how to wire everything up I made notes for a couple of things that we could try to speed up and improve. These are the top areas I believe could help optimize our CSS regression testing in Clarity:

1. Modify the Clarity Docker control script to use both the selenium/hub image and configure the stand-alone image to talk to it so that I can run all of the test suites in parallel
2. Number 1 also lays the foundation of using the pre-baked selenium/standalone-firefox image
3. Building our own Docker image with IE11/Edge

Gemini is a great tool for maintaining style consistency in a client side application. Docker is a great tool that isolates the underlying software dependencies and allows for a shared system that ensures everyone is running the same exact software across different host computers.

Together they enabled us to optimize our CSS regression testing in the Clarity project.