How we test WebGL on Continuous Integration
Tens of thousands plan their event layouts using Social Tables’ Diagram app. We render Diagram layouts in WebGL to provide the necessary performance to plan very large events. The problem is, testing WebGL is painful… and it becomes even more painful when you start using a continuous integrations services like CircleCI.
The test we are trying to run is an image comparison of the browser rendering of our app. The idea of image comparison testing is easy, you have an image you expect and your app generates another image to compare with your expected image by comparing pixel by pixel. In the world of WebGL, if you only care to run your app on the server side, you could use the headless-gl for your testing purpose. But most WebGL apps need to run on the client side and running WebGL tests on the client side is a lot trickier. There are three main problems we want to solve here.
- Setup a client side application to run WebGL under the test environment.
- Run WebGL on continuous integration services.
- Make the test run consistently on local and on server.
Setting up client side for testing
To setup a client side application we chose to use Electron. Electron uses Chromium under the hood which is the same thing that is used by Chrome. We can run our WebGL application in Electron, the same as we run on Chrome. We create the electron window by using
const win = new BrowserWindow({ show: false });to make sure we don’t turn on Electron window when we run the app. Since this is a image comparison test, we import our WebGL app to the test and make it generate images for us to compare. This sounds great, but WebGL is asynchronous so we don’t actually know when the render is finished on the frame buffer and ready for us to take a screen shot. To solve this we use gl.readPixels to force the rendering to complete on the frame buffer.
Running WebGL on Continuous Integration Services.
Most popular CI tools don’t provide a GPU. By default, WebGL will not work without a GPU Chromium environments. We need to pass --ignore-gpu-blacklist to Electron to made sure our WebGL application runs in non-GPU environment. We’ve also found it useful to print logging on the server console using --enable-logging, like this.
electron test/. --ignore-gpu-blacklist --enable-loggingMore info about ignore-gpu-blacklist can be found here.
Making the tests run consistently on local and CI
When you run a pixel comparison of two images you generate on local and on the server, you may find there are actually some differences. This could be because you are running WebGL on a different platform(if your local OS is different than the CI OS).
In addition, some WebGL extensions may not work under --ignore-gpu-blacklist flag. In order to make our test runs consistently on both local and CI environment, we choose to use Docker as a container to run our tests. To make Docker happy with WebGL, we need Xvfb to run X server on an hardware that lacks a display. We use shared volumes to gather our images generated inside Docker to our host/local. if you want to update your expected/base images or collect the result of comparison. docker.sh is here.
Xvfb -ac -screen scrn 1280x800x16 :9.0 & npm run test && cp -R ./generated-images/* /generated-images/The final step is to run your docker container inside CI service. Most CIs support docker these days. Here is the base example of how we run WebGL tests inside CI(examples provided for CircleCI and TravisCI).
This blog post would not exist without the help of my wonderful colleagues Dan MacTough, Conor Hastings, Joseph Van Drunen and Hunter Powers.
