Visual Diffing MapboxGL
“Startup mentality” is one of those phrases that can be used to explain any hasty decision — right alongside “deadlines”, “reality” and “there’s a happy hour in 15 minutes and I really would like to be there on a Friday evening rather than stuck here worrying about writing this feature using test-driven development which is the ‘right’ way to do things but really after all what is ‘right’ and what is ‘wrong’ and is this really so ‘right’ in the context of my situation right now much like some ethical thought experiment involving food scarcity and cute animals.” Whatever the reasons, there’s always some looming best practice standing over your shoulder, judging you at every turn. And really, that’s kinda lame, because it shouldn’t loom, and it shouldn’t judge.
So, where does visual diffing fit into this? Visual diffing is an automated application testing strategy that captures screenshots of your application and comares the visual differences between the status quo and proposed changes to code. In our case, visual diffing is simply a solid strategy for preserving existing behavior for poorly-tested production applications when needing to inch forward — either stemming from wanting to add features or even going back and adding tests.
This presents a dangerous catch-22: back-filling your applications with tests sometimes requires refactoring your source code to make it unit-testable. But if refactoring means “changing the code without changing the behavior”, and you don’t know whether the behavior has changed or not, you’re effectively stuck. Because my team definitely had a “startup mentality” in the first year, we churned out dozens of products with limited testing. In our second year, we’re feeling the pain of refactoring untested source code.
Visual diffing was important to use for two reasons:
- MapboxGL is a huge dependency and is hard to test
- The source code was largely untested
For this, we used Percy, which made it very easy to implement visual diffing in our workflow. It also provided a great user experience for the manual reviewers who were inevitably checking for the same issues each time.
Screenshotting WebGL
WebGL is a powerful technology for churning out lots of complex visual information in the browser. It is, however, very hard to test. It also creates some snags when trying to implement screenshot diffing. Because Percy works by shipping local HTML content back to its service, it misses whatever is in the canvas element. No good!
Enter toDataURL(). Because WebGL renders to a canvas element, we can periodically generate images and replace the element’s contents with a static representation of the canvas. We can even trigger this behavior specifically at important moments of MapboxGL’s event system. For example, we should trigger a new image each time MapboxGL triggers a “render” event.
Finally, here’s what you see in the visual diff:
In this case, our visual diff didn’t catch anything significant. The red splotches in the bottom right indicate a few labels appeared where they didn’t appear before, but we can chalk this up to some slight differences in rendering time.
On other occasions, visual diffing caught some very important changes — huge amounts of difference indicated by tons of red. This flagged to me that something had changed with our hard-to-test map. This was great because it failed to pass a visual sniff test — a behavior that would have been caught probably later on in production.
Overall, image generation solution works fairly well. The map seen in the image above is actually just a regular PNG image generated by a stub I created. A screenshot of a screenshot of a screenshot!
Check out the full source of the stub here:
