Saving the 🐟 with 🐰 : how we used WebGL and Pixi.js for temporal mapping

Note: this post is about work I did when I was working at Vizzuality.

Illegal Chinese fishing vessels attempting to flee from coast guard (© Dong-A Ilbo/AFP ImageForum/Getty Images)

Our planet’s fish reserves are dwindling because of overfishing and illegal fishing — estimated at up to a third of the overall fishing worldwide. We need to understand what’s happening in our oceans. “We need to know who is fishing where. Until now, we were blind”.

The way we find answers: a map.

Here’s to maps, humanity’s cause and solution to all problems

Global Fishing Watch is a free and global monitoring system for citizens, journalists, NGOs, governments, fishermen and seafood suppliers. It will “make illegal fishing much harder”, by providing access to detailed information on almost 60,000 fishing vessels over the last five years.

Where is this data coming from? This article from The Economist sums it up neatly:

The International Maritime Organisation (IMO) requires ships over 300 tonnes to have an Automatic Identification System (AIS), a radio transmitter which tells anyone in the vicinity the boat’s position, speed and identity so as to avoid collisions. (…)

Global Fishing Watch, an online platform created by Google, Oceana, a marine charity, and Sky Truth, which uses satellite data to further environmental causes, is a keen user of AIS transmissions. They do not just let it locate fishing vessels; they let it take a good guess as to what they are doing (boats long-lining for tuna, for example, zigzag distinctively).

Skytruth has been doing tremendous work: building the early prototype, analyzing the gigantic dataset of AIS data and providing highly efficient binary vector tiles, where points are clustered both spatially and in time. More info on this can be found in Global Fishing Watch’s technical portal, globalfishingwatch.io.

We, at Vizzuality, had the chance to implement the vision on the user-facing side of things — building an interactive and animated visualisation on top of the dataset. I want to talk here about our discoveries, failures and successes achieving the animated “heatmap” rendering style of this map, with reasonably good performance, a maintainable, high-level codebase, and a bit of sanity left.

Early explorations: Canvas 2D

We already had experience building an animated heatmap with Canvas 2D, this time not to save fish, but forests, in a project called Global Forest Watch. This for instance shows the tree cover loss in the southern Amazonia from 2001 to 2015:

Canvas 2D is an API that allows you to draw shapes, text and images into a drawing surface. But really what it does, is just basically move pixels around. Instead of using the Graphic Processing Unit (GPU), the dedicated graphics unit of a computer or phone, Canvas 2D relies solely on the processor (CPU) to render graphics. Notwithstanding this, it can achieve solid performance, at least with that kind of pixellated rendering style. The technique, in a nutshell: prepare a typed array containing all pixel values (Uint8ClampedArray), dump those pixels into your canvas all at once (putImageData()). Done.

We started early prototypes of Global Fishing Watch with that approach, but we were aiming for a different rendering style. While the pixellated style works well for analysis, we were looking for something that would convey the idea of a “pulsating” activity, highlighting variations between fishing seasons and their potential impact on the environment.

To achieve this kind of heatmap effect, we need to blend many, many times “brushes strokes” (a tiny picture with a radial gradient gradually transparent) into the canvas, each brush representing here a point in time and space where a vessel sent a signal. We can then play with the size and opacity of the brushes to encode a variable such as fishing activity.

Can we do that with Canvas 2D? There are a few techniques to optimize rendering performance: “shadow canvas”; batch drawing API calls; grouping drawing commands by color/opacity; avoid sub-pixel rendering, etc. In our scenario, this wasn’t enough. Thanks, Obama.

I wish I knew earlier, but the truth is that this battle is basically lost in advance. There’s just no way a CPU can handle moving that many brushes (we’re talking tens of of thousands), without any kind of “offline” optimization, above sluggish speeds at best. Not on a desktop, let alone with a phone CPU.

A true “heatmap” style: Torque ?

Naturally an awesome contender when you think of spatiotemporal animations is CARTO’s Torque, which we used a few times in the past and continue using a lot in many of our projects.

Torque works by mashing SQL tables into preprocessed tilecubes, then rendered into a good ol’ Canvas 2D. It can deliver very good performance with most datasets, because there is a crucial step of spatial and temporal aggregation done ‘offline’. It is a smart way to tackle the problem, but unfortunately in this particular project we faced two major challenges:

• because of the above mentioned pre-processing step, you can’t do instant client side changes to the rendering, which makes some interaction patterns, such as highlighting same-vessel points on mouse hover, harder to carry out;
• we needed a great deal of dynamic interaction with timeframes. Changing the rendered time span is totally possible with Torque, but requires changing your SQL query and/or CartoCSS code, and by doing so causing a roundtrip with the server. But, we had high ambitions:

Meaning: changing the displayed time span of the fishing events dynamically on the client side.

Additionally, while Torque is fast and reliable when drawing points, we also needed lines to render the vessel trajectories, so going with Torque would have required us to use a separate solution for vessel tracks.

All hail WebGL

So after a good deal of false starts and hair pulling, this day finally happened:

Ciao, Canvas 2D

We dropped all hope of using Canvas 2D, and went with a shiny new WebGL implementation instead. WebGL is fast because it’s tapping into the raw power of GPUs: do a lot of operations in parallel. It’s also very good at keeping your palms nice and warm.

On the programming side, WebGL is a wildly different beast to Canvas 2D. It allows your puny JS code to talk to the GPU through OpenGL Shading Language (GLSL), a language similar to C or C++. GLSL is very, very terse, difficult to debug, and hard to maintain. [whispered, sobbing voice] I’m afraid of GLSL. Can I go home now?

It turns out there are a bunch of very smart(er) people out there, doing the hard work for us: exposing GLSL functionality to a high level API in JavaScript, typically using some stage hierarchy paradigm — thinkcontainer.addChild(sprite); sprite.x = 42; (oooh the glorious days of Flash, may you rest in peace). We're talking about 2D rendering engines for the browser: Phaser, Pixi.js, HaxeFlixel, etc. Those libraries are typically used to develop games, but what’s stopping us from using them to track illegal fishing on a map as well?

So we could stay in the nice high-level-land of JavaScript, while leaving the GLSL logic to a tested and proven codebase. We could focus on highly maintainable, abstract code that ties well with our application model (Redux) and the rest of the UI rendering logic (React), while we rely on the “dirty work” being done by the rendering engine.

The question then is: how do you pick a rendering engine? Check out the project’s activity on GitHub, quality of the documentation, reputation? Yeah, sure, but more importantly: BUNNIES!

Australian rabbit population explosion, a graphical depiction

Yes, people use bouncing bunnies to measure an engine’s performance. Since Pixi.js can render and animate tens of thousands of bunnies on a canvas without breaking a sweat, it can surely render and animate tens of thousands of fishing events on a map.

Pixi.js provides ParticleContainer, an optimized container normally used to render effects that require a lot of visual objects, such as smoke, fire, debris, etc. It worked for us. We wanted to draw heatmap brushes to represent fishing signals, many times. The limitation of ParticleContainer is that while you’re allowed to play with the transform parameters of the objects, you can’t have multiple textures and tints — hence a few tricks I’ll explain later on.

As an added bonus, Pixi.js can fallback to rendering into Canvas 2D, for older setups (you might be surprised for instance, to learn that the Intel HD 3000 GPU, which equips most 2011 MacBooks, is on Chrome’s WebGL blacklist). But it turned out that the performance with Pixi.js’ Canvas 2D mode was actually tolerable, which is frankly remarkable.

Tinting and switching brush styles

So WebGL is fast and all, but it didn’t mean that we didn’t have to be a little bit smart. When talking to a graphics API such as WebGL, what’s often “expensive” is not what’s happening within the GPU realm, but rather on the CPU side.

At the lowest level, we have draw calls. A draw call is a set of instructions prepared by the CPU and sent to the GPU, and it’s usually one of the main bottlenecks when doing accelerated graphics.

One of the ways to reduce the number of draw calls is to limit the number of textures we use. Take for example this animation, showing French and Spanish vessels across the borders of the respective countries Exclusive Economic Zones. We needed different colors to distinguish Spanish and French vessels.

Instead of using two textures for the two brush colors, we had them share the same texture, using a technique called texture atlasing:

When rendering a sprite, instead of setting a texture per country/rendering style, we’ll just crop a portion of that big spritesheet, taking the relevant part for both hue and rendering styles (the solid circles on the right are used at higher zoom levels).

We end up with a single geometry, containing all sprites of the scene, using a single texture, which amounts to a single draw call. The only thing left to update are the sprites texture offsets, positions and sizes (in the GPU world, vertices UVs and transforms).

Compute graphical attributes ‘offline’

So how did we actually update those positions and sizes? We had to get them from the raw tiles data, which contains the latitude and longitude of signal points, as well as two variables encoding a “fishing score” (obtained thanks to a trained fishing prediction model). Then, we have to convert this data to point and positions on the screen.

This is a costly operation, all happening on the CPU side:

• fishing score must be translated to a sprite size and opacity, which also depends on the current zoom level;
• latitude and longitude must be projected to coordinates on the screen.

The strategy here was to precompute these values right after a tile gets loaded, instead of doing it at each step of the animation. There are future plans to “bake” those calculated values into the tiles. Which, in fact is the more typical vector tiles scenario: tiles don’t carry meaningful data (geography, toponymy, etc), but mere drawing instructions, or purely geometrical data with tile-relative coordinates.

Map / canvas interaction

While the dataset provides latitudes and longitudes, which represent a position on the surface of a sphere (angles), they have to be projected on a 2D plane first, aka “world coordinates” (using the Web Mercator projection), then converted to screen coordinates, all on the client side.

Our approach was:

1. at tile load: project latitude and longitude to world coordinates
2. at rendering frame time: convert world coordinates to viewport coordinates in pixels.

We found out that world coordinates made for a convenient and flexible “exchange format” in our scenario.

One thing worth mentioning too: it is not a good idea to keep recalculating viewport coordinates and rendering while panning or zooming. Although slightly more complex, offsetting the whole canvas while panning (and recalculating pixel coordinates offsets on map idle) was necessary to get acceptable UI response times.

Pooling

One other trick we used is probably almost as old as computer graphics: object pooling.

Each distinct point in our fishing map is represented by a sprite, which is a JavaScript object, before being fed to a WebGL shader. A naive approach consists in instantiating the number of objects you need at each animation frame, but we quickly realised that:

• creating those objects is costly: animation framerate will drop drastically because of this;
• deleting those objects (done automatically by the garbage collector): you get whole batches of frames skipped at unpredictable times.

Creating all those JavaScript objects is the cost of using higher level abstractions (in lieu of directly interacting with the GPU). You trade a little bit of performance for code expressiveness, which seemed appropriate for this project.

So to skirt that performance limitation we used the pooling strategy:

1. make a generous estimation of how many sprites you will need (in this case the estimation depends on the time span selected and the viewport size);
2. instantiate that number of sprites objects at once, store them into a pool;
3. at each frame, position on the canvas as many sprites as you need to render that frame;
4. keep the leftover sprites by just moving them off-stage, rather than deleting them;
5. whenever time span or viewport size changes, instantiate more sprites (if needed).

Rendering tracks

Analyzing a vessel trajectory in detail gives insight into not only the “where” but also the “what”. As mentioned earlier, navigation patterns can give a pretty good idea about what type of fishing activity happened somewhere.

To render vessel tracks, we tapped into Pixi.js’s Graphics API, which uses either GL lines or triangles strips. With both modes, we are not yet entirely happy with the performance, so definitely a work in progress here.

The 🐘 in the room

Why yes, I managed to write a whole post about maps and WebGL, in 2017, without even mentioning Mapbox. Mapbox basically brought vector tiles and GPU accelerated maps to the wider mapping community (including non tech). They host OSM-based or custom data vector tiles that are delivered through a JS client (Mapbox GL JS) or native clients for mobile. The “holistic” approach of the map stack allows for stunning maps, with smooth transitions, client-side restyling, multilingual labels, etc, all rendered by the GPU, usually at a good solid 60 fps for regular use cases.

One reason we went with a different approach is primarily that using Google Maps was an initial requirement, which virtually barred Mapbox from our options from the start. Add to that a completely custom GIS pipeline not designed from the start to produce vector tiles in formats consumable with Mapbox GL JS (compressed PBF files would likely fit the bill in our case).

In hindsight, I’m happy we built our own thing. It was an incredible learning journey, feeling like we were going through the last few years of advances in web maps rendering techniques, but in fast forward mode.

Plug

Vizzuality is hiring! If you’re an engineer who wants to support social good with your skills, get in touch!

Special thanks go to Tiago, who’s an amazing engineer to work with, and who manages to deal with my grumpiness by being even grumpier. Thanks to Camellia, David and Rodrigo for reviewing this post.