Saving the 🐟 with 🐰 : how we used WebGL and Pixi.js for temporal mapping

Note: this post is about work I did when I was working at Vizzuality.

Image for post
Image for post
Illegal Chinese fishing vessels attempting to flee from coast guard (© Dong-A Ilbo/AFP ImageForum/Getty Images)

Our planet’s fish reserves are dwindling because of overfishing and illegal fishing — estimated at up to a third of the overall fishing worldwide. We need to understand what’s happening in our oceans. “We need to know who is fishing where. Until now, we were blind”.

The way we find answers: a map.

Image for post
Image for post
Here’s to maps, humanity’s cause and solution to all problems

Global Fishing Watch is a free and global monitoring system for citizens, journalists, NGOs, governments, fishermen and seafood suppliers. It will “make illegal fishing much harder”, by providing access to detailed information on almost 60,000 fishing vessels over the last five years.

Where is this data coming from? This article from The Economist sums it up neatly:

The International Maritime Organisation (IMO) requires ships over 300 tonnes to have an Automatic Identification System (AIS), a radio transmitter which tells anyone in the vicinity the boat’s position, speed and identity so as to avoid collisions. (…)

Global Fishing Watch, an online platform created by Google, Oceana, a marine charity, and Sky Truth, which uses satellite data to further environmental causes, is a keen user of AIS transmissions. They do not just let it locate fishing vessels; they let it take a good guess as to what they are doing (boats long-lining for tuna, for example, zigzag distinctively).

Skytruth has been doing tremendous work: building the early prototype, analyzing the gigantic dataset of AIS data and providing highly efficient binary vector tiles, where points are clustered both spatially and in time. More info on this can be found in Global Fishing Watch’s technical portal, globalfishingwatch.io.

We, at Vizzuality, had the chance to implement the vision on the user-facing side of things — building an interactive and animated visualisation on top of the dataset. I want to talk here about our discoveries, failures and successes achieving the animated “heatmap” rendering style of this map, with reasonably good performance, a maintainable, high-level codebase, and a bit of sanity left.

Early explorations: Canvas 2D

Image for post
Image for post

Canvas 2D is an API that allows you to draw shapes, text and images into a drawing surface. But really what it does, is just basically move pixels around. Instead of using the Graphic Processing Unit (GPU), the dedicated graphics unit of a computer or phone, Canvas 2D relies solely on the processor (CPU) to render graphics. Notwithstanding this, it can achieve solid performance, at least with that kind of pixellated rendering style. The technique, in a nutshell: prepare a typed array containing all pixel values (Uint8ClampedArray), dump those pixels into your canvas all at once (putImageData()). Done.

Image for post
Image for post

We started early prototypes of Global Fishing Watch with that approach, but we were aiming for a different rendering style. While the pixellated style works well for analysis, we were looking for something that would convey the idea of a “pulsating” activity, highlighting variations between fishing seasons and their potential impact on the environment.

Image for post
Image for post

To achieve this kind of heatmap effect, we need to blend many, many times “brushes strokes” (a tiny picture with a radial gradient gradually transparent) into the canvas, each brush representing here a point in time and space where a vessel sent a signal. We can then play with the size and opacity of the brushes to encode a variable such as fishing activity.

Can we do that with Canvas 2D? There are a few techniques to optimize rendering performance: “shadow canvas”; batch drawing API calls; grouping drawing commands by color/opacity; avoid sub-pixel rendering, etc. In our scenario, this wasn’t enough. Thanks, Obama.

I wish I knew earlier, but the truth is that this battle is basically lost in advance. There’s just no way a CPU can handle moving that many brushes (we’re talking tens of of thousands), without any kind of “offline” optimization, above sluggish speeds at best. Not on a desktop, let alone with a phone CPU.

A true “heatmap” style: Torque ?

Image for post
Image for post

Naturally an awesome contender when you think of spatiotemporal animations is CARTO’s Torque, which we used a few times in the past and continue using a lot in many of our projects.

Torque works by mashing SQL tables into preprocessed tilecubes, then rendered into a good ol’ Canvas 2D. It can deliver very good performance with most datasets, because there is a crucial step of spatial and temporal aggregation done ‘offline’. It is a smart way to tackle the problem, but unfortunately in this particular project we faced two major challenges:

  • because of the above mentioned pre-processing step, you can’t do instant client side changes to the rendering, which makes some interaction patterns, such as highlighting same-vessel points on mouse hover, harder to carry out;
  • we needed a great deal of dynamic interaction with timeframes. Changing the rendered time span is totally possible with Torque, but requires changing your SQL query and/or CartoCSS code, and by doing so causing a roundtrip with the server. But, we had high ambitions:
Image for post
Image for post

Meaning: changing the displayed time span of the fishing events dynamically on the client side.

Additionally, while Torque is fast and reliable when drawing points, we also needed lines to render the vessel trajectories, so going with Torque would have required us to use a separate solution for vessel tracks.

All hail WebGL

Image for post
Image for post
Ciao, Canvas 2D

We dropped all hope of using Canvas 2D, and went with a shiny new WebGL implementation instead. WebGL is fast because it’s tapping into the raw power of GPUs: do a lot of operations in parallel. It’s also very good at keeping your palms nice and warm.

On the programming side, WebGL is a wildly different beast to Canvas 2D. It allows your puny JS code to talk to the GPU through OpenGL Shading Language (GLSL), a language similar to C or C++. GLSL is very, very terse, difficult to debug, and hard to maintain. [whispered, sobbing voice] I’m afraid of GLSL. Can I go home now?

It turns out there are a bunch of very smart(er) people out there, doing the hard work for us: exposing GLSL functionality to a high level API in JavaScript, typically using some stage hierarchy paradigm — thinkcontainer.addChild(sprite); sprite.x = 42; (oooh the glorious days of Flash, may you rest in peace). We're talking about 2D rendering engines for the browser: Phaser, Pixi.js, HaxeFlixel, etc. Those libraries are typically used to develop games, but what’s stopping us from using them to track illegal fishing on a map as well?

So we could stay in the nice high-level-land of JavaScript, while leaving the GLSL logic to a tested and proven codebase. We could focus on highly maintainable, abstract code that ties well with our application model (Redux) and the rest of the UI rendering logic (React), while we rely on the “dirty work” being done by the rendering engine.

The question then is: how do you pick a rendering engine? Check out the project’s activity on GitHub, quality of the documentation, reputation? Yeah, sure, but more importantly: BUNNIES!

Image for post
Image for post
Australian rabbit population explosion, a graphical depiction

Yes, people use bouncing bunnies to measure an engine’s performance. Since Pixi.js can render and animate tens of thousands of bunnies on a canvas without breaking a sweat, it can surely render and animate tens of thousands of fishing events on a map.

Pixi.js provides ParticleContainer, an optimized container normally used to render effects that require a lot of visual objects, such as smoke, fire, debris, etc. It worked for us. We wanted to draw heatmap brushes to represent fishing signals, many times. The limitation of ParticleContainer is that while you’re allowed to play with the transform parameters of the objects, you can’t have multiple textures and tints — hence a few tricks I’ll explain later on.

As an added bonus, Pixi.js can fallback to rendering into Canvas 2D, for older setups (you might be surprised for instance, to learn that the Intel HD 3000 GPU, which equips most 2011 MacBooks, is on Chrome’s WebGL blacklist). But it turned out that the performance with Pixi.js’ Canvas 2D mode was actually tolerable, which is frankly remarkable.

Tinting and switching brush styles

At the lowest level, we have draw calls. A draw call is a set of instructions prepared by the CPU and sent to the GPU, and it’s usually one of the main bottlenecks when doing accelerated graphics.

One of the ways to reduce the number of draw calls is to limit the number of textures we use. Take for example this animation, showing French and Spanish vessels across the borders of the respective countries Exclusive Economic Zones. We needed different colors to distinguish Spanish and French vessels.

Image for post
Image for post

Instead of using two textures for the two brush colors, we had them share the same texture, using a technique called texture atlasing:

Image for post
Image for post

When rendering a sprite, instead of setting a texture per country/rendering style, we’ll just crop a portion of that big spritesheet, taking the relevant part for both hue and rendering styles (the solid circles on the right are used at higher zoom levels).

We end up with a single geometry, containing all sprites of the scene, using a single texture, which amounts to a single draw call. The only thing left to update are the sprites texture offsets, positions and sizes (in the GPU world, vertices UVs and transforms).

Compute graphical attributes ‘offline’

This is a costly operation, all happening on the CPU side:

  • fishing score must be translated to a sprite size and opacity, which also depends on the current zoom level;
  • latitude and longitude must be projected to coordinates on the screen.

The strategy here was to precompute these values right after a tile gets loaded, instead of doing it at each step of the animation. There are future plans to “bake” those calculated values into the tiles. Which, in fact is the more typical vector tiles scenario: tiles don’t carry meaningful data (geography, toponymy, etc), but mere drawing instructions, or purely geometrical data with tile-relative coordinates.

Image for post
Image for post

Map / canvas interaction

Our approach was:

  1. at tile load: project latitude and longitude to world coordinates
  2. at rendering frame time: convert world coordinates to viewport coordinates in pixels.

We found out that world coordinates made for a convenient and flexible “exchange format” in our scenario.

One thing worth mentioning too: it is not a good idea to keep recalculating viewport coordinates and rendering while panning or zooming. Although slightly more complex, offsetting the whole canvas while panning (and recalculating pixel coordinates offsets on map idle) was necessary to get acceptable UI response times.

Pooling

Each distinct point in our fishing map is represented by a sprite, which is a JavaScript object, before being fed to a WebGL shader. A naive approach consists in instantiating the number of objects you need at each animation frame, but we quickly realised that:

  • creating those objects is costly: animation framerate will drop drastically because of this;
  • deleting those objects (done automatically by the garbage collector): you get whole batches of frames skipped at unpredictable times.

Creating all those JavaScript objects is the cost of using higher level abstractions (in lieu of directly interacting with the GPU). You trade a little bit of performance for code expressiveness, which seemed appropriate for this project.

So to skirt that performance limitation we used the pooling strategy:

  1. make a generous estimation of how many sprites you will need (in this case the estimation depends on the time span selected and the viewport size);
  2. instantiate that number of sprites objects at once, store them into a pool;
  3. at each frame, position on the canvas as many sprites as you need to render that frame;
  4. keep the leftover sprites by just moving them off-stage, rather than deleting them;
  5. whenever time span or viewport size changes, instantiate more sprites (if needed).

Rendering tracks

Image for post
Image for post

Analyzing a vessel trajectory in detail gives insight into not only the “where” but also the “what”. As mentioned earlier, navigation patterns can give a pretty good idea about what type of fishing activity happened somewhere.

To render vessel tracks, we tapped into Pixi.js’s Graphics API, which uses either GL lines or triangles strips. With both modes, we are not yet entirely happy with the performance, so definitely a work in progress here.

The 🐘 in the room

One reason we went with a different approach is primarily that using Google Maps was an initial requirement, which virtually barred Mapbox from our options from the start. Add to that a completely custom GIS pipeline not designed from the start to produce vector tiles in formats consumable with Mapbox GL JS (compressed PBF files would likely fit the bill in our case).

In hindsight, I’m happy we built our own thing. It was an incredible learning journey, feeling like we were going through the last few years of advances in web maps rendering techniques, but in fast forward mode.

Plug

Special thanks go to Tiago, who’s an amazing engineer to work with, and who manages to deal with my grumpiness by being even grumpier. Thanks to Camellia, David and Rodrigo for reviewing this post.

Vizzuality Blog

Posts on data design, user research, open data, and…

Erik Escoffier / Satellite Studio

Written by

Emoji-based mapping, foolhardy data visualisation, frontend dev with duct tape. Cofounder satellitestud.io, front-end engineer at GlobalFishingWatch.

Vizzuality Blog

Posts on data design, user research, open data, and software development. We create tools and applications with a lasting benefit to society and the environment.

Erik Escoffier / Satellite Studio

Written by

Emoji-based mapping, foolhardy data visualisation, frontend dev with duct tape. Cofounder satellitestud.io, front-end engineer at GlobalFishingWatch.

Vizzuality Blog

Posts on data design, user research, open data, and software development. We create tools and applications with a lasting benefit to society and the environment.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store