Tracing the flow of coffee and soy
An engineer’s insight into building a new tool to analyse international trade flows
Note: this post is about work I did when I was working at Vizzuality.
Since August, I’ve been working on Trase, a brand new tool developed by the Stockholm Environment Institute (SEI) and the Global Canopy Programme (GCP) that will lay bare the flow of agricultural commodities through the global trade system. From production to consumer, Trase reveals the producers, traders and transporters involved in the international trade of products like coffee, soy, and palm oil. The datasets that hold all of this information are immense and the relationships between the data points are complex, but it’s these qualities that have made working on Trase so fascinating.
Design
Upon joining the project, I was immediately impressed by SEI’s ambitious vision for the data visualisation we were going to build together. There was a plan to add even more features to an existing prototype, and it was clear that the final product would be very complex in terms of interactivity. Thus, one of the main challenges of the project was how to keep the balance between letting people explore the incredibly extensive dataset and not drowning them in too many overlapping features.
During a sprint with SEI and GCP, everything was analysed to identify the most important features. It was tough at times but we resisted the urge to cram in as many interactive features as possible. We know from experience that adding something that’s useful to only 1% of users can harm the experience of the other 99%. Beyond these discussions about functionality, we also needed to merge features seamlessly, find shortcuts for the users, and design the best way to progressively introduce users to more complex features.
Rendering
When it comes to choosing a library to render a complex data visualisation, the choice is usually quite obvious: D3. D3 has been around since 2011 when it was built by Mike Bostock while he was at the New York Times. Since then, it has matured into an incredible piece of technology that allows developers to build interactive visualisations with an outstanding level of flexibility.
Trase uses the latest iteration of the library (version 4), with a completely customised layout engine for the sankey diagram that visualises the flow of commodities from producer to country of import. It’s pretty amazing what you can do with D3. To illustrate its power, we went from a static sankey diagram to a fully animated one in just a few lines of code. The sankey diagram can be completely controlled by the user to find the trade routes and stories they are most interested in.
Bringing data to life
However, as D3 can only render graphics, it can handle neither the application state (what is currently seen by the user) nor the application model (how to parse the data coming from the database through the API). Opting to use the Redux library alongside D3 helped us deal with complex interaction flows within the data, both synchronous and asynchronous. It allowed us to keep the whole application state — including enabled filters, selected nodes and selected regions on the map — in a simple Javascript object, opening up the possibility to share the application at a precise state.
Now, the usual companion to Redux is React, a library created by Facebook to render HTML quickly. It is hugely popular and vizzuality has many React fans in its ranks, including me. But a problem we often have is making React and D3 (or Leaflet) play nicely together. At their core, both libraries have the same goal — rendering a visual interface from the data — but they have opposed ideas on how to do it.
In many of our projects, such as Global Fishing Watch, where including both D3 and React is essential, we carefully create code to bind them. But for Trase we took a much more radical stance and simply ditched React. Instead, we let D3, Leaflet, or some other native HTML5 or Javascript API’s take care of the whole rendering process. The only missing part was the ‘glue’ between Redux and the various rendering strategies, and it consists of a very simple extendable native class, that more or less mimics the behaviour of react-redux (see this implementation example).
Using Redux and D3, along with ES6 syntax and a heavy use of functional programming, allowed us to write maintainable code that it is expressive yet simple and doesn’t rely on many dependencies. I’m very happy with the final result and pleased that Trase offers a range of features that allow people to deeply explore an extraordinary amount of data. All of the code is open source so feel free to check it out yourself.
