Tracing the flow of coffee and soy
An engineer’s insight into building a new tool to analyse international trade flows
Note: this post is about work I did when I was working at Vizzuality.
Since August, I’ve been working on Trase, a brand new tool developed by the Stockholm Environment Institute (SEI) and the Global Canopy Programme (GCP) that will lay bare the flow of agricultural commodities through the global trade system. From production to consumer, Trase reveals the producers, traders and transporters involved in the international trade of products like coffee, soy, and palm oil. The datasets that hold all of this information are immense and the relationships between the data points are complex, but it’s these qualities that have made working on Trase so fascinating.
Upon joining the project, I was immediately impressed by SEI’s ambitious vision for the data visualisation we were going to build together. There was a plan to add even more features to an existing prototype, and it was clear that the final product would be very complex in terms of interactivity. Thus, one of the main challenges of the project was how to keep the balance between letting people explore the incredibly extensive dataset and not drowning them in too many overlapping features.
During a sprint with SEI and GCP, everything was analysed to identify the most important features. It was tough at times but we resisted the urge to cram in as many interactive features as possible. We know from experience that adding something that’s useful to only 1% of users can harm the experience of the other 99%. Beyond these discussions about functionality, we also needed to merge features seamlessly, find shortcuts for the users, and design the best way to progressively introduce users to more complex features.
When it comes to choosing a library to render a complex data visualisation, the choice is usually quite obvious: D3. D3 has been around since 2011 when it was built by Mike Bostock while he was at the New York Times. Since then, it has matured into an incredible piece of technology that allows developers to build interactive visualisations with an outstanding level of flexibility.
Trase uses the latest iteration of the library (version 4), with a completely customised layout engine for the sankey diagram that visualises the flow of commodities from producer to country of import. It’s pretty amazing what you can do with D3. To illustrate its power, we went from a static sankey diagram to a fully animated one in just a few lines of code. The sankey diagram can be completely controlled by the user to find the trade routes and stories they are most interested in.
Bringing data to life
Now, the usual companion to Redux is React, a library created by Facebook to render HTML quickly. It is hugely popular and vizzuality has many React fans in its ranks, including me. But a problem we often have is making React and D3 (or Leaflet) play nicely together. At their core, both libraries have the same goal — rendering a visual interface from the data — but they have opposed ideas on how to do it.
Using Redux and D3, along with ES6 syntax and a heavy use of functional programming, allowed us to write maintainable code that it is expressive yet simple and doesn’t rely on many dependencies. I’m very happy with the final result and pleased that Trase offers a range of features that allow people to deeply explore an extraordinary amount of data. All of the code is open source so feel free to check it out yourself.