Run, paste, publish — how scripts help us to rapidly respond to news
As data journalists at NZZ Visuals we automated our reactions to recurring events: By writing various scripts that promptly create maps and graphics.
Many journalists know the challenge of managing what some call the «fast track» versus the «long track» at the same time. At NZZ Visuals we work on the daily news, and in parallel we create longer term data- or visual-driven stories. We have built a range of tools and a team structure that address this challenge and make us more efficient. One of these tools is what we call Rapid Response Scripts.
What are Rapid Response Scripts?
A Rapid Response Script allows us to prepare data for a news-related graphic with the push of a button. It typically loads data from one or more sources, transforms it in whichever way we need and exports it in the structure that we need for creating a map or chart. All that’s left to do then is paste it over to our toolbox Q (made by our colleagues at NZZ Editorial Tech).
The scripts are written in R, Python or JavaScript — depending on who wrote it and what works best. They are well documented: comments explain what the code does and why, because all data journalists in the team should be able to use any of these scripts on a given day.
So — what do we rapidly respond to? We currently have scripts to…
- create a bundle of graphics around Bitcoin’s and other cryptocurrencies’ value
- visualize hurricane tracks and forecasts as well as flood data
- show conflict zones on a map of Syria, Iraq or Libya
- display active wildfires on a map of Europe
- show Donald Trump’s approval ratings…
- …and polls related to Trump’s impeachment
- display polling results for political parties in countries all over Europe
- put a terror attack into context
- show the route of a particular flight, e.g. when a plane went missing or crashed
- visualize the frequency of mass shooting incidents in the USA.
What situations nudge us to write a Rapid Response Script?
(It really is a nudge. We don’t sit in meetings and think of all the possible scripts that may be useful to us one day).
Let us start with an example: Sometime along the hurricane season 2017 we wanted to visualize the predicted course of a hurricane. We spent roughly an afternoon searching for hurricane prediction data, adding GeoJSON properties to it, tweaking them, finding out how to best make the visualization work in our toolbox Q and writing a substantial footnote on the methodological restrictions of the resulting map.
We then created similar maps as the hurricane moved, and for the next storms of the season — following a process that went roughly like this: Look for the link to the source, download the data, add properties, visualize in Q, add footnote. It doesn’t take much time on a given day, but after the nth hurricane, time adds up. At some point, we got sick of copy-pasting and tweaking GeoJSON-code — and of being uncomfortable with footnotes disclaiming a method that we weren’t fully behind.
To sum up:
- A recurring event — a hurricane; a terrorist attack; Bitcoin’s value changes dramatically —
- that is better understood when coverage includes graphics
- requires recurring resources from a data journalist, dev or designer in order to create a visualization.
We do this a couple of times. And then we typically feel 🙄 and decide to (semi-) automate the procedure.
In the hurricanes case, we took time to research, think, explore: What ideas do we (and other media organizations) have to highlight the inherent uncertainty of a storm prediction? What do we consider the most suited method? Which data sources do we need for this?
A data journalist and a designer developed a visualization in collaboration. Then we wrote an R-script that loads different types of data:
- the past track of the hurricane,
- its predicted track,
- date and time for specific points on the trajectory,
- and wind speed probabilities.
The script transforms the data, translates text, adds new information, creates a visualization within R for verification and then writes out GeoJSON code that can be copy-pasted to a map in our toolbox Q.
Why is this good?
- (Semi-)automating procedures allows us to respond to breaking news situations rapidly and consistently — so we typically offer our readers a visualization faster than other news organizations. This buys us time: Either for a custom analysis and story in this specific situation. Or for mid- to long-term projects.
- It forces us to think and talk about sources and methods and to document them. And it allows us to take the time to consult designers about the most appropriate visualization.
- When we take the time to write a reusable script, we usually also take time to review its code with a colleague, which further ensures the quality of our output. And it’s always easier to double check code and sources in a quiet moment rather than a breaking news situation.
- Automating the process leads to shared responsibilities: When there’s a Rapid Response Script, anyone who understands the script can create the hurricane map or update the Bitcoin explainer. An informal process, however, usually leads to the data journalist who took care of the first quick fix always being in charge of it. This is not ideal, for several reasons.
So our scripts live in a shared folder. Some of them are open sourced on NZZ Visuals’ methods page (which also means that our colleagues from other media organizations can build on them and share their work back to us).
Why are not all of our Rapid Response Scripts open sourced? Uuhm, so — rapid response sometimes also means rapid scripting. At first, it is often hard to gauge how often we will use a particular script. So we refrain from investing hours in perfect coding. Over time, though, we code-review the important scripts and refine their structure. We only open source scripts that we really regard as solid. As for the others — we reserve ourselves the freedom to write mediocre code.
How does your team juggle between breaking news and mid- to long-term projects? We would love to hear about it.
Our current Rapid Response Scripts have been written by David Bauer, Beni Buess, Haluka Maier-Borst, Anna Wiederkehr and Marie-José Kolly.
