Building quick interactive projects (and we’re hiring)
The week began like any other. Catchup and planning Mondays, a bit of work on Tuesday as everything is going as planned. Then…
“Hey, we’ve got this big story up this weekend. Could you build us a [insert big interactive name here] pronto? Shouldn’t be too much work, right?”
Plans are un-made, and here we go working on something new on a very tight deadline.
Obviously we’re feeling very happy about the newsroom engaging with our work so much, but the challenge of producing something of value in a short time always seems a bit daunting. Chris and I have written already about how to manage such deadlines and our high design and development standards, but recent weeks have been fun in this regard.
“How many crimes go unsolved where you live?”
This Sunday Times investigation into the number of crimes solved by police relies on Krystina Shveda’s analysis of public data. Krystina, from the Times and Sunday Times Data team, told us about this project about a week ahead of time.
Other commitments left us with three working days to complete this project.
We wanted to provide readers with a personalised insight into the story. The piece would cover the issues uncovered in England and Wales, and our interactive would be a classic “put your postcode in, see how it is where you live.”
Resolving postcodes is so easy these days thanks to the incredible postcodes.io API. In this instance we obtained matches to an LSOA (Lower Super Output Area, a geographical area built to contain consistent population sizes of about 1,500). Our analysis then matched the granularity of the data.police.uk dataset.
Maps were as just as straightforward as using react-leaflet and adding our LSOA GeoJSON overlay.
With an additional day, we added some context to the reader’s local area data thanks to this barcode plot, which shows your local area, the national average, and a visual sense of how the crime solving rate is clustered between 10 and 15% across England and Wales.
“How many slaves do you think there are in Britain today?”
This story was pitched and put on the schedule while I was away, and a lot of the work was done ahead of time. I normally like to be involved early on, and before the visualisation technique has been agreed. In this instance it was not possible — and that’s another typical example of how things occasionally work in the news business.
Based on rough wireframes, our head of digital design Matt Myers worked some great designs, and the whole thing was reviewed and agreed upon when I made it in on Thursday morning.
Chris and I divided up the work between the interactive slider and viz part and the backend database and plumbing.
We built in parallel over two days and connected our work through Chris’s Serverless API, which handled the submission of an estimate by the readers as well as returning the dataset that allowed the distribution of guesses visualisation.
Using a lambda — a small chunk of code, often just a single function — allowed us keep everything simple, and easy to maintain, and meant we could focus on the business logic and the user interface.
We also used a second Lambda function to batch process the predictions from our database into a JSON file that could be consumed by our UI component, and write it to a static file storage system, in our case AWS S3. Serverless makes it trivial to run a function on a schedule, using their rate(2 minutes) configuration syntax.
To give us absolute confidence in the code, and with any future changes, and full suite of tests were written using the Jest testing framework. These ensured that both our HTTP endpoint, and our batch process ran as expected, and returned any errors if invalid data was passed. While we could trust our UI to send valid data structures, it’s always a good idea to validate in the backend code as well.
Viz-wise, central to the success of this project was our initial discussion first thing in the morning, which put at the centre of the frontend job a d3 scale.
This scale would be, contrary to what was initially drawn up, placed at the very top of our app and passed down to the two stages of the project: the slider that asks a guess from readers, then the visualisation itself. The transition between stages was then seamless, and building with an immutable scale made positioning everything a dream.
From the components construction above, we’re able to smoothly pick up the reader’s guess and transition nicely to the visualisation itself:
- what the reader has guessed;
- two official estimates (hard-coded);
- and an average estimates from Times readers’ submissions.
A few observations:
- Despite our slider being linear and ranging from 0 to 30,000+, we observe a stepping of guesses (see the taller bars every 5,000 increment).
- Many picked 30,000+, which surprised us.
- Based on the two points above, I wonder what would have happened if we had given a free text input to readers…
End note
If you liked this article and want to work with us, we have two job openings going at the moment.
One is a data journalist role [1] that would work with the Times and Sunday Times data team, and the second job [2] is for an interactive journalist. You can get in touch with me with any question at basile.simon@thetimes.co.uk
1: https://newscareers.broadbeantech.com/jobs/view/3240.html
2: https://newscareers.broadbeantech.com/jobs/view/3296.html
