Open Source Journalism: Data, Transparency and the Big Story

AP Photo/Jeff Roberson/Captions by Paul Szoldra/Business Insider

When images of camouflage-clad cops sitting atop armored personnel carriers in Ferguson, Missouri lit up the Internet, New York Times visualizations editor Tom Giratikanon wondered what data might be relevant to tell the story of the militarization of U.S. police forces. The answer? An obscure U.S. Department of Defense program called the Excess Property program (1033D). This program transferred surplus military gear such as armored personnel carriers, assault rifles and grenade launchers to local police forces at subsidized rates or for free.

The 1033D program was not a new story. Washington Post reporter Radley Balko had covered the topic extensively in his book, Rise of the Warrior Cop. Regardless, scant comprehensive data existed on the full scope of the military materials transfers to local police departments, many of them in smaller U.S. communities where violent crime was virtually non-existent and terrorist targets not in evidence.

So he sought to obtain all the transfer data from the Department of Defense. This meant a data file with hundreds of thousands of entries to clean up. Many of the transfer items were non-lethal items such as towels and boots. Giratikanon had to sift out the assault rifles and Mine-Resistant Ambush Protected vehicles to properly tell the tale.

He did and published this amazing visualization. It went viral on the Internet, garnering millions of pageviews. And then Giratikanon did an interesting thing. He put the data that he had painstakingly cleaned up on GitHub for anyone to use. In effect, he open sourced the value of his laborious work.

From there, an analytics startup Mode Analytics pulled in the data and built their own county-by-county visualization. Our talented data journalist Alice Corona also took the NYT data, reshaped it to fit best into Silk and generated her own visualizations — along with some big questions. (Such as, why did Alabama and Florida end up with so much of the military surplus gear?). She posted her additional data sources here. At each step of the journey, additional insights and ways to see the data surfaced. Below you can see a graph from the Silk showing the total acquisition cost of per state in dollars per inhabitant.

Data from us-military-gear.silk.co

We are entering an era of Open Source Journalism and its rather exciting. Publish an infographic, table, map or other visualization is no longer sufficient. The expectation is growing that the data sources and files data must be made publicly available, particularly if it results from a Freedom of Information Act (FOIA) request through an organization like FOIA Machine or MuckRock. The Guardian publishes data it maintains on its Data blog. The startup content site Five-Thirty-Eight, which focuses on covering the news through the lens of statistical analysis, also maintains a GitHub repository for its data. The Los Angeles Times, the BBC, and BuzzFeed are other organizations doing data-driven journalism and regularly pointing to the sources of their data.

Open Source Journalism levels the playing field. Every neighborhood blogger in California or New York or London can now post visualizations using the very same data as the biggest news organizations in the world. And the blogger can focus that data down on the local impact. Open Source Journalism also makes the news process more transparent. Errors can more easily be spotted and the news organization’s work can be replicated using the original data source, much like the standard for scientific research. We’ve all been journalists for some time now, since the pajama bloggers broken into the top ranks of media. In the very near future, we’ll all be data journalists, too. It’s going to be great fun.