Look at the Logs! (FOIA Edition)
In our recent public records examination of Theranos, a now-disfavored medical technology startup, we noticed that FOIA requests sometimes precipitated a breakout of unflattering revelations about the company. Our investigation led us to wonder: what else is it possible to learn from agency FOIA logs?
FOIA, or the Freedom of Information Act, allows any party to request documents from the U.S. federal government. Along with other required disclosure laws, FOIA is part of the legal framework supporting governmental transparency in the U.S. The utilization of the law to gain information has many benefits for both private and public sectors, as evident from agency logs. The logs are filled with litigious law firms, university researchers, inquisitive journalists and a few curious citizens seeking a peek into the bureaucracy.
For some agencies, there may be almost as much to glean from FOIA logs as the FOIA results themselves — including a revealing glimpse into modus operandi of the most notorious FOIA requester in the nation.
To that end, we have added a number of FOIA logs from agencies that proved helpful to our investigation of Theranos into Enigma Public, including the logs from Securities & Exchange Commission (SEC) and Centers for Disease Control (CDC).
To explore the sorts of trends and patterns possible to glean from a FOIA log, we examined the most recent log from the SEC.
The SEC has a wealth of data on publicly traded companies. As one might imagine, there are many who are interested in the public filings of big businesses. Though FOIA is a tool available to the greater public, perhaps unsurprisingly, private commercial enterprises submit the bulk of SEC FOIA requests.
There are, however, a few standout organizations who have acted in the public interest. News organizations, for example, make up the second largest share of requests. Of that portion, Probes Reporter (an industry publication specializing in SEC investigations) and Vice News dominate.
These prolific requesters have two very different styles: Probes Report put out a steady rate of requests (about 50 requests per week), whereas Vice News submitted large batches of requests at once (with notably higher rates of success).
Vice News had bit of a secret weapon: Jason Leopold, a journalist known to some agencies as the “FOIA Terrorist.” While Vice News was Leopold’s employer at the time of the requests, the FOIAs represent efforts to research and source his own beat, in his capacity as an individual reporter. Given that 100% of the Vice News FOIAs note him as the requester (along with his continued endeavors at Buzzfeed News), it seems his moniker is well-earned.
A poke at just one slice of FOIA logs reveals a glimpse into a community of watch dogs. With the added help of some text analysis, there’s even more to uncover. Names and entities recovered from the logs can be further contextualized by linking additional datasets. Connecting data at the entity level offers an opportunity to discover new relationships or leads — to ground pieces from various sources in the bigger picture.
The full potential of FOIA, however, has yet to be realized. Our search for ingestion-worthy FOIA logs revealed one of the obstacles to utilizing this data source to its fullest extent — few FOIA logs are created alike. Some agencies offer FOIA logs that are merely two columns wide, whereas others, like the SEC, offer the full name and organization of the requestor. The richness of the data across agencies is quite inconsistent.
Perhaps most frustrating of all is the format. Many FOIA case logs are released as PDFs — a common frustration when it comes to public data. Though there are a number of tools to automate the processing of PDFs, the format is consistently more cumbersome than a slew of CSVs or a chunk of JSON.
Without additional legal framework regulating the quality (as well as the necessity) of public disclosures, FOIAs remain a subject worthy of inquiry onto themselves.
Originally published at www.enigma.com on March 1, 2018.
