How we polled the entire SC State legislature and shared the results live

An experiment in real-time accountability


In the weeks following the shooting at the Emanuel AME Church here in Charleston, SC, a great deal of attention was focused on the confederate flag flying at the Statehouse grounds in Columbia.

As Dylann Roof’s motives became clear and photos of him with confederate memorabilia were shared widely online, the flag was widely assailed as a symbol of racism.

On the morning of June 23rd, political figures in South Carolina were beginning to publicly call for the flag’s removal and rumors were swirling that Governor Nikki Haley would soon join the chorus of voices in favor of removal.

The flag’s fate, however, rested in the hands of the state legislature. A 2000 compromise which moved the flag from atop the Capitol to a monument in front of the building set a high bar for any further changes: two-thirds of both houses of the legislature would need to vote in favor of any bill attempting to remove the flag.

At The Post and Courier we decided to begin reaching out to state lawmakers to get their opinions on the topic.

Spreading out the task

I got involved in the project when our Special Projects Editor Glenn Smith asked me how we could store poll responses in a way that would let many reporters collaborate at once. In total, we were attempting to contact 169 people, so it obviously made the most sense to spread the task out among our newsroom.

The most obvious solution, I thought, would be to use Google Spreadsheets.

This would allow all of our reporters to edit one spreadsheet at the same time. It would handle revision history and allow our entire newsroom to view results in real-time.

Thankfully, I had already been working on getting more of our newsroom on Slack and using Google Docs for drafts, so there was little training necessary. In a few minutes, our spreadsheet was setup and posted to a new Slack channel and we were off.

Over the next hour I watched as our team worked to fill out the spreadsheet with responses from lawmakers. I found it captivating, which led to an idea: what if we shared these results online? Would the public find it interesting? Would it encourage more lawmakers to go on the record?

Making it public

I consulted Glenn about it as well as our web editor, Laura Gaton, and our executive editor, Mitch Pugh, and we agreed it was an interesting idea.

I sketched out a quick concept for a layout. I figured we’d start simply with two tables and two charts. The charts would show the responses we were getting and the tables would list each lawmaker in either chamber so that people could check our work and look for individual legislators. It took about two hours of work to get the page up and running.

One good thing about Google Spreadsheets is that it already has a feature that allows you to export basic charts and tables to embed elsewhere on the internet. This would be fine for tables.

Google’s charting options, though, were less optimal. They could not be updated in real-time and their design options are not very flexible.

Drawing a live chart

One of the greatest challenges of developing in a newsroom, especially in breaking news situations, is that you’ll never have enough time to do everything that you really want to. You need to identify and implement solutions quickly. Often, these solutions need to be scaleable in case your content goes viral.

This means, I think, that you simply must be prepared to use off-the-shelf solutions to problems. In-house frameworks take time to develop — you can’t always reinvent the wheel, even when you’d like to.

I frequently use a javascript library called d3.js to produce interactive charts and graphics. It is very useful and has a wide variety of fuctions, but it can do very little “out of the box,” so the speak. Producing a quality visualization with just d3 requires some significant development time.

I have been working to build an in-house framework of re-usable d3 visualizations, but I hadn’t yet tackled anything that would be useful in this situation. I needed live charts and I needed them quickly.

Some of the types of charts that d3plus can be used to create quickly.

Thankfully, there are a number of projects which extend the d3 library to allow you to create interactive charts quickly using common design patterns. I chose to use one called d3plus, since I had used it for prototyping in the past.

Setting up a basic pie chart with d3plus is very simple and requires very little code. Once the d3 and d3plus scripts are loaded on a page, the object d3plus can be called with various methods to set a data source and control the style and features of the chart.

var data = [
{“value”: 100, “name”: “alpha”},
{“value”: 70, “name”: “beta”},
{“value”: 40, “name”: “gamma”},
{“value”: 15, “name”: “delta”},
{“value”: 5, “name”: “epsilon”},
{“value”: 1, “name”: “zeta”}
]
d3plus.viz()
.container(“#container”)
.data(data)
.type(“pie”)
.id(“name”)
.size(“value”)
.draw()

This code would create a pie chart and insert it into an HTML element with an id of “container” using data contained in the data object. But this will only generate a static chart, and the data must be included in the page source.

To get live charts, we need to obtain data from somewhere else and tell the chart to constantly check that data source and update when it changes.

var chart = d3plus.viz()
.container(“#container”)
.type(“pie”)
.id(“name”)
.size(“value”)
// Create a function to set the data source and draw the chart
function setChart(data) {
chart.data(data).draw();
}
// Use jQuery to get data once and draw the chart
$.getJSON("http://data.postandcourier.com/data.json").done(
function(data) {
setChart(data);
});
// Set a timer to check the data source and re-draw the chart
var updateIntervalMS = 30000;
window.setInterval(function(){
$.getJSON("http://data.postandcourier.com/data.json").done(
function(data) {
setChart(data);
});
}, updateIntervalMS);

This will do the same thing as earlier, but now the data source can be any arbitrary JSON file and the chart will grab that file at an interval you set and update the chart.

It’s important to call the .draw() method on your chart anytime you change the data. D3 will conveniently take care of animating any changes for you.

Getting the data

Now we have a chart that can automatically update as the data source changes, so we need a dynamic data source.

I looked for a way to export Google Spreadsheet data in a format like JSON, but I couldn’t find a solution that I thought I could implement quickly enough.

I couldn’t think of any quick solution other than manually managing a JSON file. Thankfully, I didn’t need to visualize too much information and all updates would almost certainly come during normal business hours.

I wrote a very basic Meteor app and deployed it on our data server. Basically, all it did was function as a frontend for editing a basic JSON file containing vote totals. The resultant JSON looked like this:

[{"Chamber":"Senate"},
{"data":[
{"Response":"Aye","Value":93},
{"Response":"Nay","Value":18},
{"Response":"Not Voting","Value":6},
{"Response":"Absent","Value":6}]
}]

The app was probably unnecessary as all it did was make updating the file easier. The file could just as easily be edited in a text editor and uploaded via FTP, for example.

I was manually obtaining vote totals from our spreadsheet, usually whenever an update was posted to our Slack poll channel. Once I saved a change to this, the update would spread to anyone viewing the page within 30 seconds.

Screencap of one of the earlier versions of the page

How it turned out

Most lawmakers responded

When we first began contacting lawmakers, we never really expected to get all of them on the record. I wondered, though, if posting the results online would encourage more responses. It turns out that it almost certainly did.

On the second day of running the poll, we saw “yes” responses pass two-thirds in the Senate. This happened after the head of the state Democratic Party contacted us on behalf of the remaining Democrats who hadn’t yet responded to let us know that they would be voting to remove the flag.

I used Google Spreadsheets to create a new table featuring only the lawmakers who hadn’t yet responded. In this table, I included their office phone number and twitter handle (if they had one).

Almost immediately, interested citizens began using the page as a resource to contact their lawmakers and share their opinions. In fact, I received one particularly encouraging email from a woman who said that until now she had never contacted any of her representatives in government before.

I know that our newsroom received at least one call from a lawmaker who said that he was getting phone calls from constituents who were concerned that he hadn’t responded yet to our poll. He told us he was in favor of removing the flag.

Public interest was very high

For the duration of the poll, it remained one of the top pages on our site. It was widely shared on social media and generated meta coverage of its own.

We received an enourmous amount of feedback on social media as well as via email. Nearly all of it was positive. It seems that the public was as captivated by the live tally as I was.

Our poll successfully predicted the outcome of the vote

Most of the responses we got came in the first three days of our poll, but responses continued to trickle in for days thereafter. We quickly passed the two-third threshold in the Senate, but the House was slow going. Many lawmakers were on vacation and were difficult to reach. Finally on June 30, we passed two-thirds support in the House.

About a week later, the House and Senate conveined in special sessions to debate removing the flag. Both chambers voted in favor of removal. The final votes were remarkably consistent with our poll. Very few lawmakers deviated from how they told us they intended to vote.

As journalists, we attempt to hold lawmakers accountable to the public and we also aim to increase transparency in the political process. I would conclude that live polling is a potentially effective strategy for accomplishing both of these goals. It has the added benefit of increasing our own transparency while simultaneously engaging the public by inviting them to follow a story as we report it in real-time.

I can see this kind of live coverage as a valuable addition to our collective toolkit as journalists.

How it could be done better

The greatest flaw, I think, with our poll had to do with the fact that our spreadsheet was not directly connected to the live charts. I had to manually curate and update the vote totals.

This was not entirely bad, because it did allow me to exercise some editorial judgement over updates. There were a handful of instances where reporters entered responses incorrectly in the spreadsheet, which I was able to avoid pushing to a live audience.

Still, it was inconvenient and would make it difficult to run something after hours.

I think that technology stacks like Meteor are particularly well-suited to this type of challenge. Meteor combines Node.js, MongoDB, and a reactive real-time frontend template engine (although you can use a third-party framework like React, if you prefer). It allows you to rapidly build and deploy collaborative, real-time web apps.

When we started the poll, I did not have time to develop a full app to handle this, but I’m currently working on developing a robust platform for live coverage of this type for use in the future.

This would be my biggest piece of advice to anyone looking to replicate this project: try to plan it ahead of time, to the extent that it is possible. Develop a plan for the following:

How will you store data? Who will be inputting that data? If they are non-technical journalists, how will you ensure that they input data correctly?

How will you display your data? Either use an off-the-shelf library like d3plus, nvd3, c3.js, chart.js, or highcharts (required license for commercial use), or spend the time now to develop your own custom framework using something like vanilla d3. Some other frameworks that could be very helpful here include The Guardian’s fantastic Ractive.js, and Facebook’s React.js, either of which could be used in addition to — or perhaps in place of — d3 depending on your needs.

How will you make a live connection between your data and your charts? The simplest option is to do what I did: just manually do it. A much more robust solution, though, is to actually connect your database to your frontend. Again, something like Meteor is ideal here since it does all of this automatically although this is almost certainly not something you can build and deploy in a single day, at least not as an individual. If you do not have the flexibility to deploy Node and MongoDB apps on your server, you could at least do something in between these options using PHP and SQL.