This past Thursday, Facebook released a white paper which “does not contradict” the U.S. intelligence community’s assessment that its platform was used to carry out influence operations leading up to the 2016 election. Though much has been written about the role of bots and influence campaigns — both foreign and domestic — in the election, it’s unclear what role — if any — such campaigns may play post-election. Have the bots stuck around to continue pushing their messages? Have they disappeared? Have they moved on to other causes and upcoming elections? Have new “bot armies” taken their place?

Our group previously provided evidence of a coordinated social media influence campaign during the 2016 election season marked by sudden, simultaneous shifts in language across multiple platforms, including Facebook. For those analyses, we identified nearly 30,000 accounts posting duplicate content (ex: multi-word comments each identical to at least 4 others) on Donald Trump’s official Facebook page and a fan-created page during the campaign. We considered this pattern of posting behavior to be consistent with that of automated accounts, or bots. When repeating this process using comments from both before and after the election and inauguration (July 14, 2015, to April 9, 2017), we once again identified roughly 30,000 likely bots — 32,003, to be exact.

These updated analyses indicate that many campaign-era bots remain in “maintenance mode” today, continuing to post somewhat regularly and ready to pounce whenever a particular administration action piques their interest. New bot networks have also been deployed in response to specific events since the election. For the time being, bots continue to be actively involved in the social media ecosystem and, without intervention on the part of the platforms, show no signs of letting up anytime soon.

(Throughout this article, we refer to any account which posted at least one 10+ token comment identical to 4+ others in our dataset as a “bot.” However, this method of bot identification is far from perfect; for example, any human-operated accounts copying and pasting “viral” messages would be classified as a bot for our purposes, and more sophisticated automated accounts that generate unique content for each post would not be caught by our somewhat crude classification system.)

The Election

Bots were most active on Trump’s Facebook pages right before the election, with nearly 15,000 active in the 30 days leading up to November 8, 2016. Though bots made up only 3% of active accounts, they posted 14% of all comments during this time.

This pre-election spike in bot activity mirrored an overall increase in posting, even among non-bot accounts. However, the jump in bot activity happened earlier and was more sustained, whereas non-bot activity increased rapidly in the days just prior to the election. Both peaked on November 9th, the day after the election.

Immediately after the election, posting levels dropped across-the-board, for both bot and non-bot accounts. Of the over 25,000 bot accounts active before the election, 44% continued posting after November 9, 2016.

The First 100 Days

During the first 100 days of Donald Trump’s presidency (although technically, only the first 80 days were included in our dataset), 32% of bots active pre-election continued to post — conversely, 63% of bots active post-inauguration were active pre-election. Bots generally remained active at post-election, pre-inauguration levels, posting roughly 12% of comments per day. Non-bot activity, on the other hand, trended downward over time.

Relative daily posting activity tracked similarly across both groups, likely a result of all comments being posted in response to page posts. For example, if a page produces 5 posts one day, and only 1 post another day, the day with 5 posts would present more opportunity for bots and non-bots alike to comment.

However, several notable differences in activity occurred. For example, during the first 2 weeks of the Trump presidency, 3 events provoked divergent responses from bot vs. non-bot accounts:

Non-bot accounts showed a dramatic spike in posting behavior on January 20, 2017 — the day of President Trump’s inauguration. Though bots also increased activity relative to the preceding days, the spike was nowhere near as drastic. Conversely, a spike in bot activity occurred on January 27, 2017 — the day President Trump signed an executive order halting immigration and refugee resettlement from 7 Muslim-majority nations. Whereas non-bot accounts posted substantially more comments on Inauguration Day than on the day of the so-called “Muslim ban,” for bots this pattern was reversed. Finally, a spike in non-bot commenting on January 31, 2017 — the day President Trump nominated (then-) Judge Neil Gorsuch to the US Supreme Court — was not accompanied by as large an increase in bot activity.

These different patterns of posting suggest that bots and non-bots have different “interests:” bots were less interested in the pomp and circumstance of the 58th US Presidential Inaugural Ceremonies and nomination of a traditionally conservative judge to the Supreme Court, while relatively more interested in an executive action affecting foreign relations, than non-bots.

The Syria Strike Spike

Post-election, April 7, 2017 — the date of the Syria strike — was the most active day for bots, with just under 2500 bot accounts posting over 6500 messages. On that day alone, bots made up 12% of active users and nearly one in four comments. Though posting by non-bots increased as well, it was not nearly to the same degree as the increase in bot posts.

Looking into the content of the bot posts, almost all were, predictably, about the Syria strike. Interestingly, more than half of the active bot accounts posted one (or more) of 3 almost identical messages, which in total made up just under half of all bot-posted messages that day:

These blows increase our determination, strength and will, and you are still condemning your great lie by calling it a revolution. Israel has increased its frustration with the strength of the Syrian Arab army, which is fighting on all fronts at home and this border is Syria, with its strength and successful leadership, the people are growing in love with its army and leadership. We resist and resist because we are right owners no matter how you and all the traitors and those who call themselves kings We will not surrender here the Syrian Arab Army. Here are the protectors of the homeland. [Posted 1500+ times]

We Syrian people denounce the attack on Our air base, which has always played a major role in the fight against terrorism. [Posted 1000+ times]

Trump you exceeded your limits in this strike American Zionist Saudi Arabia. We will tell you that the Syrian Arab Army is the myth and master of this time are the protectors of the homeland and its immune system. The reply is coming. Here is Syria Assad. [Posted 350+ times]

Of the bot accounts which posted those messages, only 8 (less than 1%) had posted prior to the election. Only 2% had posted prior to April 7, 2017. In contrast, of the bot accounts which posted other messages, 1 in 5 were active pre-election, and two-thirds were active prior to April 7. This suggests that additional bots may have been deployed to push pro-Assad messages in response to the Syria strikes. It’s worth noting that even when discounting messages from these “new” accounts, the remaining posts — greater than 3500 in all — still represent one of the highest single-day post volumes post-election.

Dueling Botnets

Nearly 1 in 3 bot accounts active during the early days of the Trump administration had not posted prior to the election. Some appear to have been activated immediately post-election, while yet another group of roughly equal size was deployed post-inauguration. Both of these groups have maintained a stable presence since then, with the exception of the day of the Syria strike. As mentioned earlier, the surge in active bot accounts that day likely represented an independent “bot attack,” but for visualization purposes, they are counted as part of the “post-inauguration” bot group.

This observation raises the possibility that separate actors may be controlling independent “bot armies,” each with specific — and perhaps conflicting — interests and intentions. And even as pre-election bots drop off, more may step in to take their place.

What’s Next?

Civic engagement with our elected officials, including the president, as well as other interested citizens is a critical component of democracy and US government. The Internet and social media were supposed to usher in an era of “democratization of knowledge,” wherein everyday citizens — not just elected officials and those with special access — could have unprecedented access to information about the inner workings of government and be able to discuss policy with others across the country — and the world.

Then came the 2016 election, in which Donald Trump surprisingly (at least to most political pundits, pollsters, and legacy media outlets) overtook frontrunner Hillary Clinton in the Electoral College to become the 45th President of the United States. Countless election “postmortems” have focused on the role of “fake news” and social media in influencing public opinion. In these less rosy takes, the “democratization of knowledge” has also increased public access to misinformation, often pushed by “users” who don’t have America’s best interests in mind.

Moreover, the massive prevalence of automated content and content posted by disingenuous actors (see also sockpuppets and trolls) violates an implicit compact users make when they submit their personal information to social media platforms: that they will be engaging with other actual people. When Facebook users visit an official or fan-created page of their president, they expect to receive updates about his actions and interact with other users — positively or negatively, civilly or less-than-civilly — in response. If over 1 in 10 comments on a typical day are the work of automated accounts, however, the opportunity for discourse collapses.

In the months since the election, Facebook has taken multiple steps to combat fake news and police fake accounts. On the fake news front, they’ve begun flagging “disputed” news stories, posted tips on how to spot fake news, and even taken out full-page print ads in newspapers in France and Germany. Regarding fake accounts, they recently shut down 30,000 accounts in France in advance of the presidential election and disrupted an international spam operation. Notably, these Facebook-initiated anti-bot measures took place directly after the period covered in our analyses of Trump’s Facebook pages. It remains to be seen whether they will have any impact site-wide, or whether fake accounts are so embedded in the social media economy that their continued use is all-but-inevitable.

Nonetheless, one thing’s for certain: they’re not going away on their own.

Continued Bot Infiltration of Trump’s Facebook Pages was originally published in Data for Democracy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Bribery, extortion, and wire fraud.

Those are the charges filed against Willie Cochran, a Chicago alderman accused of stealing “at least $30,000 from a charitable fund for poor constituents.” Cochran is the latest in a long line of Chicago officials indicted for corruption charges — over 1,000 since 1973.

Pretty stiff competition to get into the Chicago Hall of Shame.

So, how does an interested citizen stay informed about their local representatives? It seems impossible to keep track of legitimate political contributions, let alone the shady ones.

But, what if you could trace the money, based on publicly available data?

Enter Data for Democracy and data.world.

We at Data for Democracy are a group of 1,300 self-organized volunteers who work with data in order to improve the global community. D4D uses data.world, the social network for data people, to discover and explore data while collaborating on data projects.

Last weekend, hundreds of volunteers met up in major cities across the country (and remotely) to participate in the first D4D hackathon. We worked on dozens of civic projects, leveled up our data skills, and ate our weight in pizza (or, at least we did in Chicago).

Here’s what our group did in less than 24 hours together, and all the data you need to dig in yourself.

It all started with an idea Stephanie floated before the hackathon:

“We could link alder-people by caucus or other ties, see their connections on the graph through politics and through lobbyist connections…maybe something in Shiny app form that people can play with… a network igraph would be pretty rad,” she mused on the #city-chicago Slack channel.

We considered the possibilities — what if we could link individual contributions to an alderman’s voting records? Could we track which special interests were influencing votes, understand what, when, and how much lobbyists were paying caucuses, and find a way to to put it all out in the open? That would be pretty rad.

So, we set out to answer this question:

How does money make its way from a business or special interest into the hands of Chicago’s elected officials?

Step 1: Get the data

We started with the City of Chicago’s open data portal. A quick search yielded a number of relevant datasets, including a list of all registered lobbyists since 2012 and their political contributions. It (almost) sounded like the work was already done!

But, as we explored the data, it wasn’t all sunshine and rainbows. Client industries were classified haphazardly, column definitions weren’t always clear, and tracing lobbyists to legislators would require joining multiple files. That meant we’d have to download each dataset (breaking the provenance chain), and create new files from them. We wouldn’t be able to share our findings or process alongside the data, making it more difficult for others to benefit from our analysis.

That’s where data.world came in. While preparing for the hackathon, we imported key datasets from the City of Chicago portal and other sources. We gathered data about Chicago lobbyists as well as a master dataset with links for each of the Chicago hackathon projects. This gave participants a single destination for all the relevant sources:

Step 2: Get the people

On Friday night, we all met in person for the first time. After weeks of chatting over Slack about potential data projects, scrutinizing our hackathon playlist, and debating who makes the best pizza in Chicago, we were finally ready to hack all the things!

Each lead pitched their idea to the group, and our 35 local participants broke out into three projects:

• Chicago EPA started investigating Chicago’s water quality in light of funding cuts to programs that clean up the Great Lakes.
• Bolsa Familia got to work analyzing the Brazilian social welfare programand its relationship to the 2014 election of Dilma Rousseff.
• And of course, our project, Chicago lobbying, set out to connect the dots between special interests, lobbyists, and Chicago aldermen.

Step 3: Get to work

Once we split into groups, Team ChiLo got down to business. We white-boarded out the problem and broke it down into tasks. We dug into the data on data.world and researched visualizations similar to what we envisioned for our end result.

With only 10 hours together in person, our task list felt daunting. For the weekend, we decided to limit our scope to Chicago’s 50 aldermen instead of all elected officials.

We got to work cleaning and joining datasets in the data.world query tool (with the help of this handy tutorial), classifying the funding sources by industry, and joining alderman voting records with legislation details. We wrangled the data with a lot of R, some Python, and even good old-fashioned Excel. People of all skill levels contributed, using their tool of choice.

Of course, there were ups and downs. Early on, we were excited to find unique IDs for lobbyists…that turned out to not be unique…that after more investigation we decided might actually be unique. We had to make a last-minute decision to constrain the time period to fewer years than we had hoped to analyze. But we persevered, and by half-way through the first night, we had enough data cleaned and transformed to spin off a dedicated Chicago lobbyists dataset:

We added file descriptions and column definitions so others could understand the variables:

Using the built-in query editor, we joined three files to combine data about aldermen, lobbyists, and clients into a single table:

We exported that query and added it to the dataset as a new CSV. Opening the file full-screen, we got a good look at the data alongside the column metadata and visualized the data by selecting different columns to chart. Apparently, the most contributions come from the hospitality industry — over twice the amount of the next-highest sector (technology). Who knew!

Step 4: Make it visual

By the next morning, we were ready to start on our showcase visualization. We settled on a network graph with a node for each funding source (orange), lobbyist (light blue), and alderman (dark blue). You can filter by alderman and see contribution amounts by hovering over each connection.

By the end of the hackathon, we had racked up two new datasets, a bunch of R scripts, some new SQL knowledge (well, for a lot of us!), and a kick-ass D3 visualization. Really, something for everyone.

Plus we met some fellow data nerds, which was (at least!) half the fun.

At the end of our time together on Saturday, we recorded our progress for the rest of the teams to watch during the final showcase. You can see our video here:

But the fun didn’t stop there.

In the last week, we’ve added two different Shiny apps, built using R Studio and the data.world SDK, to our list of accomplishments.

The first shows donations to aldermen by year. Lining them all up in a graph starts to show the magnitude of contributions to one aldermen over another. The second, inspired by the Louisville hackathon team’s voting record visualization, makes it easy to search for a topic and see each alderman’s vote on the related legislation (like, say, Chicago’s 2016 Olympic bid):

These dashboards were inspired by interesting tidbits we uncovered through some exploratory visualizations shared over Slack:

…and we’re continuing to investigate the data, add new sources, and discuss our findings both alongside the data and in Slack:

One crucial discovery we’ve made since then that the lobbyist IDs are not unique to each client-lobbyist relationship after all, and we have redone our app to reflect this. While we can’t pinpoint the flow of dollars with as much granularity as we had hoped, the app still allows us to visualize how clients are connected to aldermen via lobbyists, and consider the broader web of interests at play. After all, a vital part of data science is being able to modify your position as the data — or your understanding of the data — changes.

So, what’s next?

After most hackathons, it’s rare to keep in touch with the people — or data — you met over the weekend. But, since the D4D community started remotely (sharing everything online over Slack, Github, and data.world), we’ve already picked up where we left off!

Part of the magic of Data for Democracy is that “what’s next” is really up to each participant. Anyone can join (or create!) a new project and we’ve got plenty to choose from. So, join us on the Data for Democracy Slack team. We’ll be waiting for you in the #p-chicago-lobbying channel.

This article was written in collaboration with Sharon Brener and Stephanie Kirmer, and originally posted on the data.world blog. And of course, it wouldn’t be possible without the hard work of Nate Wroblewski, Stephen Hoover, and all the other D4D Chicago hackathon participants!

This is the first installment in a series of posts highlighting the work done during D4D’s March 2017 hackathon. Check back in the coming weeks for more hackathon spotlight posts, as well as the conclusion to our Election Transparency post series!

Hackathon Spotlight #1: Linking lobbyists and Chicago legislators with data was originally published in Data for Democracy on Medium, where people are continuing the conversation by highlighting and responding to this story.

After a flurry of activity that brought in 220 new members and pushed us over 101,000 messages on Slack, the first Data for Democracy hackathon has ended! Thank you so much to everyone who helped to make this event a success by generously contributing time and energy, whether online or in person. The work to use data for social good still continues! Read on to find out more about the projects that took place this weekend, and how you can get involved.

The People

They say a picture is worth a thousand words, so here are some of our favorite photos from this weekend’s in-person meetups, where volunteers collaborated on data projects of local, national, and international scope — and, in many cases, met each other in person for the first time!

The Product

In addition to the work that was done on existing D4D projects, some very exciting projects got underway this weekend. Here are just a few examples of what was done!

On Friday evening, longstanding D4D partner ProPublica (in conjunction with the New York Times and the Associated Press) posted financial disclosures for 180 White House staffers. D4D volunteers rose to the occasion immediately to convert these documents into structured and accessible data, available here on data.world. They also created the above preliminary visualization to map the network of connections — people are rendered in red, and organizations are shown in green.

The Louisville team built an API of the 2017 Kentucky General Assembly, to make it easy for anyone in the state to look up the voting record of their legislators. The above is a first attempt at a visualization using this data — the Democrats who voted for GOP-sponsored bills.

Volunteers in NYC analyzed child adoption rates and created a decision matrix to explore how adoption is affected by race.

And there’s much, much more that took place this weekend — check out our Youtube playlist for recordings of our community showcase, where volunteers presented highlights from their work! We’ll also have a “Spotlight Series” on this blog in the coming weeks, featuring a more detailed look at several hackathon projects.

The Potential

So, what comes next?

If you couldn’t participate in this weekend’s hackathon, no worries! A weekend, as action-packed as it might be, is a short timeframe. Lots of these great projects have only just gotten started, and will be continuing past this weekend. We encourage you to join our Slack team, catch up on what’s happened so far, and then jump in to help. This was our first (and hopefully not last) global hackathon, and any future similar events will be announced through Slack.

We’d also like to emphasize that Data for Democracy started out as an online community, and remains Internet-based first and foremost. No matter where you are, you are still very much welcome and encouraged to be involved. The bulk of the activity takes place on Slack, so you needn’t worry about missing out!

We enjoyed the pizza, the stickers (thanks, data.world!), and the great coverage in GeekWire and local media. But ultimately, this hackathon was meant to provide an opportunity for a community of civic-minded data enthusiasts, spanning all levels of skill and experience, to come together and figure out ways to contribute to the public good through the use of data. Thank you all for making this happen, and for reminding us of what is possible when we work together and put our minds to it.

Let’s keep going.

The Data for Democracy Hackathon is Over! was originally published in Data for Democracy on Medium, where people are continuing the conversation by highlighting and responding to this story.

What:

The first Data for Democracy Global Hackathon.
Data for Democracy is a community of people using data to drive better decisions and improve the world in which we live. Join us for our first ever hackathon, where groups and individuals across the world will be working together on civic-oriented data projects.

Who:

The Data for Democracy community, including you!

Why:

This is a golden opportunity to apply your data and technology skills and interests for social good. Whether you’re new to the D4D community or have been an active member for months, and whether you’d like to jump on board with an existing project or get a new idea off the ground, you’re welcome to be a part of the hackathon!

When:

The hackathon will be from March 31 (Friday), 6 pm EDT, to April 2 (Sunday), 2 pm EDT.

Where:

If you can’t make it to an in-person meetup, no worries — a lot of activity will be happening remotely! Email us at team@datafordemocracy.org to get an invitation to our Slack. Check out our projects on GitHub, and some of our datasets on data.world.
The in-person meetups are being organized in some major cities. Find the one nearest you in the following list!

Austin, Texas:
Saturday, April 1, 10 am — 4 pm CDT
Dev Bootcamp
1705 Guadalupe Street
Austin, TX 78701
Register here

Boston, Massachusetts:
Saturday, April 1, 9.30 am — 3.30 pm EDT
55 Massachusetts Avenue
Building 5, room 5–134
Cambridge, MA 02142
Register here

Chicago, Illinois:
Friday, March 31, 5 pm — 10 pm CDT
Saturday, April 1, 10 am — 2 pm CDT
Metis Chicago
1033 West Van Buren St, 3rd Floor
Chicago, IL 60607
Register here

Louisville, Kentucky:
Friday, March 31, 6 pm — Saturday, April 1, 6 pm EDT
LouieLab
745 West Main Street
Louisville, KY 40202
Register here

New York City, New York:
Saturday, April 1, 9 am — 9 pm EDT
NYU Center for Data Science
60 5th Ave
New York, NY 10011
Register here

Seattle, Washington:
Friday, March 31, 6.30 pm — 10 pm PDT
Saturday, April 1, 8 am — 10 pm PDT
Sunday, April 2, 8 am — 5 pm PDT
Ada’s Technical Books
425 15th Avenue East
Seattle, WA 98112
Register here

Washington DC:
Saturday, April 1, 10 am — 4 pm EDT
WeWork Crystal City
2221 South Clark Street
Arlington, VA 22202
Register here

How:

We’d like to thank our sponsors and supporters who are helping to make this possible.

• Ada’s Technical Books is a Seattle bookstore that seeks to open the doors to technology and science as widely as possible, and provide a space for the tech community to gather.
• The Civic Data Alliance is Louisville’s volunteer Code for America Brigade, and advocates for open data, hackathons, and civic engagement.
• data.world is a social network for data people. It aims to provide a platform for data scientists to discover and share cool data, connect with others who share their interests, and work together to solve problems faster.
• Dev Bootcamp is an immersive coding bootcamp with locations in Austin, Chicago, New York, San Diego, San Francisco, and Seattle.
• LouieLab is a civic innovation hub and coworking space, designed to promote and build smart city projects through collaboration between private companies and the city of Louisville.
• Metis provides full-time immersive bootcamps, evening part-time professional development courses, online resources, and corporate programs for data scientists in Chicago, New York, San Francisco, and Seattle.
• The MIT Association of Computational Science and Engineering Students is a social organization for graduate students within the Center for Computational Engineering at MIT.
• The NYU Center for Data Science aims to create the country’s leading data science training and research facilities, and equip researchers and professionals with tools to harness the power of big data.
• Red Oak Strategic provides full-stack data science consulting services, including data engineering, predictive analytics, visualizations and data-driven strategies in the finance, corporate, political and other verticals.

Data for Democracy Hackathon: Happening This Weekend! was originally published in Data for Democracy on Medium, where people are continuing the conversation by highlighting and responding to this story.

So much has been going on since last month’s check-in that it’s hard to know how to put it into words. So let’s start with some numbers!

Data for Democracy has reached:

1038 members

More than 1430 GitHub commits, across 37 GitHub repos

Over 75000 Slack messages

That’s right — we’ve passed the 1000-volunteer mark and are still going! You can be one of the next thousand, just by emailing team@datafordemocracy.org. Read on for some exciting things and upcoming events you can be a part of, if you join the D4D community.

Project Updates

Here’s what some of our active projects have been up to in the past month, and where and how you can jump in.

Assemble

The Assemble team, which is developing tools for researchers to use to study online communities, has been focusing on developing and deploying approaches to community detection. Within the next few weeks, they hope to put together a cohesive approach to community detection that can be easily picked up and used by researchers.

The team chose to turn their sights to this because they view community detection as a vital component of several larger buckets of intriguing research.

• Information diffusion and social contagion: Namely, how information moves into and across a community. The past decade or so has seen a veritable explosion of research regarding how information flow happens in online social networks, and the team considers it important to understand how this happens in “far-right” and related communities.
• Bot detection: An important aspect of studying online social networks is the detection of digital personas. Why are they there? What are their characteristics? There are a great many avenues of research available for pursuit in this area.
• Language usage across communities: There are opportunities to build on recent research to predict changes in community composition based on their language use — for example, by examining the idioms, euphemisms, and memes that the community uses to communicate, and the graph properties of how this language is structured.

A team member, Henri, has made major headway in developing and releasing the preliminary version of a community detection algorithm. His work has been constantly supported by his other teammates who are working on data collection and curation across dozens of online social network platforms. It’s still in its early stages, but initial findings are promising, and the team has high hopes that this will evolve into a one-stop solution for community detection and bot detection.

How You Can Get Involved:

There’s still a lot more work to be done with curating source datasets and developing algorithms, and the team would love more fresh perspectives.

If you’re interested in novel approaches to community detection in online social networks, consider jumping on board at the #assemble Slack channel! This is a golden opportunity to study some emerging communities and how social contagion works across the “far-right”.

Drug Spending

The Drug Spending team is currently neck-deep in data acquisition, consolidation, and wrangling. Information on drugs and drug spending comes in a whole range of disparate datasets from multiple organizations, with little universal information to cross-match between them. In order to dive further into the available datasets, the team first needs to make sure they exist side-by-side in a cleaned and easily accessible form.

In line with this, the team has mainly been focused on acquiring and cleaning data sources. They are working hard at data consolidation — building indices and references between datasets, such that they can easily be associated with each other.

They have also ventured into making initial visualizations and exploratory analyses of these datasets. One volunteer, Stephanie, has been building an R Shiny app to visualize manufacturers and the drugs they produce, based on Medicaid and Medicare data. Another volunteer, David, is using a related dataset to visualize year-over-year increases in drug spending.

How You Can Get Involved:

The team is currently grappling with drug classification, and trying to develop a framework to associate drug names with their general therapeutic uses at a large scale. The project could greatly benefit from the input of contributors who have domain knowledge and experience regarding pharmaceuticals and health policy. If you fit the bill, your help would be very welcome! Check out the datasets and the #drug-spending Slack channel.

Election Transparency

The Election Transparency team has been busy over the last few weeks, working on collecting, normalizing, and standardizing historical data on county-level electoral outcomes. This effort continues to pose new challenges, as the format of the data varies widely — some states provide Excel spreadsheets, while others only make data available in PDFs. The team continues to work through these hurdles and find creative ways to build out their (publicly available!) dataset.

They have also started to build models and create visualizations, leveraging their newly cleaned data to help explain the outcome of the 2016 U.S. presidential election. One contributor, Robert, drew on the dataset to build a Shiny app that displays county-level Presidential election results back to the year 2000.

Moving forward, the team is starting to collect data and shapefiles to explore the impact of redistricting and gerrymandering on electoral outcomes. Through partnering with the OpenElections Project to build precinct-level data for all statewide races, the team hopes to address questions including:

• What are appropriate measures of a fairly-drawn district?
• How much do current district boundaries deviate from a fairly-drawn district?
• How well do current districts represent the demographics of the country?

Stay tuned for more detailed posts about the Election Transparency project in the coming weeks! These will delve into the nitty-gritty of the data collection process, and the key findings that have been made so far.

How You Can Get Involved:

The team is always interested in additional projects that will further their goal: to make the electoral system more transparent and easier to understand for everyone. If this is of interest to you, check out the project details and join the #election-transparency Slack channel, and let them know if you have an idea!

Internal Displacement

Based on a challenge set by the Internal Displacement Monitoring Centre (IDMC), the D4D Internal Displacement team is building a tool to populate a database with information about displacement events, which can then be used by both machine and human analysts.

So far, most of the team’s efforts have been focused on building a Python back-end to scrape, classify, and extract information from articles in IDMC-provided datasets. Information retrieval is proving to be an interesting challenge, in part due to the complexities involved with natural language processing. The team has recently succeeded in finalizing the database schema for the project, and is now shifting gears to building the front-end app.

How You Can Get Involved:

The team is looking for volunteers who are interested in helping to build the front-end app to visualize and interact with the database.

In addition, the team will continue their work to further refine the back-end code, work on the tricky issue of natural language processing, and implement online machine learning for new documents. It’s a good time to join the #internal-displacement Slack channel if you’re interested in any of these areas!

ProPublica

So far, the collaboration between ProPublica and D4D has centered on two main threads: data relating to official foreign travel of elected representatives, and data on House Expenditure reports. The team’s work has largely focused on loading and cleaning these datasets to standardize them, remove duplicates, and wrangle the data into a convenient format for analysis.

The foreign travel dataset is based on the House Official Foreign Travel reports, published quarterly by the House Clerk. ProPublica hopes to eventually use this dataset to track how official foreign travel expenditure has changed over time, and in particular whether and how it is influenced by political and international events.

The House Expenditure project was one of several undertakings adopted by ProPublica in 2016, after the closure of Sunlight Labs — an open source community run by the Sunlight Foundation, which sought to use data to increase government transparency and accountability. Through working with D4D on this dataset, ProPublica aims to detect unusual variances in spending by lawmakers. ProPublica also hopes to eventually add a search interface and make the data available for download.

How You Can Get Involved:

If you have any interest in investigating lawmaker activity and ensuring accountability, ProPublica is looking for more volunteers. Feel free to browse the data and join the #propublica Slack channel if it piques your interest!

USA Dashboard

The USA Dashboard team, which is creating a dashboard that will display key metrics for various regions of the USA, has been moving data into PostgreSQL; the D4D community can now access the data through Mode, and carry out reports and preliminary analysis there. The team is currently working on defining data documentation and writing dictionaries, in order to facilitate more targeted analysis.

How You Can Get Involved:

The team is seeking domain experts who can provide input on how to count crime reports to make a fair comparison across cities.

As the project broadens its focus to explore new metrics, including economy, poverty, and healthcare, the team would also like to recruit domain experts in these areas, who can help to develop and frame research questions based on the available data.

If you’d like to be part of this effort, drop the team a line at the #usa-dashboard Slack channel.

D4D in the News + KDNuggets competition

Besides all the activity going on internally, D4D has also received some great press coverage elsewhere!

KDNuggets reposted last month’s blog update, and we received a shout-out in a TechCrunch feature about Data.World, one of D4D’s most enthusiastic partners and supporters.

We’ve also sponsored a “Data Science vs Fake News” contest, in collaboration with KDNuggets and Data.World. The deadline for submissions was on March 10, but if you missed it, no worries — there’s another upcoming opportunity to show off your data science chops!

D4D Hackathon

We’re excited to announce the very first Data For Democracy Global Hackathon! More information about this event will be given in an upcoming post, but here are the key facts:

FROM: March 31st, 2017, 6 pm EDT
TO: April 2nd, 2017, 2 pm EDT

Anyone can participate, simply by signing up as a D4D member (again, shoot an email to team@datafordemocracy.org and we’ll get you sorted out)!

Since this is a global group, many people will be working remotely, using tools such as Google Hangouts, Slack, and GitHub. In-person meetups are also being organized in some major cities; again, more details will be coming nearer the date. Feel free to arrange a meetup of your own!

The Hackathon will conclude with a showcase on April 2nd, 2 pm — 3 pm EDT, where the D4D community will display and demonstrate what they built during the Hackathon, in a series of short presentations.

If you have a cool project idea you’d like to get off the ground, or want to get involved with D4D but don’t know where to start, this weekend will be an excellent opportunity to dive straight in. Don’t worry about not knowing anyone — by the end of the Hackathon, you will.

Keep an eye out for more news in the coming weeks!

D4D at the Three-Month Mark was originally published in Data for Democracy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Since its inception in December 2016, the small community of Data For Democracy volunteers has grown into a network of over 700 people, spanning a range of locations, timezones, and backgrounds — as you can see on our brand new website.

This group of passionate and civic-minded people is applying a diverse set of skills and knowledge to an equally varied selection of projects, and has made remarkable progress in the past two months! Here are some highlights of what we’ve been up to.

Assemble

The Assemble project is working to develop a toolkit and technological infrastructure that researchers can use to study online communities and their characteristics. With Ben and Nick at the helm, this scrappy team has rocketed past various milestones including:

• Setting up a streaming data pipeline and database, with the generous assistance of our friends at Eventador. Mode Analytics has also donated their platform for D4D use, meaning that Assemble’s social media data, in Mode, will soon be coming to you via Eventador!

• Holding a weekend hackathon to develop and refine Twitter data collection capabilities.
• Starting work on a beginner-friendly scraping project, with the first milestone being to collect the 2017 congressional record.

Drug Spending

This team, headed by the intrepid Matt and Jennifer, is focused on researching where and how Medicare tax dollars are being spent, and presenting these findings in clear and accessible ways. They’ve recently started constructing a Shiny dashboard that will make these visualizations much easier.

USA Dashboard

This project is working on creating a dashboard that will display key metrics for various regions of the USA. They’ve gathered crime data for various cities including Chicago, Washington DC, New York City, and Philadelphia, and have developed a roadmap for where the project is headed next, both literally and figuratively! You can also hear project lead Sean discuss this work on the Partially Derivative podcast.

Election Transparency

Led by Scott, Chris, and Rachel, the Election Transparency project works on collecting and normalizing county-level election results, to be shared with the public. The team has put together extensive datasets of election results and population demographics, and these are all published and available for browsing thanks to the support of our friends at Data.World. Moving forward, the team plans to create various models and visualizations that will help to explain election outcomes.

ProPublica

This team carries out data analysis to support the work done by the non-profit investigative journalism organization ProPublica. With the fearless leadership of Eric and Ryan, the team is investigating campaign spending expenditures, and has recently developed a nifty text-scraper to navigate some messy reports of government officials’ foreign travel.

What’s Next?

“Is D4Ding a verb? Because that’s what I’m doing all weekend.” — Ben

This has just been a small sampling of the many projects Data For Democracy is currently tackling. Even more new projects are always taking shape — whether initiated by our own community members, or taken on through partnerships with other organizations. In particular, we have some exciting collaborations brewing with the Cities of San Diego, Los Angeles, and New York.

If one of our projects piques your interest, or you’d like to propose an idea of your own, come join us! There are all sorts of ways to get involved, no matter your level of commitment, skill, or experience.

If you’d like more detail on possible ways to jump in, and why you should be a part of this, stay tuned for upcoming posts!

The First Two Months of D4D was originally published in Data for Democracy on Medium, where people are continuing the conversation by highlighting and responding to this story.