Why the NGP VAN model has been awesome for democracy (and how it could be more awesome)

There has been a ton written about #VANghazi so far. I started writing this for publication but it got a bit long, so I’ve given it a quick edit, broken it up into four parts and posted it here. If the conversation is still going in January, I’ll revisit revising and publishing it then.

Here’s the best writing about VANghazi so far:

Will Conaway, NationBuilder: Why the NGP VAN model hurts democracy (Medium)
Josh Hendler, former DNC: Why The Bernie Breach Isn’t About Technology, And What’s At Stake (Medium)
David Atkins, CA Activist: An Explanation of What Bernie Sanders Staffers Actually Did and Why It Matters (Washington Monthly)
Dave Dayen, Journalist: The real scandal in the Bernie/DNC feud is the one nobody is talking about (Salon)

Earlier this week I had the strange feeling of waking up and checking my social media feeds to find very large numbers of people discussing a topic that has consumed large amounts of my life but, before this week, has seemed utterly inaccessibly nerdy: voter file technology and the rules by which campaigns and parties provide them. The suspension of the Sanders’ campaign access to the national voter file system called the VAN by the DNC has now been a multi day issue and even came up in the debate.

There seems to be some consensus on this issue fizzling out, but here’s why I think it’s important enough that I’m up late writing something about it two days before Christmas: I believe getting this right is critical for the future of the Democratic Party and the progressive movement — and even US democracy itself, given the participation crisis of 47 million (and rising) voters not showing up for off year elections. And contrary to the arguments being made, the party, vendors and the broader movement are doing a pretty good job of it despite the rough sledding this week.

In part I of this article, I’ll look at some of the technical and policy arguments being waged against VAN, and in part II I’ll talk about the situation in California where the party has chosen to use a different voter file system, provided by Political Data, Inc. (PDI)

A bit about my background and biases: I’ve been a software developer for over twenty years, and from 2007 to 2010 I was the California VAN administrator for a small shop called California VoterConnect, and then for a few months afer that working for TargetSmart Communications. I haven’t made a wooden nickel from voter file work since I moved on from TargetSmart in 2011, and while I still follow it and I am active in the civic engagement space, I’m not planning on getting back into data, so I have zero financial incentive for any of this. And I think both Sanders and Clinton would be outstanding presidents; I donated a few bucks to Sanders after the story broke out of frustration with a perceived pattern of Democratic party insiders pulling this kind of thing every four years.

My overall theory of change is cribbed from Joe Trippi’s great book about the Dean campaign, The Revolution Will not Be Televised. Trippi argues that we’re in a slow transition from the top-down, broadcast, money dominated system, to a more networked, participatory and social system. If this hypothesis is true — and I hope it is, since it’s hard to have much hope for American democracy otherwise — then data systems and how they are made available is a critical issue to get right.

But I heard VAN sucks

I have enormous respect for both David Dayen and David Atkins, who both have written widely circulated pieces about this. Unfortunately they’ve made some claimes that are contrary to my experience as both a field organizer, voter file provider and application developer.

David Dayen’s writeup on salon.com claims that:

“What’s important for Democratic campaigns is not the software, but the underlying data. Anyone can serve that data and give campaign tools to slice and dice it.”

And David Atkins claims in his Washington Monthly piece:

“The first thing to understand is that NGPVAN is a creaky voter database system that looks, and feels like it was put together in the 1990s.”

Additionally, Will Conway, an employee of NationBilder (another competing system organizing tool that includes a voter file, which is nonpartisan, but signed the biggest political technology deal with Republican state parties and is currently powering the Trump campaign) writes:

“…a piece of campaign technology broke because it’s bad. The reason the software is bad is not because the developers at NGP VAN are in some way inferior — it’s because there is no incentive to improve their product.

The VAN5 launch video and this analysis of NationBuilder’s data quality are both good rebuttals to these arguments, but I have four more lines of disagreement based on my experience:

First, what VAN, PDI, and NationBuilder all do is technically fairly difficult to do right, but scaling out to the incredible amount of traffic that VAN handles around election day is particularly challenging. I know, because I’m one of not very many people who have both run a campaign data operation and written software for the kinds of tools that can power them. David and David, both of you are invited over to my place in Oakland to talk database architecture and scaling performance next time you’re around if you genuinely thinks it’s so easy that any old firm can do this. We can run some queries, do a little SQL, talk about forecasting infrastructure needs for election day loads. It’ll be great.

Second, while it’s true the data is important, the capabilities of the software are absolutely critical and shape the scope of the kind of field campaign that can be run. More about this shortly.

Third, there is absolutely competition to the VAN. The number of organizations attempting to jump all over this — some of them with borderline ridiculous spin that Will from NationBuilder’s piece included, claiming that NGP has no reason to innovate — should be enough evidence of this. There are a number of different systems available that the DNC and progressive organizations could switch to at any time. Organizer is an outstanding and recently developed app that works with NGPVAN and other back ends. While not a direct competitor, Catalist functions as a sort of data repository for progressive groups and after a somewhat rocky beginning. The DNC and large numbers of progressive organizations pick VAN largely because it does things no other system can do.

Fourth, there is no reason to believe that a more market based approach would produce a better outcome than the more-or-less socialized approach the Democratic party is currently taking. There’s even an existing alternative approach that seems to indicate it would be vastly worse: the famously chaotic data infrastructure on the GOP side.

OK, maybe the VAN is good, but is it more like a monopoly like AT&T, or more like infrastructure, like roads and bridges?

But the question remains: is the political data market more like the market for artisinal chocolate, where lots of small vendors are a good thing for consumers, or is it more like the market for roads and bridges: infrastructure that everyone shares but is expensive and tricky to build and maintain?

I believe strongly it’s the latter. Of course VAN isn’t doing everything right, but I believe there’s a preponderance of evidence that the approach the DNC and VAN have taken by centralizing this service has been a huge structural advantage.

The main counterarguments revolve around VAN being a centralized point of failure. It could have a technical failure, or it could have a policy failure, where party insiders foul something up. The Sanders campaign fiasco last week was arguably a mixture of both.

But here’s the thing: compared to the technical failures on the Republican side — remember Romney’s much hyped turnout tool, Orca, which crashed in a flaming mess on election day in 2012? — the technical issues on the VAN side have been relatively trivial. There’s a saying: good software takes ten years. In my experience that’s true for everything from the VAN to the tool I use to dj from my laptop. While I was working at VoterConnect, slow individual queries or systematic performance issues did crop up periodically. But they were always handled quickly and aggressively by VAN support. It was to this day the single best supported system I’ve ever used, and their rapid turnaround quickly became legendary among our users and campaigns.

That being said, I always encouraged campaigns to have backup lists stored before the last weekend of the election. Any complex technical system is vulnerable to failure. Vendors that make contrary claims should be regarded with deep suspicion.

And on the policy side, what’s amazing about this moment is that there are thousands of people watching the Democratic Party and what it does. They can’t pull this kind of crap anymore. I’m far from the only one who cares enough to stay up late writing up detailed arguments because lots of people now get why this is important.

In the initial wave of spin a lot of my friends said, “screw the Democratic party, they’re so corrupt, Bernie should quit.” But that’s exactly the wrong approach. We’re just now getting to the point where the corruption is becoming visible. The solution isn’t to abandon the party, it’s to get more involved and love it harder, even while admitting there is a hell of a lot more work to do.

Why software (not just data) really matters: VAN vs PDI in California

The PDI vs VAN situation in California, viewed through the lens of the organization I built, California VoterConnect, is an excellent illustration of the importance of the software itself, not just the data. VoterConnect was started in 2007 by some of the sharpest and most strategic progressive donors in California, with Steve Phillips taking the lead on shaping the strategy. They were enormously frustrated with PDI and felt that outsider campaigns, particularly those trying to reach racially and ethnically diverse communities, were priced out of the market.

Our first topic was to do a full technical evaluation of both VAN and PDI. VAN had tremendous, overwhelming technical advantages over PDI: it was vastly faster, had (even at the time) super flexible and granular data sharing capacities, and was cheaper and available and under a much more flexible license that would let us share data between groups. On top of all that, they only worked with progressive and Democratic groups.

After the technical evaluation, when we described our organization’s mission and proposed structure to PDI, their response was succinct: “We don’t think there’s anything interesting here for us.” The contrast with VAN’s founders, who flew to California to talk through our strategy, couldn’t have been more stark. Sure, they wanted our business. But why didn’t PDI?

The primary benefit of the VAN — it’s ease of use — became apparent over the next few years. I was able to go to a nonprofit or small campaign doing voter contact in underserved areas, do a two or four hour training, and leave their team with access to the tools and capacity to run their own field campaign from top to bottom, with only minimal ongoing assistance from me. They could set their own targets, pull their own lists and print them or send them to mobile devices, then put the results back in. I did several trainings to roomfuls of organizers through a translator where I was the only one speaking English.

I have a number of theories as to why CA Dems still haven’t switched to VAN. Part of it was my own fault: VoterConnect’s primary and really only mission was to stay laser focused on delivering voter data to organizations that were serving underrepresented communities. We focused on executing towards that goal and were tremendously successful, including getting to breakeven revenue in three years. Persuading the party appartus to switch never rose above a tertiary goal. But for reasons that were made largely above my pay grade, we ended up selling the project to another firm that eventually shut it down.

But the main reason is that it hasn’t mattered as much as in other states. We are as far from a Presidential battleground as you can get. And for most downticket races, traditional top down field operations with one or two data managers at the top and everyone working off their lists works just fine: there’s no urgent need for a distributed approach. In many places, even more rickety “cosmetic field” operations, that are really just checkboxes for campaign staff but result in very limited volunteer engagement, are plenty to win given the outsized Dem majorities in most places.

But there’s still a lost opportunity here. California should be leading the change from broadcast to participatory politics. Instead, we’re a forgotten backwater.

Virginia 2008: What the Participatory Future Could Look Like

My wife and I were in Northern Virginia two weekends before the election in late October of 2008. The VoterConnect VAN was humming providing data for 140 campaigns across California, but my favorite Aunt had died and I couldn’t bear the thought of not being with my family. We attended her funeral on Saturday and on Sunday we did my aunt would have encouraged us to do if: we ventured out to join the swing state Obama field campaign.

What we found shocked us. We went to four different field offices — the energy was palpable and electric in all of them. The first three kept sending us to smaller and smaller neighborhood offices. When we got to the fourth one, the neighborhood leader who couldn’t stop grinning told us, “You’re never going to believe this. But we’re out of turf. We’re… I think we’re actually done.”

Ordinarily a field campaign that reaches perhaps 10% of the voters in a given area could be considered a raging success. This kind of coverage was basically unheard of. Of course this level of energy didn’t come from VAN: it came from eight years of Bush and an exciting chance for change in Obama.

But having the VAN was what turned that organization from a seething mob of political potential energy into an actual electoral force. It wouldn’t have been possible to do this nearly as effectively with a more difficult, slower, more centralized system like PDI, even if they had been able to handle the incredible technical challenge of scaling their system to handle that kind of national load.

The feeling of that energy, that level of mass participation has stuck with me since then. Eight years later, I remember the expression on that organizer’s face like it was ten minutes ago. That’s how democracy should feel. That’s how it should work. The data matters, but the tools absolutely matter too, and the DNC and NGPVAN have — this recent debacle notwithstanding — done a largely terrific job moving us towards that future so far.

The biggest single problem facing American democracy right now isn’t Donald Trump. It’s our lack of Reflective Democracy. We’re not going to reach the more than 47 million voters that are threatening to go missing again in 2018 by running TV ads at them or even through Facebook. Would a distributed army of engaged activists going door to door be a better bet? Maybe, but if so, the leadership for finding out if that future is possible is not coming from California. But the long-term investments in VAN by the DNC and state parties outside California may yet have laid the foundation for it.