Pocket Democracy: Empowering Voters using Google Cloud Vision, IBM Watson, and RevSpeech

A more informed voting experience.

The Issue

“If I don’t know either candidate in the race, I’d go by picking the Democrat over the Republican, then women over men, then names that sound like they come from like some kind of racial minority or something, then from there we’re just straight up guessing.”

Over 30% of voters fail to complete their ballots every year. Political scientists attribute this to an absence in information which causes the SAT effect- if you don’t know, don’t answer it. Even more, researchers have found that candidates listed first on the ballot can receive up to 5% more votes. When they don’t have the information they need, candidates’ names, ethnicity, and gender can affect how voters make decisions (Axelrod and Murphy). The above quote from a voter we interviewed vividly illustrates this fact.

There are several issues surrounding voter engagement, voter registration, and disenfranchisement policy, but, for the scope of this project, we focus on the specific interaction of the registered voter filling out their ballot. We ask, how might we help a voter make a more informed, more personal decision at the booth?


Mailiis Law- Back-End Developer

Piu Punia- Back-End Developer

Anna Kambhampaty- UX Engineer & Copywriter

Joshua Shao- Product Designer


We used insights from observational research and interviews to inform our design.

Assumptions we made about our users:

  1. Voters care about making a more informed decision
  2. Voters are rational; their decisions are driven, to at least some extent, by empirical data and reason, not just emotion and sentiment
  3. Voters have access to a mobile smart phone (we know this is not the case for all voters, we examine this further below as an issue to be addressed)
  4. Since we are dealing with users who already care enough to vote, we are assuming they have at least some base level knowledge about the political landscape
  5. Voters do not have all the information they need on candidates to make the best decision for themselves by the time they get to the polls

Observational Research

The New York State Democratic primary was just two days ago, so, conveniently, I was able to identify pain points by observing voters as they went in and out of the polls.

Time of day: 4:25 PM

Location: St. Luke Lutheran Church (Ithaca City, Ward 4, Districts 2 & 3)- a prominent polling place for students and Ithaca residents living on Cornell’s West Campus or off-campus.


  1. Voters did not spend much longer than around 5 minutes each in the poll.
  2. Voters that came in groups tended to pressure their friends who were taking longer to vote to hurry.
  3. In response to being asked how they felt after voting, most responded very positively, taking pride in having voted.
  4. Everyone who voted took an “I Voted” sticker and often placed it on their shirts.
  5. In response to where they were going after they voted, responses ranged from athletics practice to club meetings. No one responded that they had leisure time following.
  6. In response to where they were coming from to vote, many responded class, academic meetings, or the library. Few responded that they had leisure time prior.


  1. Voters were generally in a rush.
  2. Voters did not want to spend much time in the polls.
  3. Voters had things to do before and after voting. They took time out of their days to do this.
  4. Voting was an important, prideful duty to citizens. They enjoyed the act and felt it was important.

User Interviews

I conducted two in-depth, in-person user interviews for research purposes. These interviews represent two distinct voter types that we aimed to design for:

Ryan Matsumoto: 23, Male, Google Employee, New York

Ryan, registered with a political party, votes every election and reads the news daily.

Hannah Morris: 20, Female, Mechanical Engineering Student, New York

Hannah votes in all elections, including primaries and local elections. Issues particularly important to Hannah that affect her voting decisions include environment issues, gay rights, and women’s rights. Hannah reads the news almost every day and stays informed for herself but tends to be quiet on political issues.

I asked both users to look at the following ballot and walked through a mini mock vote casting interaction with them to identify exactly where the pain points lay.

Here are their answers to some of the particularly revealing questions:


Q: Are there any names on this ballot you don’t recognize?

A: “Yes, the last six.”

Q: For a specific position where you don’t know either candidate, would you ever leave it blank?

A: “No, I would try and figure out who to vote for.”

Q: How do you make your decision in this case?

A: “I’d google their names and look at recent news relating to their positions on issues and past experiences.”

Q: Are there any particular sources or news sites you’d go to for this?

A: “Since it’s mostly local politicians I’m not familiar with, I’d look for prominent local news sources for information.”


Q: Are there any names on this ballot you don’t recognize?

A: “I didn’t know the names for sheriff or state committee.”

Q: For a specific position where you don’t know either candidate, would you ever leave it blank?

A: “I personally wouldn’t leave any blank although I don’t have any good reason to this.”

Q: How do you make your decision in this case?

A: “If I don’t know either candidate in the race, I’d go by picking democrat over republican, then women over men, then names that sound like they come from like some kind of racial minority or something, then from there we’re just straight up guessing.”

For Ryan, we attempt to make a tedious process easier. For Hannah, we do the same but also add more logic to her voting process. These are the two main user types we are designing for.

Note, we interviewed two active voters. We can assume that less active voters might know even less of the candidates. And, even though it’s simply a mid-term primary election ballot we looked at, we were dealing with 14 different elections. This is a lot on the line, but also a lot to ask the layperson to stay informed on prior to coming to the voting booth.


Questions that guided our speculative brainstorm session included:

  • How can we get voters to make more informed decisions?
  • What information might voters need to do this?
  • Where do voters get their information from?
  • What information do voters trust?
  • How do we take into account varying degrees of interest in political issues?

Information we thought voters may need:

  • Links to trending news articles on candidates
  • A sentiment analysis of news articles mentioning the candidate
  • Candidate background
  • Candidate stances on relevant issues

Focussing on such a specific interaction led us to realize that we needed to create a tool that synthesized the above information for a person to easily make sense of as they were filling out the ballot in the voting booth. Here’s what we came up with-

Solution & Prototype

Full prototype- https://brave-lamport-9d4a08.netlify.com/

Our solution is an augmented reality experience that allows a user to scan their smartphone over their ballot. Our app, Pocket Democracy, then picks up the names on the ballot and lets the user click them to reveal relevant information, popular news links, and sentiment analysis of articles relating to the candidate. Pocket Democracy also supports speech-to-text and text-to-speech recognition and processing.

Prototyping & build tools:

  • Google Cloud Vision’s Optical Character Recognition API
  • Google Cloud’s Text-to-Speech API
  • IBM Watson’s Discovery News API
  • RevSpeech’s Speech-to-Text API
  • React.js
  • Node.js
  • A-Frame
  • Sketch

What we were able to Build:

Tech Flow

We developed a web app that first processes an image of the ballot using Google Cloud Vision’s Optical Character Recognition API to detect and then extract the text from. We grab the candidate names in the form of text and pass them in queries to IBM Watson’s Discovery News API. We use this API to scrape the web and gather the relevant information on the candidate- stances on prominent policy issues, news links, and sentiment analysis of news articles. We also utilize RevSpeech’s API to implement a speech-to-text feature for accessibility reasons. A user can say a name into the app, and it will pull up the same relevant information on the candidate. The app also has the ability, thanks to Google Cloud’s Text-to-Speech, to speak the relevant information it scraped back to the user. Beyond just accessibility, this also makes it so that the user does not need to be in front of a ballot and can get informed prior, as well.

Our Mockup:

To illustrate the full user journey and incorporate Augmented Reality

Potential Issues & Moving Forward

We’re not trying to make or unfairly bias a decision for the voter. We’re trying to give the user the information they need in a quick, digestible manner to make the best decision for themselves.

There are three key issues that need to be addressed in moving forward with this project- information framing and algorithmic bias, smartphone accessibility, and human tendency to act on emotion rather than reason.

Information framing bias can occur when the inclusion of certain information pushes a person to view the issue at hand through the context of that category. For example, if we include a candidate’s race on their profile, a user may think that, when voting, it is highly important to take into account race. A similar effect can occur with information exclusion (Barocas and Levy). With this in mind, we have to be highly cognizant of what information we decide to include and exclude. Further research and consultation with experts in law and ethics will likely be required to do so. To further reduce bias, we’d need to interview a larger and more diverse set of voters on voting criteria.

Algorithmic bias will ensue since we are pulling information from news sources, which may hold their own biases and are thus magnified through the app’s reiteration of them. Further, the biases of the algorithm and interface designers will be imposed through the app itself. A multi-faceted, diverse set of designers and developers can help limit bias.

In the realm of phone accessibility, though we designed this technology for a smartphone user, it is not our aim to push the falsehood that informed voting is a privilege, given only to those who have the means to own and operate a smartphone. To combat this, an idea we have is to partner with polling locations and municipalities to provide smartphones for voters to use while they’re voting. We also hope to follow digital accessibility guidelines in a further iteration of our product’s design, to ensure availability and usability of content, including those with hearing, visual, motor, or cognitive impairments (Expanding Accessibility to Digital Spaces Through Improved Policy and Practice).

Lastly, it is important we consider the fact that humans are not always rational. “ Even when we gave them empirical data that pushed them one way or the other, that had no impact, or it only hardened their emotionally biased views” (Packard). If humans act purely based on emotion, they might not find any proper use for this app. However, the existence of this app will hopefully address that issue to reemphasize that reason and facts matter, especially while carrying out a civic duty.

Before moving forward with our project, extensive research in information ethics and user testing for accessibility and usability will be required. Then, we can iterate on our design in an informed manner to make it as accessible and equitable as possible. Algorithmic and news source bias should also be addressed in the future. We’d like to implement a personalization feature as well as a simple text input feature. We also need to more smoothly connect the varying components of our project. A reminder of our original mission- to help the voter make an informed decision with ease for themselves!

Thank you for reading! As always, please feel free to reach out with questions, concerns, and musings.



Media, tech, art, & the collisions of them all. Senior at Cornell. Previously NYTimes, CNBC, and Syracuse Office of Innovation.