POLLY: User Testing Report

In the past two days I conducted user tests with the target group of seven high schoolers as well as two millennials. I started off by briefly explaining the POLLY app, what it is and what my goals are for the users. I passed out paper prototypes and then moved to the laptop where I had an Invision Studio proto they could click through. I recorded all of the conversations on my phone and went back afterwards to transcribe them. Below are my biggest takeaways on improving what I have so far as well as detailed notes and transcriptions below them.

MAIN USER TEST TAKEAWAYS

1. It was clear from all of my interviews that a simple intuitive onboarding would be needed to get the user accustomed to the features and how to use POLLY. Onboarding is always done at the end, but the user testing gave me insight into which areas needed the clearest onboarding instructions.
2. The points system throughout POLLY needs to be clarified. Users were unsure if they were able to keep their points and status levels if they traded in their points to get merch or donate to charities. Many brought up having a dual system of XP (experience points) and some sort of coins you can collect for rewards or donations. I will be focusing on this aspect since it seems important for young users used to gaming.
3. How points are distributed needs to be ironed out. Users like being rewarded with points for all of their various actions within POLLY, but the challenge is how to subtly give clues at the points they can earn.
4. The home screen needs to be improved. There are a lot of nice features they feel they would use but overall the interactivity seems cluttered. Will look into collapsing sections and having better UX/UI for the home screen. Some users are used to scrolling so didn’t find the list of features an issue. Still worth exploring to make it cleaner and more user-friendly.
5. They like the flow of the quizzes and the language of the copy. The UX of fact cards need to be improved. Swiping horizontally is preferred, keeping the content tight and digestible. There were debates whether to follow wrong answers with simple facts cards or at least the option so users can learn from their mistakes. Definitely worth exploring.
6. Start to fill in the gaps with simple screens where they get a congrats on a level achieved.
7. Some users like the ideas of badges when finishing a level, like Mastery of Climate Change if you finish say 3 levels of that issue (TBD). Could translate to coins for rewards or just be badges of honor.
8. Making the game infinite levels is preferred but provide notifications that the user should try different issues out. Look at return engagement.
9. They love the Corporate Social Responsibility (CSR) aspect of the donations to causes. Need to look into this for presentation. They also love merch and suggested popsockets, stickers, bags on top of the apparel and water bottle ideas.
10. Work on the information architecture. More wireframing!
11. Is there a need to make the home screen customizable? Do we offer avatars?
12. The response was positive overall. The users expressed a genuine desire for POLLY to be a real app that they would use.

“I feel like every kid in school should have this app.”

My next steps are to synthesize all of these takeaways and really push the prototype further.

SESSION 1

Kinza & Paula (classmates)

My first session was with two IxD classmates. I showed them Sketch artboards and they gave me some comments on UX/UI as well as commenting on copy language. This was helpful as that evening I had to make my Invision prototype for the high school user testers to use the next day.

Because they both have a UX/UI background, I received some helpful comments on some of the UX flow issues I ran into as well as some UI elements I was trying to iron out. Additionally there were some wording edits I made for feature headers in particular that I was able to change and clarify as I shaped the clickable prototype for my user tests the next day.

There were some suggestions by them to make the home screen less scrollable and perhaps break them down into menu sections like:

• Learning
• Quiz
• Fun

I definitely will look at the Information Architecture in more depth this week so that the UX is clearer.

SESSION 2

A = Ady (17 yr old — Walter Panis HS Senior)

T = Tyler (17 yr old — Walter Panis HS Senior)

SESSION 3

N = Nick (16 yr old — Lakeland HS Senior)

K = Kati (17 yr old — Walter Panis HS Senior)

SESSION 4

M = Mitch (14 yr old — Croton Harmon HS freshman)

J = Joe (14 yr old — Croton Harmon HS freshman)

SESSION 5

B = Bella (17 yr old — Croton Harmon HS freshman)

In the second and third sessions the next day, I had 2 group of two users, mostly high school seniors and one junior. Two of them were familiar with my project as they have been helping me throughout the process, although the latest permutations look very different from when I last met with them. The latter two were seeing this app for the first time. At the end we joined the two groups and had an open discussion.

The fourth session involved two brothers, freshman in high school, very different from the seniors in terms of political involvement.

The fifth session was with a 17 yr old high schooler named Bella. My audio actually didn’t come out cleanly so I lost some of the interview. I reached out to her on some key points however.

All provided great honest insight. I decided to combine the comments below for future reference.

I utilized the paper prototypes by asking them to write notes directly on them . It was an informal session so we did have discussions throughout. The individuals were opinionated and unafraid to debate when they disagreed on something. Even though most of the sessions throughout the day consisted of two people plus myself simultaneously, I felt I was able to get honest answers from all of them. Many of these sessions resulted in open discussions. We kept it loose and informal so I was peppered with questions and comments as we went through each element from each screen.

The first thing I’d like to show you are some paper prototypes of POLLY and then I will show you a prototype on my computer. Please look at the paper protos and point out any questions you have about any of the elements. Feel free to write on them.

“So what is the plan on getting this app to the public?”

“Kinda like how when you get started on Instagram, it highlights various features for you so you know how to use it.”

“Honestly it’s really easy to advertise things on instagram.” “Some people slide over to the Stories tab and there are the little advertisements on there.”

“I like the illustrations.”

Now let’s go through the screens from the beginning. I’d like you to click and scroll through the proto and tell me about each element from top to bottom, what you think their function is, what you think will happen if you click on something.

ISSUES SCREEN

Users really like the visual language and illustrations of the various issues menu. This was a consistent comment throughout all the interviews.

K: “I really like these illustrations though”

M+J: “I like how they look and this one it says SUGGEST AN ISSUE. I think that’s good because it makes it more interactive for the users.”

ME: Yes, if a lot of users suggest an issue and feel strongly, our content team can add it for later.

Now let’s check out the home screen which shows all the features that POLLY offers the user.

HOME SCREEN

A: “OK I see that the top has a profile pic and name. What are these icons for? I thought it was for customization. I’m into the idea of making it customizable and yours.”

A: “I’m not sure what Charity $ is or what Current Level means. What do they represent?”

M+J: “What does it mean by charity?”

T: “So you can only donate to charity?”

T: “Oh, I like the word Pollytics a lot. I think whoever came up with that had a spark of genius.”

T: “I like the points, level and money being up top because it’s a good reminder of the system that POLLY is all about.”

T: “Maybe equal the screen share between the quizzes and what you’re gonna get (rewards).”

T: “So clearly you want easy access to the quiz and store and view representation of your points and your level.”

T: “I do think it’s a good idea to have all these features, but it’s not entirely intuitive. Is the idea that you scroll down?”

T: “I definitely think there should be a button that is on the upper left or right so that you can quickly switch to a tab. If you want to get all the way down here. You don’t want to scroll every time.”

T: “Maybe add a way to collapse the features so you see them all at once and don’t have to scroll so much. Like on Instagram (hamburger) how you can click and all the feature options slide over from the right)”

ME: Yes you scroll and it will prioritize based on key features. What about collapse?

T: “Maybe a Reddit type: ‘collapse the comments’”

T: “Probably start off as with all features open and the user can see all the features the first time and then it’s collapsed.”

A: “Should you be able to re-order the features, customize to users preference? The quiz can always be first.”

(Maybe the user can customize what they want collapsed and re-order the features)

The HILL Status UI (Citizen<Mayor<Governor<President)

A: “I like this angled look for the status. It’s a good visual of moving up in status.”

T: “I think the angled thing is good visual representation of progress.”

A: “What are these icons for (Editing and settings)?” Editing your username and picture.

A: “I’m into the idea of customization and making it yours.” “Maybe different themes”

A: “When I see that (edit) box I think, comment, like twitter or instagram”

M+J: “We get it that you advance as you get the points. Once you get to that level, it keeps you at that level so where it says Citizen it will change to Mayor. And then you can use those points for different things.”

ME: Suggestions on points. How can I keep the pts that I have but also cash them in for something?

M+J: “So the points you get from the quiz that go towards being a mayor and stuff can be XPs but the pts that you redeem maybe they can be kind of sepaparet or similar. In a game they’d kinda be coins and you’d redeem for stuff.”

M+J: “If you have those titles maybe you get coins”

M+J: “Pts go towards your rank?” “If it doesn’t take that long (to complete a level) I’d say maybe the points are XP for citizen, mayor, etc.”

ME: They don’t understand the levels on homes screen. Replace with the coins.

“You would need a shop area where you can exchange for charity or merchandise.”

QUIZ LEVELS

ME: Feedback gave me decision to change this simply to QUIZZES

N: “This is clear. The point system I think is how many points I will get.”

ME: (It’s how many the user CAN get. Need to work on this.)

K: “I need to do the Pollytics Level 1 first before Immigration Level 1 because its grayed out”

LATEST QUIZ

T: “This makes me think its is the last quiz I did.” (CHANGE this wording!)

T: “Maybe MILESTONE QUIZ or CURRENT EVENTS?”

A: “It’s the last quiz you left off of? Oh, it’s the most current quizzes?”

K: “Well I think if someone had this app and it wasn’t the last quiz they took they would know.”

N: “Call it Milestone Quiz”

M+J: “Maybe call it Monthly challenge or Featured Quiz or Featured Topic. Earth Day or BHM or Citizenship and under it could be Gun Control..Space Exploration”

ARTICLES OF NOTE or FOR YOU

A: “I like ‘FOR YOU’ but maybe it can be ‘ARTICLES FOR YOU or ARTICLES OF NOTE” (Should be able to save these articles, add a little tab or three dots)

N: “These are articles that could give you points”

A: “Its for you specifically.”

K: “Maybe ARTICLES FOR YOU”

K: Change it to maybe ARTICLES FOR YOU or FOR YOU

KK: “The whole thing is FOR YOU but the wording seems fine.”

T: (should be able to save it) a bookmark or three dots

M+J “FOR YOU I like the recommended for you .So you can get pts from reading one of the news stories? Oh those are news stories? I see the NYT. So it’s based on the issues you’ve chosen? I like that.

ASK THEM ABOUT THE +PTS BUBBLES

T: “It’s clear you just randomly chose numbers.”

A: “You don’t want to make it discouraging that you have to do 1000 quizzes.”

N: “That’s how many points you will get.”

M+J: “These are the points you could get if you got everything right. You can kinda tell because the green matches up with the green from home screen.”

M+J: “Anytime you see this, it means you will have a chance to earn these points.”

SAVED

ME: This was very clear to users.

N: “I figure you go through the quiz, save content and later it will show up”

K: “Are they facts that you took from the quizzes so you can save facts later?”

M+J: “These are facts you save for the quiz challenges you’ve beaten or completed. And they show up there after so you can check it later. I like that.”

LATEST LEADERS

Without seeing the profile pics, they thought it was about political leaders.

A: “I was confused earlier because I just heard you say LATEST LEADERS, and didn’t see it”

T: “Ohhhhhhhhh, I thought it was like leaders in Congress and govt.”

T: Just call it LEADERBOARD or HIGHEST SCORERS!

K: “Does this mean people who have the most points?” (CALL IT LEADERBOARD)

M+J: “Definitely call it leaderboard.”

VOTING 101

T: “It’s pretty straightforward.”

A: “I will probably see certain facts or FAQs, what you can do as a voter, what you’re actually voting for, what your reps do.”

T: “I think this might be a good opportunity to use a map, sign up through little bubbles.”

A: “I think that was the main problem at the beginning of the project, that people don’t even know where to start.”

K: “I think this could be FAQ or common confusion about voting since Gen Z and millennials don’t want to seem stupid so you’re afraid to ask the right questions.”

N: “I think it’s important”

T: “it’s important but I think if you have to axe anything, this would be it”

ME: Figure out the point system for this section also.

CHARITIES AND MERCH

ME: How do you feel that’s it at the bottom of the scroll?

A: “I think it’s fine you don’t need to see it all the time. I think a notification would be good. You’ve reached enough points to donate to charity or get some merch.”

T: “So when you buy merch, does the money go to the charities or to Polly?”

ME: (Good point. Need to figure out how the merch works, does the user have to make the decision between merch or donations or can they get merch anyway? Do we partner with companies to produce quality merch at cost?)

M+J: “I like the idea of merchandise but how would you pick it up?”

M+J: “Hat, a drawstring bag, fix the hoodie type. If you cash in your pts to buy something it would make sense to make the pts a bit higher so shipping is included.”

POINTS

T: “You should never lose your status (by spending your money).”

T: “Maybe two different currency systems.”

T: “I think the game should give you XP points and gold or whatever the real world equivalent of that gaming concept is.” (XP = Experience Points)

ME: Need to start fully considering the gaming point system, which is yet to be fully resolved. Look into gaming. The takeaway is to have experience points and something you can spend.

A: “You definitely don’t want it where you’re the highest user and you go down in the leaderboard because you lost your points contributing to a charity.”

M+J: “Maybe you do your quizzes and sets and then you get a treasure chest and you tap it to open and it gives you enough money to spend.”

T: “money and points have to be separate as a paradigm.”

T: -treasure chest, check from a Super PAC, support from a coalition (what is relevant analogy for politics or does it need to be that literal? Can we do gold coins? Maybe with Polly the bot on the coin!

POLLY PLAYLIST

A: “Is the POLLY Playlist real right now?” YES!

QUIZ INTRO SCREEN — POLLYTICS LEVEL 1

K: Polly Bot — “I like that he’s an eagle.”

ME: Do these lines mean anything to you? ____ ____ ____

N: “They are the levels how level one is under the first one.”

K: “Or the whole thing is level one and there’s three sections under level 1, that’s how I interpret it.”

K: “So I think if you click LET’S GO and you see it in action, it will make more sense”

M+J: “I think the three lines mean three parts of the level. The icon looks like you can click on it.”

QUIZ SCREENS

1st question

K: “I think the picture is good and that it tells you the amount of points you will get is good. That’s if you get it correct, right?”

N: “Whats the upper right corner? “How many points you have so far?”

N: “I like the speaker icon that it will read the question for you”

A: “I like the CHECK button because if you misclick on the answer, it doesn’t automatically finalize it.”

K: “Now it’s obvious what the progress bar is for”

M+J: ”The three bars are the different questions or which part of the quiz you are on?” “I see the progress. That makes sense.”

M+J: “The next one I’m gonna get wrong to see what happens. OK, I see it turned red but it still says +2 which seems weird. Maybe it changes to 0 or +0. Maybe if there is a 4 answer question for +2, you can get a second try to get +1. Like a quiz correction. In tests at school they give you another chance. Maybe a TRY AGAIN button at the bottom.”

M+J: “Can you maybe get a second chance to score half the points?”

M+J: “Actually, I think it should be on the same point system. Its too complicated to change the pt system if you get it wrong.”

2nd question

N: “Its cool that there are two-line answers also.”

K: “I like that the line turns green to say that you answered correctly”

3rd question about bipartisan

A: “I like that the points turn red because I got it wrong, but I think it’s confusing that it still says +2” (change this to say 0 or +0)

FACT CARD

K: “What are those points for?” “How do you get the points?”

K+N: “How do you know someone’s gonna read it or not?”

K: “Maybe if there’s a timing system, like no one’s gonna take 5 seconds to read it.”

T: “Why don’t you just have separate screens (cards)” “Maybe swipe as opposed to scrolling up and down?”

N: “With the swipe you definitely will want to be able to swipe back to a previous card.”

T: “I would prefer three separate swipeable cards and be able to swipe back and forth between them”

K: “And you can save the facts that you like right with the little bookmark?”

(simulate the audio)

They laugh but like it.

M+J: “You can save it with the bookmark.”

ME: Right now it’s scrollable vertically what do you think?

M+J: “I’d rather have it scrolling than three different cards.”

M+J: “I wouldn’t want a timer. I think I’d like to take my time.” “I don’t think the questions should be bookmarked, just the facts.”

CARDS

ME: If there were three would you read them?

K: “Yeah” “Personally I don’t need points to want to read it, but I can see how you should provide incentives for everything. I feel that if you’re interested in using the app you won’t need points. But you’re trying to get people who are less interested in using it so you might need to make it super easy for them”

ME: Do you think for the cards, if the content was a little longer, you would scroll down a bit or is this the attention span (one screen) what you would recommend?

T: “Yeah, definitely, one screen.”

SECOND SET POLLYTICS LEVEL 1

ME: Would you like to see facts first or would you like to jump right into a quiz?

A: “Totally. I would want to know what I know so definitely quiz.”

T: “Test your knowledge before you try to learn more.”

K: “I think the first sentence is a bit weird. This party is known for the color blue.”

OPINION

ME: Every once in a while, POLLY will want to ask your opinion.

N: “Do you get points either way?”

T: “What does the app do with this information?”

T: “that’s a way to monetize the app, the data”

K: “I like this feature.”

A: “I like the opinion piece. I know we had talked about a message board type of thing but I think this is just an easier way to share your opinions.”

M+J: “I like this idea.”

ME: So this might be able to add up to a screen later that summarizes your opinions to give you an inkling of what party you align to more.

2 LEVELS FINISHED

ME: Do you think it would be too long to finish 3 levels?

K: “Not at all. That would be only like 9 questions plus the facts.”

A: “So you’re almost a mayor if you do this twice?” (have to figure out the point system)

A: “Maybe to become mayor you have to go through a few of the issue levels, which forces you to branch out.”

CONCLUSION

ME: For levels do you think it should be infinite or have a max levels?

A: “You should be able to be well-versed in a bunch of topics. So once you do a bunch maybe it should be like Hey, maybe you should check out these other issues out too.”

T: “There should definitely be an end game like if you’re level 25 Immigration, maybe you should move on”

N: “But what if there was an end game, what’s the point in having the app then?”

K: “Right , you want to learn everything you possibly can and not want to think you’re too smart for the app, like if you cap it somewhere.”

A: “I think human nature is you’re gonna want to move on to another thing”

T: “Yeah, so the app can encourage other things as well.” (notifications)

A: “This feels like it’s a combination between Buzzfeed and Brainpop but like politics”

K: “Everyone loves Buzzfeed and everybody loves Brainpop.”

A: “If you combine what kind of bread are you and science facts, I guess that makes politics!”

FACTS

Climate section

N: “So let’s say I dunno the exact percentage of scientists who believe that climate change is real and I choose the wrong answer, do you think that maybe at that point, it might be good to intro the fact? Like if you get it wrong on that screen then the card will pop out?”

K: “I was thinking that too, if you don’t get it wrong then you don’t need to see the card.”

N: “Maybe besides the right answer being highlighted, a card will pop out and explain why it is.”

K: “Or even if you do get it right, like it was a lucky guess, you can see an option of Tell me More”

N: “Don’t force it but give them the option”

K: “That’s like a way to have them read the cards so instead of having them at the end, have them pop up directly after the question so that you read it press the prompt and you move on to the next one.”

T: “I would never force it because the you’ll just get annoyed”

ME: Even if you get it right?

T: “Yeah, well even if you get it right you can get the option of learning more.”

ME: I did think of a rhythm of having quiz, learn fact, quiz, learn fact.. But it seemed disruptive.

K: “But maybe just give the option to read it whether or not you get it right.”

N: “Just don’t force it.”

“I’d rather have a couple fact cards in the same place than quiz fact quiz fact.”

END DISCUSSION

N: “When do you think this will come out?”

K: “I would really like to have this app. I think it’s awesome.”

A: “Do it! Or I’ll take your idea.”

N: “I feel like every kid in school should have this app.”

K: “When you’re learning about it, then you’ll know how you feel about certain topics so then you’ll want to vote. Whereas if you weren’t really exposed to those topics before then you wouldn’t know how you would want to vote so then you don’t have an opinion and you don’t vote.”

A: “And once you know more, you start thinking, OK well then how can I take action on this? Then you can look that up in other sources. You don’t need to provide this on Polly: who are your reps, this is how you act, this is what you do, because they will seek that information for themselves once they are educated on that. For me , once I got more educated on climate change, I started looking into ways where I can help and be more active.”

ME: So do you think I should have some sort of TAKE ACTION in this section?

A: “You can encourage it, but I don’t think you need to do too much education in it.”

A: “Maybe you can have something somewhere where it says something like Dont forget to take action on something you care about.”

N:“But what if someone wants to take action and they don’t know how.”

K: “But the whole point of the app is to have you educated not action plans”

N: “But why not have that in the app?” “It already has everything you need, why would you have to venture out”

Are there any apps that are political that you like?

T: “Buzzfeed I guess really does actual journalism, like the Cohen thing. You mean Buzzfeed, the app that asks me what type of cheese I am?”

B: “There are clubs in my school like debate club and there is also a climate change/environmental awareness club which I feel like would really enjoy the app and that’s how I could potentially find out about it. Also, for the merch, how about stickers and pop sockets?”

B: “The app looked really awesome and I loved all the images and design!”

M+J: “I know there are people who don’t know anything about politics and have no interest in it. Then there are people who will say things about politics but have no context but don’t know anything about the topic. They might say something because their parents talk about it. So I think people should be more educated about politics.”

M+J: “I think POLLY is a good idea. My opinion is I would download it if it was in the app store. I like the charity thing as a way to contribute to causes you care about.”

“You should leave off where you were like an autosave option if you quit.”

“I think an avatar thing could be cool like you put in image from your camera roll. Or even the illustrations you did the gun can be an avatar. Maybe after you finish levels of quizzes you get a badge. LIke earth.”

“You could have a badge screen where you have Mastery of Climate Change. Accomplishments. Mastery of Gun Control topic. Earth giving a thumbs up. A little screen for the badges.”

“A badge icon that you click on it and brings you to a second screen.”

“Maybe read certain amount of articles and you get Scholar Level.”

“Complete levels and get badges, like a separate screen of accomplishments.”