A self-guided journey: the unmoderated testing of the Planet 4 prototypes
How Greenpeace tested new engagement features for the future of our websites (Part 2 of 4)
👈 Part 1 | Part 2 | Part 3 👉 | Part 4 👉
Planet 4 has a vision — to offer engagement opportunities to our supporters that are meaningful, inspiring and create real change. We also want to reach out to new audiences and get closer to the next wave of activists that can drive the change the world needs. To achieve this, we realized that we needed to rethink the way Greenpeace works on the internet, bringing Planet 4 up to the next level as an engagement platform.
As announced in Chapter #1, the Greenpeace + Redhat sprint brought in two clickable prototypes. We believed that this new set of features could help us to get closer to what we first envisioned, bringing more people to action. Since we should not build anything from assumptions, we decided to test these concepts with different methodologies — a qualitative user research conducted by an external testing agency (UX Studio) and quantitative research based on unmoderated online tests.
Choosing the right methodology and tool
Planet 4 is used in more than 40 different countries, so we needed to make sure we were testing the prototypes with a diverse audience while giving our supporters the chance to test and give feedback on the new concept. As the September climate strikes were getting closer, the opportunity to get the prototype into the hands of young audiences was at our grasp.
In order to reach these multiple groups, we decided to create an unmoderated online test, where the participants could guide themselves through each prototype journey.
So we started looking for an online tool that allowed us to create tasks and ask follow-up questions to a large number of participants. So after some research, we found a tool called Maze. Not only did it cover all our requirements for the unmoderated testing, but it also allowed us to track interactions, create heatmaps and collect answers to specific questions, giving a better view of the participants’ experience with the prototype.
Designing the unmoderated test
Following the same testing criteria document of the moderated test (next episode!), we designed two journeys in which the users were given specific tasks to complete on a Greenpeace website. In-between the different scenarios, the questions were helping us to understand how the users felt and behaved about a specific action.
After reviewing the journey a few times, we first went out during the September climate strikes and invited activists for testing. In the next couple of months, we also recruited website visitors, hired testers from Maze and reached out to the amazing internal Greenpeace community to support in distributing the unmoderated tests across their channels.
Key takeaways
Even though the two prototypes reached more than 400 participants, only less than 200 actually completed the full journey. Most of them were from Gen Y & Z, but we had a mixed sample of participants in terms of engagement levels –some of whom never visited a Greenpeace website before.
“The prototypes’ goal was to understand if these features inspire our audience to take action, re-engage and bring others along. We wanted to get a clear direction of travel on how Greenpeace should take Planet 4 forwards as an engagement platform, based on the needs and interests of our target audience.”
From the questions’ outcomes, we noticed that users were most interested in features that allowed the opportunity to customise their experience and to take actions that can go beyond the online engagement. The ability to choose actions based on the time estimation was highly appreciated by the participants and also the possibility to keep a track of their achievements.
In both prototype tests, we were also able to confirm the assumptions that users would be more willing to take actions when they are able to visualize their impact. On the opposite, we also noticed that the users were not so interested in sharing their actions on their social media, neither gave too much importance to connecting their Greenpeace account with their social profiles.
The heatmaps features also helped us to evaluate the usability of the prototype and identify design elements that could be improved. The tool was also providing data on users’ first and second clicks, which allowed us to review the tasks we were asking them to perform.
We also added an open question at the last step of the journey, allowing the participants to leave qualitative feedback on their experience. Some of these comments were very powerful, especially when mentioning aspects of the concept that we were trying to investigate with the prototypes:
- “Interesting set of concepts! As a digital professional myself it’s sparked a lot of ideas, and harnessing gamification could be a cool way to inspire action.”
- “The site is good, it seems mobile friendly. I do like the idea of keeping track of your progress at volunteer work. Helps people stay committed.”
- “This prototype experience was my first w/ Greenpeace and was a bit slow/ lagging on my iPhone 7 on Firefox and no other apps running. Otherwise, it was a typical mobile experience as expected.
- “It was super clean and easy to navigate. Just the right amount of information and statistics to keep me interested in doing more.”
- “It went well until I had to save my progress and wasn’t able to find a way to do that”
Lessons learned
After a couple of months, several refinement sessions and two rounds of tests, we can say we learned a lot with this process! Here are some tips for you to plan your next unmoderated test:
- Create your prototype for an unmoderated experience — after the first round, we noticed that users were getting lost on the prototype screens or trying to interact with elements that were not clickable. So we had to spend some time later in adjusting the design and making sure the users were able to interact with all the elements and features we needed to evaluate.
- Understand the test limitations — the unmoderated experience can work well when searching for quick insights on the design and usability of a product, but it has some limitations when evaluating a whole new concept.
- Keep it simple! Remember that your participants might be testing your prototype while drinking a coffee or waiting for the next train. To decrease your chances of losing them for the notification that will pop up, make sure you are defining short, clear and actionable tasks. You can also check more tips for writing usability tasks here.
- Ask the right questions — since you can’t ask follow-up questions later, make sure that you are crafting the right questions to feedback on your research objective. Always useful to check some guidelines, such as avoiding questions that could lead the user to a specific answer.
- Set a scenario and create a consistent journey — Define the expected path you want your users to take, and design your missions based on this journey. It can also be helpful to give the users some context and create a figurative scenario to your tasks.
- Run a pilot — call up your friends and family to pilot your test :) They will certainly spot a few issues and help you iron out the final experience. Search for insights on the design, tasks’ instructions and questions!
- Work on a distribution strategy — while some of the testing platforms allow you to hire external testers, you might also need to find participants on your own. Define the number of completed tests you need, the audience you want to target and work on strategies to recruit at least 3xs more participants :)
This entire journey — from designing to analysing the results of the unmoderated test — taught us a lot about how user research is necessary at each stage of the product iteration. As we move forward with the product continuous improvements and the development of new engagement features, we will certainly use these testing methodologies more often.
⭐ Interested in knowing more or joining this discussion? Leave your thoughts or comments below, tweet at #GPP4 or reach out to the P4 team via email :)
