Complete a learning cycle with a watch party

Highlights reel watch party!

What is a learning cycle, and why did we introduce it?

Early on, we established 2 team norms to listen to our users regularly and celebrate wins, but in all honesty, we weren’t practicing these norms enough. To fix this problem, we came up with the concept of a learning cycle, which we defined as a (1) shipping a version of the product to users, (2) learning from users through a research study or data analysis, (3) sharing the learnings with the team, and (4) pausing to celebrate at the end of the learning cycle. This ensured that we learned from our shipping (rather than just shipping) and celebrated at the right moment (when we shipped and learned something).

For our v1 product launch, we used a 3-week diary study to learn about user behavior

We were preparing to launch a v1 version of our product to a small cohort of 10 users and we needed a structured way to learn from it. We consulted our user research experts and they suggested a diary study, which is a method used to understand user behavior over a period of time. They suggested a diary study because we wanted to understand very deeply how early adopters used our product. Specifically, we wanted to know what our super users and churned users were doing in the product.

Our diary study was designed to last 3 weeks, with a 1 hour onboarding call to learn about what users expected our product to do, a midpoint checkin so we could understand the behaviors of our detractors and super users, and a closing call to gauge their overall experience. During the study, users were asked to fill out 3 surveys a week (a “diary”) and answer questions about how they used the product. We also sent out a final survey at the end to rate us on a scale of 1–5 for various criteria (satisfaction, ease of use, trust, & quality). We planned to use this as a baseline so we knew if we were improving over time. This study was intensive for both the participant and for the researcher, but we thought it was necessary to understand our early users deeply.

We learned that key team members weren’t participating in the user research even when given the opportunity

These diary studies are pots of gold for understanding users deeply, and we encouraged everyone, including designers, engineers, VPs, and marketing, to sit in on these calls. However, we ran into a few issues at this point:

1. Everyone said they wanted to sit in on interviews, but when given the opportunity to sit in, they didn’t sign up or volunteer. And, when we made the recordings available to watch, no one watched it.
2. When someone did sit in on an interview, they only observed behaviors for one participant. This created a bias that one participant represented all participants.

Watch parties for the win

To solve this problem, we brainstormed ideas with user research on how to give the team better access to users. We came up with the idea of an interactive watch party. User research would grab highlight reels from our 20 hours of footage and distill it down into small clips that the team could consume in a 3 hour workshop. We would all collaborate in a Miro board and respond to the highlight reels in real time so that everyone remained engaged. We also made it fun by giving everyone credits to expense lunch, candy, or popcorn.

The watch party was wildly successful, and I knew that it worked when one of our engineers said “Wow, I had no idea how strongly this user felt about our lack of support for nice visuals. I’ve seen the feedback in our readouts, but seeing their faces really showed me how much it affected them.” Our engineers ended up advocating for smaller roadmap items that I was struggling to explain the importance of. I find that the best way to build empathy and convince others is to get the whole team observing users in real life.

Watch party improvements for next time

The biggest piece of feedback we received was that there wasn’t enough time for discussion. They left with a better understanding of our users, but they didn’t feel like they knew what’s next. Even though we were able to distill content from 20 hours down to 3 hours, our discussion totaled only 45 minutes. Next time, we’re going to condense it to 1 hour of clips and 1 hour of discussion.

Closing out your learning cycles with watch parties

The watch party ended up being a great way to mark the end of our learning cycle in addition to a research deck readout. We celebrated our win at the end of the watch party rather than after we shipped our first version and that helped us build a culture around learning.

Ingredients

1. Research repository for uploading research recordings — We used EnjoyHQ.
2. Video editing tool for creating highlight reels — We used UserTesting to clip videos and cut to the best quotes.
3. Remote whiteboarding tool to make watch party interactive — We used a Miro board.
4. Conferencing tool — We used Zoom to host the watch party with one person sharing their screen and sound to play the videos.
5. Survey tool to collect feedback — We used Google Forms.

Recipe

Prep for the meeting.

1. Product or user research should create a calendar invite well ahead of time so that the whole team can make it. We recommend blocking off 2 hours on the calendar. Invite the whole team involved in making product decisions as required.
2. User research create generic profiles for each person. We created visual cards like this for each participant.
3. User research should analyze the raw videos and content and select highlight reels for key insights. Create themes for the team to grasp and then tailor videos to those themes. For example, we created a theme for Onboarding and Usability for each of our apps (browser plugin, web).
4. Create a miro board for everyone to participate in. We had a section for Participants, each Usability section, Opportunities, and Final Survey results.
5. Create an agenda for the watch party. Here’s what we recommend:

• Meet the participants (5 minutes)
• Usability and Behavior videos (40 minutes)
• Break (10 minutes)
• Discussion of the sections (15 minutes)
• Opportunities (15 minutes)
• Discussion of opportunities (15 minutes)
• Final survey results (10 minutes)
• Wrap-up/Next steps (10 minutes)

Watch Party Meeting

1. Follow the agenda and have one person present their screen with sound.
2. Tell the team to use post its to add reactions into the correct section. User research could group together post its with similar comments after each section during the meeting while the videos play.
3. Send a survey after the session to collect feedback.

Thanks for reading our recipe. Our goal is to share our experiences working together using a short-read format with instructions that you can try on your own — just like a cookbook. We hope you found it helpful!

Katie Le & Nicolas Backal