How We Run Continuous Discovery at FreshBoks

Read on if you’re interested in waking up every Monday morning with a handful of interview sessions automagically booked in your calendar, with only one hour of setup work per month.

This is a tribute to Teresa Torres and her fantastic book, Continuous Discovery Habit. This article is all about how to make continuous interviews happen. If you haven’t read the book itself, it is totally worth it.

Before I begin, I’d like to share a research project I’ll use as an example in this article. This will help you better understand the reasoning behind each step.

The project that the team is currently working on is called the FreshBooks AppStore. It is a standard third-party app marketplace that allows FreshBooks users to find and install apps that can extend the functionality of FreshBooks. As part of continuous discovery, the team is interested in learning how we can assist FreshBooks users in discovering the right apps, especially those who have never used an app before.

Part 1: Find the right target users, continuously

Before conducting any research, it is essential to identify your target user. Whether you plan on conducting continuous interviews or not, narrowing down your target audience will ensure you receive high-quality feedback and insights. By defining a specific scope for your research, you can ensure that the candidates you select are relevant and can provide valuable information.

There are many tools available to filter the list of users in FreshBooks. I use Fullstory to apply multiple filters. For instance, my current research goal is to determine how users discover the right apps. Therefore, I have asked Fullstory to filter all the users who visited the FreshBooks AppStore and successfully clicked the install button.

Use Fullstory to identify the target audience.

Once you have a shortlist of users, depending on the tools and setup, you may need to gather more segment-related information about them. In FreshBooks, we import the user list into Looker to fetch segment information such as industry and app usage statistics. This helps me better target the right users I am interested in. For example, one of my current research assumptions is that users who have never used an app may spend more time picking up the right one. This means I need to talk to non-app users. So, I fetched extra app usage statistics from Looker and filtered out all the users who have already used apps.

User Looker to get additional segment or behaviour data

After filtering by segment data, we’ll next clean the user data to have a list for recruitment. At FreshBooks, we use Google Sheets for this task. We remove duplicate entries and clean up records with empty names or email addresses. This is the last opportunity to refine the user list based on the research goal. For instance, in this project, we are more interested in learning from paying customers. Therefore, we filter out all the trial users and download the user list with a maximum of 150 users.

Use Google Sheets to clean things up.

We aim to have 150 Users on our list because our invite-to-booking rate is around 5-6%. This means that if you invite 150 users at the beginning of the month, you can expect about 7–9 users to sign up. This translates to 2–3 users signing up each week. However, it’s important to note that the sign-up ratio may vary depending on the industry and customer base. Therefore, it’s recommended to experiment and find out the correct number of users for your specific industry or customer base.

You should have a final user list to send out the user research interview invitations. Before we do that, here is a summary of the steps to prepare the user list:

1. Apply the right filter according to your research goal
2. Refine the filter to make a list of approximately 500 users (for this month)
3. Export and download the user list
4. [Optional] Collect more user data from Looker to focus on the right segments and behaviours.
5. Upload the user list to Google Sheets.
6. Remove duplications and further refine the list according to your research goal.
7. Export and download with max 150 users records

Protips (it’s all about save your work for next time)

1. Save as a segment filter in Fullstory for future reference
2. Save the looker query for future reference
3. Keep a record of the attributes of the user on your list for future reference (e.g. 0 app+paid)

Don’t do

1. Get a massive user list with few filters
2. Add way too many segment data that you don’t need. It will slow you down in decision-making.
3. Don’t invite more users than you need. It’s a waste of user attention.

PS: You don’t need to have a specific customer segment in mind to start your search. You can start with a broad group and still gain exciting insights.

Part 2: Setup the recruiting channel, continuously

Before using recruiting tools to send out invitations, it’s important to block a weekly time slot in advance for your team. Discuss with your team to find the right time slot and create a placeholder event to block it. In the era of remote work, there are tons of meetings every day, so it’s better to save the time slot in advance to avoid any trouble. Once a user signs up for the time slot, you can easily update the event to confirm and notify the team.

Block the timeslot and invite your team.

Now that we have everything prepared, we can use recruiting tools to send out invitations. At FreshBooks, we use userinterviews.com as our user research recruiting tool. It allows us to upload a CSV user list, which we prepared in Part 1, and send out invitations with branded email templates we set up in advance. It also handles scheduling. We input our available time slots, and users choose the best time that works for them. Once a user successfully schedules a session, userinterviews.com will automatically add the session to everyone’s calendar. There are many alternatives for recruiting, such as userzone, usertesting.com, Maze, and more. The point is to use a tool to save time for setup.

The recruiting tool is a big time saver.

Now that you have sent out the invitation to users, the next step is to join those user interview sessions and learn insights. Before we discuss the interview, here is a summary of the steps for the second part, where we prepare for the interview:

1. Block weekly time slots for user interviews and invite anyone interested
2. Hold the time in everyone’s calendars as a Placeholder to begin with
3. Once you have the user signed up, please update it to Confirmed and update the Zoom link and who participates.
4. Use recruiting tools to automate the recruiting and scheduling.

Protips

1. Add buff time between two sessions. 15 mins is the right amount of time for our team.
2. Use the same Zoom link to avoid confusion. (hot tip: turn on the waiting room to avoid having your following interview join your current interview)
3. If you use userinterviews.com, set requested participants to more than you need to keep the project open and going.
4. It is expected that 2-3 people will not show up. To avoid this, you can invite attendees for each session closer to the actual session time instead of inviting everyone at once. This way, there will be less chance of no-shows.

Don’t do

1. Update your team at the last minute. It’s best to keep your team updated for no-shows or reschedulings.
2. Forget to pay the participant after the interview.

Part 3: Synthesis and record interview snapshots continuously

You can schedule two weekly user interviews by following the steps in parts 1 and 2. This section focuses on how to document, synthesize, and share insights from the interviews.

During user interviews, it is recommended to have two interviewers - one to lead the interview and the other to take notes. At FreshBooks, we use Airtable to keep track of our interview questions and notes. A pro tip is to have a product manager and even an engineer join the session so they can exercise their empathy muscle.

Airtable for note-taking

When scheduling interviews, it’s important to allocate time for reviewing and synthesizing the findings. You can also use this time to refine your interview questions if necessary. It’s crucial not to delay extracting insights from the interviews. Procrastination can lead to unnecessary stress and wasted time. So, make sure to prioritize this task, and you’ll thank yourself later!

Review notes on the same day.

As a team, agree to a snapshot layout and create a deck (one spot to hold them all!). We based ours off of the example in Continuous Discovery Habits, Teresa Torres (pg 68),

The interview snapshot includes owner information, company information, segmentation, a key takeaways section, important quotes, links to notes and recordings, and tags so you can easily search later. Looking for a public, professional photo of your interviewee?.

Make the interview summary scannable or include key takeaways for easy comprehension by others who were not involved.

Once done, remember to share your snapshot in your team slack channel!

One of the snapshots we created

Looking back, I thought it was impossible to interview users every week throughout the project lifecycle. However, once the team was able to validate their assumptions on a weekly basis, we could never go back to the old waterfall design process that only talked to users at the “Discovery Stage.” The old process allowed so many wrong assumptions to be baked into the project without us even realizing we were wrong.

Glad to see you’ve made it this far. I hope you enjoy reading and successfully implementing it in your organization.

PS: Thank you, Tiff Nogueira. She is my PM partner at FreshBooks, and she co-wrote this article.