“Start, Stop, Continue” Debrief
Easily drive evidence-based actions with your team
Written by George Zhang and Yuri Choi
For UX researchers, it’s often difficult to find an efficient way to debrief our team of stakeholders after a large (or small) volume of user interviews, lab evaluations, or field studies.
We observed three main challenges trying to ensure the impact of our research findings: overload, alignment, and buy-in. First, everyone who participated in the research, from researchers to stakeholders, are prone to experiencing an overwhelming amount of data consisting of notes, videos, transcripts, pictures, and other artifacts. Second, sometimes it may be difficult to align on and commit to high-value action items in a timely manner. And third, there is always a possibility that stakeholders can reject research findings if they feel they were not involved enough in the research process or they feel that their ideas are underrepresented.
To address these challenges, we developed a simple and fast solution: the Start, Stop, Continue debrief exercise. We’ve piloted, iterated, and refined this exercise with the help of our UX researchers at Uber in the past four years. This exercise ensures stakeholder involvement and buy-in by opening a forum for them to share their own observations and ideas. We’ve aimed to make this debrief inclusive and transparent, rather than exclusive and opaque.
Now we’re pleased to share the SSC debrief broadly. We welcome your comments and feedback.
What is SSC?
The syntax “Start, Stop, or Continue doing X” has been broadly used in many team activities such as postmortems, brainstorms, and so on, where “X” stands for a specific action item. However, we have enhanced the SSC syntax by appending the observed evidence to the framework. In this way, we always provide the rationale behind the proposed action.
For example, “Start [X = remembering the username and password], because [Y = 4 of 5 participants forgot their usernames or entered wrong passwords].”
This syntax is very handy to use because it not only helps the team boil down their vague ideas into concrete actions, but also builds their confidence by grounding their ideas in evidence. This debrief exercise also makes it easy for the team to rank and vote for the actionable ideas.
When to use the SSC exercise?
Anytime you run a study debrief with your team. We’ve had success with this exercise after usability evaluations as well as foundational interviews/field studies. We believe it, as a general debrief exercise, applies to other research methods such as cognitive walkthrough, task analysis, but haven’t verified it.
The Prerequisites
The success of SSC debrief depends on the following conditions:
- Members of the debrief team must observe several research sessions so they are not so biased when sharing insights.
- They must be trained to observe user studies objectively. For example, they must be able to differentiate their own ideas from their observations.
- They must take notes of their observations and ideas while observing participants. They should take notes independently and try to avoid inter-observer inferences until the debrief.
Observer training before interview sessions
As the UX researcher and moderator, you need to explain what the SSC debrief is and teach your team how to observe and take notes effectively and with intent to complete this exercise.
First, introduce your observers to the Start, Stop, Continue debrief exercise.
- Tell them why it’s important to make objective observations and not to mistake their own ideas for objective evidences, and explain to them the structure of SSC syntax.
- Show them that they will be filling out post-its like the ones above so that they know what kinds of things they should be paying attention to.
- Tell them that they are going to be generating a lot of these post-its — at least 1 post-it for each observation they make.
- We can dive into the details at the time of the debrief.
Now that they understand the goal of the exercise, teach them how to observe and take notes so they can easily refer back to their notes and effectively contribute to the exercise.
- An effective observer intensively watches, listens to, and records the participants’ attitudes, behaviors, reactions (visual and audible), words, and messages. They try to identify recurring themes over time and across users, but not to immediately jump to solutions.
- To avoid jumping to solutions, emphasize that observers should take notes of what exactly happens. It is better to record what the participant said as close to verbatim as possible or at least summarize what the participant said, not what observers think the participant meant or would want. They should be able to read their notes and be able to confidently say, “This is what the participant said,” or “This is how the participant behaved,” or “This is how the participant felt.”
- Participants may sometimes provide solutions. For example, a participant proposes, “You should send me emails updating me about my application status every week.” If this is the case, it can still be valuable to write it down. However, observers must understand WHAT problem it solves and WHY the participant thought their proposed solution would solve their problem because participants could be making assumptions. The problem might be they don’t know the status of their application frequently enough (ie. weekly). They check their email regularly so they think emails would be the best way to get updates. They think weekly updates are sufficient because they expect the application to take more than a few days. In this case, we can identify two assumptions that participants has:
Email is best — but what about push notifications or text messages from the app itself that they use everyday?
Weekly is good — but what if the application takes less than a week? Shouldn’t we set the expectation that it can take less than a week? Or what if the application takes significantly longer than a week? Should they get weekly emails for 20 weeks straight? Or could they visit a website to check the status as they need?
- Encourage observers to identify recurring themes within and across participants. Tell them to record the repetitive findings, not to ignore them. Having it written down on paper makes it more compelling.
Tip: Stakeholders can be biased if they have not observed enough sessions. Always have them record the participant ID’s for each post-it so you can assess the frequency of the observation.
How to run the Start, Stop, Continue Debrief?
After training your observers, you can now I. conduct your SSC Debrief and II. track your impact over time.
I. Conducting the SSC debrief
1. Preparation
What you need to prepare
- Large 8x6” post-it pads for each observer participating in the debrief. The standard square post-its are too small and hard to read when on the wall.
- Sharpies. It is difficult to read anything on the wall when written in regular pen because it’s too thin.
- A timer.
- Large wall to post and cluster everyone’s post-its
- Start, Stop, Continue Overview Deck
- Book a meeting room for ~60 minutes
- Invite anyone who observed several sessions
Set up
- Designate three areas on the wall for “Start,” “Stop,” and “Continue” items (see figure).
- Distribute post-it pads and sharpies.
- Explain how the debrief works (you can show this deck to your group).
There will be three main sections to this exercise, addressing things that we should “start” doing, “stop” doing, and “continue” doing, respectively. In each section there will be 3 main steps: 1) 5-minute brainstorm, 2) Roundtable Read-Out, and 3) Recap + Vote.
2. Section of START
Step 1: 5-minute brainstorm
- Everyone in the room is required to jot down ONE idea per post-it, always following the format of “Start [Action Item], because [Observed Evidence].”
- The goal is to generate as many post-its as possible. Don’t overthink things. Write out all possibilities even if you think they’re not good. Don’t waste time cleaning/editing your ideas.
- Encourage the room to refer to their notes if they can’t remember.
Tips
- Shorten/extend the time limit to suit your needs.
- Some people may forgo the [Observed Evidence]. You can do some “quality control” to ensure people are adhering to the framework. Encourage the group to provide the [Observed Evidence] either immediately after the 5 minutes are up or as people are sticking post-its to the wall.
Step 2: Roundtable Read-Out
- Choose one person to start the read-out.
- Have them start reading one post-it out loud at a time. If they have forgotten to write down the “Observed Evidence,” have them write it down.
- After each post-it is read out, have them stick the post-it under the “Start” section of the board.
- After the first person has finished sharing and posting all their post-its, move onto the second person and have them read and post their ideas.
- Cluster similar ideas with the post-its already on the wall.
- Continue until everyone has posted all their post-its on the wall.
Tip: Some people may have forgotten to write down an idea that someone else has shared. Add “+[INITIALS]” to the corner of the existing post-it.
Checkpoint: At this point you should have clusters of ideas under the “Start” section of the board (see image above).
Step 3: Recap + Vote
- Go over each of the clusters with the room.
- Ask anyone who wants to “+1” an idea if they agree with an idea they didn’t have a post-it for. Add their initials to the bottom corner of the post-it.
3. Section of STOP
- Repeat steps 1–3.
4. Section of CONTINUE
- Repeat steps 1–3.
II. Tracking the SSC debrief impact
- Take a group selfie 😊.
- Take pictures of the completed board.
- Collect all post-its. Make sure to retain the categories so it’s easier to type up.
- Make a copy of this spreadsheet template.
- Type each post-it up into the spreadsheet. You can ask your teammates for help!
- Share the spreadsheet with your stakeholders and debrief teammates. Have them contribute to the spreadsheet (e.g. column “Changes incorporated into next iteration?”)
- Revisit this spreadsheet to track progress of changes implemented as a result of your research and debrief (e.g. after 2 weeks, 1 month, etc.).
Variations of SSC debrief
As a general debrief exercise, the SSC can vary according to your specific research situations. We will just name several variations below, and welcome to come up with your own variations.
#1 SSC debrief per interview session
- Conduct the debrief exercise after EACH interview session then collate, categorize, and tabulate the post-its after all interviews have been finished.
#2 Asynchronous SSC debrief
- Record all your sessions.
- Assign one video per stakeholder.
- Conduct the debrief as described in the article. Emphasize that they must record the participant ID on each post-it.
- Alternatively send observers a survey (detailed in “Variation 4: Virtual Form”)
#3 Virtual SSC Debrief
- Example tools: Mural, Figma, Miro, Google Jamboard, etc. (some are free or have free trials)
- Mural example below:
#4 Questionnaire note-taking
- Have observers submit a form (google form example | questions in google doc) after each session, during the group debrief, or asynchronously.
- Tip: enable the form to collect the email address of the submitter.
#5 SSC brief after your research share-out
- Not ideal, but insights and facts are fresh in your stakeholders’ minds. This is an opportunity to have cross-functional stakeholders engage and collaborate with each other.
One more thing!
If there is anything you think we should “start, stop, or continue” doing (😉) or if you’ve tried this exercise and have feedback or recommendations, please leave your thoughts here! If we get enough responses, we’d be happy to publish a follow up article.
Thanks for reading!
George and Yuri.
