7 Powerful Ways To Get Feedback From Users (In Scrum)
In a previous post I talked about the ‘case of the missing users & customers in Scrum’. While there’s a lot of focus (also thanks to DevOps) on ‘shipping fast’, we often forget that ‘shipping fast’ and ‘building what the user needs’ are two sides of the same coin. It’s wonderful to ship fast. But without actively involving the user, it is nothing more than a technical exercise. How can we build good products if we never talk or see real-life users? In this post I offer 7 powerful ways to make this kind of interaction happen (directly and indirectly).
What feedback?
But before digging into practical approaches, lets take a minute to talk a bit about ‘feedback’. What kind of feedback are we looking for, in particular?
- Does a feature actually solve the problem that a user is trying to solve? Every feature that you build, every page/screen that is part of the product should serve some purpose for the user. Why build it otherwise? Perhaps the feature automates an otherwise manual process. Or it makes it more convenient than alternatives. Either way, we need to ascertain if our intention to solve the problem is achieved. Users will not use a product that feels tedious, difficult or useless to them;
- Do people understand how to use a particular screen or feature? Does it feel intuitive to them? Or is a user often ‘blocked’ when using it;
- Do people find using a product pleasant and convenient? Why or why not?;
- What new ideas emerge when people use a particular feature? These ideas might improve the feature itself, or they might lead into new features that make the product even more valuable;
1. Involve users during the Sprint Review
Invite users of your product to attend Sprint Reviews. This presents the team with an excellent opportunity to validate new features with real users. How do users feel about the new features? Do they understand how to use them?
A great way to test this is by handing a user the keyboard and the mouse and to let them use a new feature. Don’t explain what they should do (“click on this”, “go to that page”, etc). Instead, provide them with a goal you’d like them to achieve (e.g. “try to create an export of only the flex workers from this week”). When a user gets stuck or runs into something they don’t understand, don’t help them right away. Instead, ask how they expected the feature to work.
It is important to create a safe environment for users to provide feedback. You’ll notice that people are quick to apologize for not being able to find their way around a feature. They may even apologize for errors in the site (“Sorry, I didn’t mean to break it”). Although it’s really a problem of the product, users often feel ‘dumb’ or ‘slow’ if they can’t figure it out. Especially when a whole team is watching. So stick a light-hearted, informal approach and thank users for their time.
When you’re having difficulties to get users to attend your Sprint Review, try to understand why. Maybe the aforementioned safe environment is lacking. Maybe it takes took long. Or maybe the goal isn’t clear (and neither is the pay-off). Offering a gift voucher or a nice lunch may also help.
2. Go on a User Safari
Organize a ‘User Safari’ to visit groups of users in their work environment. This is fun and great way for a Scrum Team to see how people use a product. A lot of developers often have no idea in what kind of environments their products are used and by what kind of people. A nice bonus of a User Safari is that it’s often rewarding and motivating to developers.
Organizing a User Safari isn’t difficult. Find a group of users or a particular facility and ask if it’s okay to drop by with a group. Maybe it’s even possible to get a tour! After that, it’s best to split up and join different users as they go about their day-to-day-work. Sit next to them and observe how they use the software. Ask open-ended questions like “How does this feature help you in your day-to-day work?” or “What can we do to make it easier for your to use the product?”. Depending on how much time you have, you can switch around a bit. At the end of the User Safari, do a shared Retrospective with your team. What stood out for the team? What new ideas or improvements emerged?
A few years ago (2014 or so), I worked for a company that delivered products for workforce management. Wanting to know more about our users, we (4 developers) visited one of our customers. We quickly noticed how noisy and chaotic their work environment was. Telephones rang continuously, people inquired loudly about the availability of certain flex workers and people walked in with questions. We also noticed something else. While on the telephone — the telephone tightly jammed between a planner’s head and shoulder — the planner used our product to change the planning for a particular flex worker. Keeping the phone between the head and shoulder meant that the planner’s head was tilted. This made reading the texts and buttons on the screen difficult. This wasn’t helped by their small monitors. Back at our office, we increased the font size for buttons and other texts and improved overall readability on small screens. It was a small change for us, but it greatly improved the usability for this customer.
3. Organize a Product Bazaar
A Product Bazaar is a wonderful and energizing way to rapidly get a lot of feedback from a large group of users and other stakeholders. It involves setting up a number of market stands where aspects of a product can be reviewed with visitors by members of the team. The Scrum Teams break up into smaller groups (or even pairs or individuals) and join one of the stalls. One stall may focus on a particular feature, while another is about the Product Roadmap. And yet another can be about the look and feel of the product. The possibilities are endless.
Product Bazaars work best when you divide the visitors into smaller groups and use rounds (of 10 to 20 minutes) to cycle the groups along the stands. The easiest way is to ask visitors to self-distribute across the stands in equally-sized groups, and then start cycling with a bell or some other signal. How much space you need depends entirely on the number of stands and visitors, but make sure that it isn’t cramped. Use a conference hall, break room or perhaps a number of office spaces that you can use as impromptu ‘stands’.
Add decoration and props to really make your Product Bazaar stand out. It will also make it easier to draw people to your event. Find actual stands, add a coffee bar or an inflatable air castle. Or maybe there’s someone in your team who’s willing to play some live music. Make sure to add clear signs to the various stands to tell visitors what they focus of the stand is. A nice way to gather feedback is with ‘idea boxes’ or walls with post-its next to the stands. This makes things really visual and helps the various groups to build on each others work. If you’re having trouble getting people excited to drop by, offer them something in return. Like a gift card or really good coffee.
I’ve organized quite a few Bazaars over the years. But the coolest one was for a gamification startup called YourStory. I had been involved with this startup for a while, and we were eager to test our new ideas with users. So we posted an invitation on Facebook and approached enthusiastic users. A local community center was willing to host our Bazaar for free as long as we brought food and drinks. Because our team consisted of five people, we set up five stands on opposite sides of the room and took some time to decorate it with flyers, posters, post-its and merchandise. We got around 40 visitors that evening that cycled the stands in groups of 8. The CEO talked about his general vision of the product, I asked people to offer ideas and suggestions for our road map, the designer asked feedback on his new designs, the developer showed new features and an enthusiastic user talked about his experiences with the platform. Not only did we get a lot of invaluable feedback, we also energized the people present to participate. A number of people to decided to chip in on the funding afterwards, which was a nice bonus.
4. Do Guerrilla Testing
Another way to “go to the users” is with a technique called “Guerrilla Testing”. Originally employed for User Experience (UX) testing, the idea is simple. Go to a place where you’ll find potential users. This can be the lunchroom or a meeting point in your building. If you have a lot of external customers, visit sites where the product is used. Or simply go to a coffee shop or the park. With a laptop in hand, ask people if they can spare a few minutes to help you improve a product.
Again, the best kind of feedback comes from goal-based behavior. So ask a user to perform a particular action or achieve a particular goal. Write down any observations or feedback. You can even film the session if the user doesn’t mind. Rinse and repeat to gather feedback from different users. It’s also a great way to get a sense if who your users are and what they’re looking for.
For the startup KerkWijzer, we seized the opportunity to place an actual stand at a conference related to our product. This was an excellent way to do some guerrilla testing on a new workflow. We got ourselves two monitors, a keyboard and a mouse and set up the stand. We decorated it with banners, a large map and dressed up in lab coats and clipboards (as ‘researchers’). Every passer-by was asked if they’d like to give us feedback. Thankfully many did and sat down with us to click through the workflow. We recorded their feedback, asked what they liked and didn’t like and identified what parts of the workflow they struggled with. This extended testing session not only resulted in invaluable feedback, it also yielded a lot of interest in the product (dozens of new users signed up afterwards).
5. Use (short) surveys
Use short surveys — either within the product or through other means — to ask what users think of a new feature or the product in general. You can show the survey at the end of a particular workflow associated with the feature. Or only for a group of users. The survey can consist of a rating (with stars or points) and a field where people can specify what could be added or improved to increase the score. Or you can ask for a Net Promoter Score to see how likely it is that a user will recommend this product to friends or family. Again, you can add an open-ended question to ask what would increase the likelihood of a recommendation.
Take a look at one of the many (free) platforms for creating surveys, like Google Forms, SurveyMonkey or SurveyGizmo.
6. Investigate usage Metrics
Although direct feedback from users is obviously the most valuable, usage metrics offer another approach to see how people experience a product. There are many usage metrics available. I’ve listed the ones below that I like the most:
- Number of unique visitors: The number of people that use your product or application. This can be broken down to time-intervals, like month, week or day. It is a simple and fairly rough metric, but there are many tools available to measure this easily (like Google Analytics);
- Bounce Rate: The number of people that visit or use your product or application, but click away or stop using it almost immediately. Apparently it did not meet their expectations. A high bounce rate for the homepage of your web application or website is no need for worry, but a high bounce rate for internal pages does indicate room for improvement;
- Number of clicks: The number of clicks a user has to perform to complete a particular action or workflow tells you something about complexity. When we are developing software, we tend to (naturally) click through the workflow in the most optimal way. But real users may experience things differently. They may require more clicks to complete the workflow, perhaps trying to click somewhere where that isn’t clickable, or navigating back and forth between steps in a workflow;
- Time: As a corollary to the number of clicks, it is also useful to measure the time people spend in a particular workflow or process. This (again) tells you something about the complexity of the workflow;
- Conversion: If your product has certain marketing goals (and they probably will), like getting people to sign-up for a service, or purchasing something, it is useful to measure the conversion rate. The conversion rate represents the percentage of people that complete a particular workflow (like a sign-up or purchase-process) compared to the total number of people that started a workflow. Conversion rates not only help us to identify broken workflows (a sudden drop in conversion), they represent the ultimate metric to determine if your product is generating leads, purchases or other valuable business goals;
There are many tools available that allow you to measure these kinds of usage metrics, like Google Analytics, NewRelic or Piwik. Either way, make sure to augment usage metrics with direct feedback from users. Numbers never tell the whole story.
7. Use A/B-testing
A/B-testing is a technique where users of your product are presented with slightly different versions of the same product. One version may have a new feature, while the other doesn’t. Or there are different versions of a new feature. In any case, there is usually a ‘control’ of the product that hasn’t changed and there are one or more ‘variations’. In all versions, we collect usage metrics (see above) to determine how people use the feature. The version that works best, based on what users tell us and what the metrics tell us, is picked for the main product. A/B-testing is common practice in tech companies like Google, Netflix, LinkedIn or Etsy.
The advantage of A/B-testing is that you can get feedback from a lot of users without having them in the room. You can also test different versions more easily. A limitation of A/B-testing is that it requires technical adjustments to allow different versions to live side-by-side. Thankfully, there are a lot of approaches (e.g. feature toggles) and useful frameworks to make this easier (e.g. Sixpack or Alephbet).
One of the Scrum Teams that I’ve been a part of was responsible for building time-tracking software. The product was developed iteratively and in close collaboration with customers and users. At some point, we decided to develop a new dashboard. This dashboard acted as the ‘stepping stone’ for users visiting the application, providing them with key metrics and actions. We weren’t sure if new users would actually like the new dashboard, so we decided to run an A/B-test. After the release of the new version, we defaulted all users to the new dashboard (version A). But we added the option to switch back to the old dashboard (version B). We felt that people switching back was a strong indicator that we missed something. Although the vast majority of users continued to use the new dashboard, a few users went back. We interviewed them afterwards and identified improvements for the new dashboard. Several sprints later, the old dashboard was phased out in favor of the new one.
Just do it!
In this post I offered 7 ways for getting feedback from users. There are many more, like magic estimation or creating a product vision together. They key point to take away from this, is that this is not something that is either optional or a ‘nice to have’ in Scrum. In order to create successful products, we need frequent feedback from the people that are using the product. Only with this feedback, can we successfully navigate the complex domain of product development. It helps us to validate our assumptions and test ideas. So the best thing you can do after reading this post is, well …. do it :)
