Bug 報告拳 Hōkoku-ken — The Way of Bug Reporting
In my brief (but wildy exciting) time living in Japan, I was often astounded by how engaged and motivated local people were about their basic tasks. Even folks with jobs that seemed repetitive and back breaking often looked straight out of a 1950s Commerce PSA, or that first scene in the Lego Movie!
A boss of mine told me that’s because many people understood 1. much of their jobs were thankless, but very important to the people around them, and 2. all had a technique that you could learn to be as efficient as possible. An elderly bartender at my local izakaya often described his approach as Yakitori Ken, literally Fried Chicken Skewer Fist, but more accurately the Way of the Chick Skewer (and let me tell you they were AMAZING). He was weirdly proud to use the least amount of movement to flip dozens of meat and fish skewers over the flames at once, and it was kind of beautiful to watch. Hey, if you can be one thing, be efficient I guess!
Reporting Bugs and QA, are maybe the most simultaneously thankless yet critical practice for all SaaS companies. Early in in your company’s life cycle, Bug Reporting and discussions maybe as frequent as your Customer Development conversations with early users and partners, and on top of an opportunity for feedback, these support related interactions are an opportunity to show how quickly and thoroughly we are able to resolve issues. A well functioning Customer Success and Support team can turn even critical bugs and issues into a chance to shine and drive value and delight with your service.
Early on in your SaaS business you may have few Customer Success Managers or Associates. Heck, this function may not formally exist at your business and it’s being handled by a founder (how I discovered Customer Success!) or even directly by members of the dev team. Regardless of who you maybe, you will be in the same boat, as you’ll be expected to balance the needs of those who have the problem, with those who can fix it (the User and the Developer, respectively). This is the Art of Bug Reporting, or Bug 報告拳 Hōkoku-ken.
The user is reporting a bug for a reason, they want your app/software to work better than it currently does! However, what few but the most zealous of power users are willing to do is write a full military SitRep or medical Post-Mortem of the issue.
Meanwhile your dev team will demand as much information as possible, and for a good reason, they want to be able to fix the issue as soon as possible! And in order to eliminate several variables that could make the problem, well practically anything outside of their control (hardware or browser issues for instance), they’ll need key pieces of information.
Key Questions Developers Are Looking For
Key Identity: this is a piece of information that can identify which user from which account the issue(s) are happening to. This can be an email address, phone number, username, etc, and will locate this user in your database.
Platform: Did this bug appear on a native app on Mac/PC/Linux, an iPhone or Android app Note the model year and type for any device, and does your product support it?
Operating System: What version of the Operating system are they running? Does the user need to update it? Browser type/version. NOTE: When working with large enterprises using VDI or user management systems, they maybe forced to use older OS and Browser versions for security.
Account Context: some products allow users to have multiple accounts, either in a freelance capacity or to separate business and personal. Noting the instance/environment/account etc you are working in can be critical info, depending on custom set ups. [doesn’t believe PROJECT is useful, will get this info from user info already most likely, could be part of multiple projects in the future, not for user]
Feature: Which feature is breaking. Is this happening across platforms, and across users?
Before/During/After: What happened immediately before you tried to use the feature, what happened as you used the feature, and what happened after (error message, freeze, etc.) BE AS SPECIFIC AS POSSIBLE.
Frequency/Repeatability: Does this happen every time you use this feature, some of the time, or occasionally?
The Evidence: Do you have a screenshot? Do you have a short video of the issue? Can you send us the app log or open your browser’s DevTools?
Getting as many of those questions answered will help your Developers trouble shoot the issue as quickly as possible, but may exhaust your users to provide. See how much you can get them to give up front, perhaps in some sort of form. Conversely, you can get the basic info, and follow up in a phone conversation or written back and forth. Phones will of course get the information faster, but you usually want to go with the user’s preferred mode of contact, as long as your process scale can support it.
Either way, thank the user for every new piece of info you get from them as you make the report and trouble shoot, add that you know you understand this [process can be frustrating, and let them know they’re help is crucial to getting the problem fixed, which should get them up and running. It’s never too late to remind the user what their getting out of their help!
Support Segmentation: Severity and Priority
Now that we have defined what we need to troubleshoot, we can address segmentation. Like all parts of the Customer Success world, Segmentation allows us to see which issues need most of our attention any given moment. And by looking at two key factors, severity and priority, we can infuse a good amount of objectivity into what could be a very subjective process.
Severity relates to the level of impact at which a bug/issue is affecting your service or product. This of course relates to the functionality and quality standards you are presenting to users.
Severity Levels –
Critical — functionality is completely down and/or there is a major security breach/liability
Major — a major feature is down but can use other parts of the product
Moderate — Noticeable issues that do not block the functionality of the product, but are unpleasant.
Minor — does not affect functionality, barely noticeable.
Priority
Priority relates to our schedule of resolving issues and bugs, and will create an order and a timeline to the use of our developer resources. Essentially, which bugs are most important to my business? While Severity can at times be assessed on a bug-by-bug basis, Priority necessitates the consideration of customer experience as a whole and compares bugs to each other. Factors that can affect priority:
1) How many users and accounts may the issue affect and which ones?
2) How long will it take to fix?
3) How much resource demand will it take to fix (note, estimate quickly)
4) Are we already working on something that may fix this?
5) Is there a stability risk in shipping a fix?
6) Is this affecting an important client at a key time in the relationship?
Streps of Priority
Urgent — The business as a whole is impacted across the board, we will be losing money! We need to act immediately (ASAP)
High — Major functionality is restricted and with a high business impact across many accounts and users or a very valuable account, but there is a temporary work around that maybe hard to use. Fix as soon as we can get to it (1–2 days)
Medium — Functionality is somewhat limited, but there is an easy work around. Business is not impacted majorly. This can be fixed soon (1–2 weeks)
Low — Functionality is not limited, and there is no business impact. This can be fixed whenever we have spare time (1 month)
Lowest — Is this even a bug, do we need to fix?
Let’s look at some examples:
Ex. 1
After a recent release of our main features is broken when used with an older version of Edge. Several of our biggest clients with serious SLAs run older versions of browsers for their on-prem-network due to security concerns, so they cannot switch browsers.
Designation: Major/Urgent
While the bug only affects one feature on specific, earlier version of browsers, this has a large business impact. Though clients running older versions of software can expect slower recovery times, this is impacting key clients using a major feature, without a work around, and could lead to lost revenue and breaking SLA terms if the issue is not resolved quickly. Reach out to affected clients as soon as possible, explain the situation, why this bug is affecting them, and what the turn around time on a solution would be. From a Dev standpoint, this is will require the immediate attention of multiple resources, if not an all-hands problem. Fix ASAP.
Ex. 2
While running analysis for our clients correctly, a feature does not display the correct UI for when the process is finished, leading users to believe it is almost finished but never done. This only affects Chrome, and a hard page refresh solves the issue.
Designation: Moderate/Medium
The bug does not prevent users from getting the value of the feature. Using a different browser, or doing a simple refresh once the user notices the issues, will yield the desired result without too much effort. Push out email to users noting the basic work around. Fix 1–2 weeks.
Ex. 3
Project avatar times out on upload when file is larger than 5 MB. This bug affects 1–2 screens with a small impact. Uploading a smaller file size will fix the problem.
Designation: Minor/Low
This bug will not be noticeable to most, and has a simple work around. However, it does affect the onboarding process in certain situations, so we will want this fixed at some point. Fix when there is a free moment in the next few sprint (1 month).
Conclusion
The Way of Bug Reporting is truly an art, a give and take between customer and developer needs, as well as technical and business considerations. A good Customer Success Manager is always a skilled collaborator and communicator, and must be obsessed with maintaining transparent expectations between all stakeholders and teams involved. Now go out there and practice some 報告拳!