Vaporware to Software: Going from Prototypes to the Real World
Your team has been working diligently at building your new product, engaging users along the way to evaluate how well it addresses the needs of your target audience. Launch day finally arrives and the reviews are mixed. Feedback is coming through from all directions — while the core function of the product appears to be working, some of the interactions are clunky and navigation isn’t as clear as it should be in some places. Oh no!
The good news is that this scenario isn’t uncommon, in fact, it’s a normal part of the product development process. The trick is in how you and your team have prepared for this moment and how you respond to the feedback. Our team here at Craft have navigated a few of these launches, so we have picked up a few things along the way that may help you traverse the early days of your new product’s life.
Prepare for Launch
An integral part of the pre-launch phase of your product is shifting how you think about testing with prototypes. Too often, teams view strong prototype testing results as an “all clear” indicating the product has been fool-proofed. In reality, prototypes are an important step in testing critical assumptions you have made about the product and its users; however, they are not a guarantee that you have gotten all of it right.
Prototypes, particularly those built in design tools like Figma and Sketch, are frictionless representations of how the product can operate in a perfect world. Your product, on the other hand, exists in the real world, with all of its imperfections. Prototypes aren’t bound by the limitations of necessary technical decisions, back-end performance, technical debt, latency, real live data, and tactile interaction, but your product is. With that in mind, you need to surrender yourself to the idea that it isn’t a matter of if you will receive constructive feedback about the product, it’s a matter of when. More than likely, that “when” is the moment someone starts using it. Accepting this as a normal and healthy part of a product launch should put your team into the right mindset to hear what comes through and not immediately run for the panic switch.
Now that you have accepted that you will receive feedback about the product right away, you can shift how you are planning to allocate your team immediately following launch. While everyone will likely want to move onto the next highest priority feature in the backlog, you will want to make sure you are holding capacity across your engineering and UX teams to address critical issues that come through in the early going.
Establish a Beta Program
Before launching for general access, identify a population of early adopter candidates to participate in a Beta Program for your product. The Beta Program operates like a soft launch in that there is a working, coded product in the hands of users but you are still very much in “learning mode” as opposed to business as usual. The difference in the learning now is that you are getting to evaluate the one thing that prototype tests can’t teach you: how the product performs given all of its real world constraints.
Beta programs should be tightly time-boxed and have structures in place for participants to get acclimated to the product and then provide feedback in a useful way. As you progress through the program, make sure to assess participants’ overall interest in continuing to use the product. A question I have used in the past for this purpose is:
“On a scale of 1 to 5, where 1 is not at all upset and 5 is very upset, how upset would you be if this product was no longer available tomorrow?”
If participants indicate they would be upset if you took the product away, you are likely on to something!
Putting your product through a Beta Program requires a give and take on both the product provider and the participant. Participants need to make themselves available to provide in-depth feedback and answer questions posed by the product team as they use it; however, they need to get something in return. This could be free or discounted access to the product for a set period of time. Your product, finance, marketing, and sales teams should align on what is most appropriate given the constraints of your business.
Separate Signal from Noise
When you get into the feedback loop with your Beta participants, be prepared for the deluge. Feedback can range from the big picture of the product’s functional purpose all the way to the preference for a specific button color. Even if this isn’t your first trip through the product life cycle, the sheer volume of feedback can be overwhelming. Unless you’re encountering a critical application flaw (e.g., it’s core value proposition isn’t functioning), it’s important to refrain from hastily making a change without parsing through the feedback themes. The good news is that there’s a simple way to bring some order to the perceived chaos!
Gather your cross functional product team and list the individual issues you heard about on sticky notes. Try to keep the feedback oriented in terms of problems observed or experienced, rather than solutions. Problems are what we actually heard about, solutions are simply ideas we have to address them. Once you have everything listed out, look for patterns and begin to cluster those stickies and label each cluster. Now, you can plot your cluster labels on a simple 2x2 matrix:
- One dimension that measures the frequency the issue came up across your Beta participants
- Another dimension that measures the severity of the issue
Once this exercise is complete, you should have a clear sense of serious gaps that erode at the product’s value (upper right quadrant) all the way down to infrequent issues that appear to be more of a minor inconvenience than anything else (lower left quadrant). Naturally, your primary areas of focus should come from that upper right quadrant in order to get the product into a valuable state. After that though, the priority order gets a little more nuanced and will be influenced by both their placement on the framework and your business goals.
Issues in the lower right quadrant (Low Frequency/High Severity) will need a little more investigation. While you may not be resolving them right away, you will want to dig into them to try and reproduce to confirm whether or not it is an actual issue. Issues in the upper left quadrant (High Frequency/Low Severity) are items that you can pursue as needed. They may pale in comparison to big experience breaking issues, but eventually, they will be things that will need to be dealt with. Items in the lower left quadrant (Low Frequency/Low Severity) are more for your awareness. If they become more prominent or more severe, you can always reassess.
Prioritizing Feedback Using the Matrix: A Budgeting/Spending App
Let’s illustrate how we might use this framework using the example of a fictional consumer budgeting/spending app.
Higher Frequency/Higher Severity
These would be issues that many of your Beta participants have encountered that directly impact the core value proposition of the product. While these are hopefully issues that you have sussed out in prototype testing as well as development QA, it’s always possible for one or two things to arise. In the case of our hypothetical app, this could be an issue in which half of your participants were unable to complete the workflow for connecting their bank account to the app, therefore preventing the app from analyzing their spending activity. These are issues that you will want to address right away because you don’t have a product otherwise. Make sure you have capacity reserved on your engineering and design team to tackle these items quickly.
Lower Frequency/Higher Severity
While you may not make any wholesale changes to the product, these are items you’ll want to keep your ear to the ground on. In the case of our hypothetical app, this could be an issue in which you heard one or two participants encounter a chronic issue with the app crashing when they try to dig deeper into spending trends. You should try to recreate the issue yourself and keep listening for feedback as it rolls in. You may end up moving this issue to the upper right quadrant as you get more data or you may discover that it was a fluke.
Higher Frequency/Lower Severity
These issues are more of an “annoyance” than anything else. While they aren’t creating a big problem, everyone seems to be experiencing them. In the case of our hypothetical app, this could be a case where all of your participants encountered a slow load on a budget they had saved previously. While they certainly don’t carry the same gravity as the major issues in the High Frequency/High Severity quadrant, you’re eventually going to want to deal with them. As you start to smooth out the major issues, these “little annoyances” may eventually become bigger, absent anything significant to compare them with. Long story short, these are issues that will be addressed, just not right away.
Lower Frequency/Lower Severity
All feedback is good feedback, but not all feedback is really actionable. If you have one off issues that didn’t really break the experience, it’s probably not worth investing the time right now to address those items. In the case of our hypothetical app, this could be a case where one or two participants encountered a data visualization that did not render on the screen in a way that was legible. Who knows, you may hear these come up again later as the product proceeds into general access, at which point you may want to address them.
The discipline you develop documenting and decisioning feedback can and should be repeated throughout the product’s lifecycle. In reality, we never stop gathering feedback and identifying new opportunities to make a better experience.
Develop Some Solutions & Test Them
With our high priority issues identified, it’s time to start the Ideate-Prototype-Test cycle again. As you re-enter this cycle, it is critical that you start with the observed problems to ensure any solutions can clearly tie back to issues users encountered. This requires discipline and, not to mention, a few team members willing to be those people who constantly ask the others, “what problem are we solving here?”
There could be multiple solutions to explore so work with your product team to generate ideas that address the experience issue and are possible to deliver with relative speed. You can use your Beta participants to test these prototypes to assess whether you have actually addressed the problem.
The real magic comes if you are able to turn around a solution in production quickly. Nothing delights a user more than when they have tangible evidence that their feedback was heard and addressed. Moving quickly on critical issues is a great way to engender loyalty and some advocacy/word-of-mouth promotion with your early user base.
It just got real
Prototypes are a representation of what we want our product to be. That first release is what your product is. Going from design to production shows you a lot about the feasible reality of the product we imagined. While our natural inclination is to bring the product to the masses as soon as possible, starting small with a Beta Program gives the opportunity to fine tune all of those things that prototypes cannot teach us. Moreover, it provides the tools and exposure to critical feedback that your team needs to embrace the idea that the product is never finished.
