Convincing Your Organization to do User Research
I love to create systems that help people more easily accomplish tasks and clearly communicate information. I lean toward creating templates for data collection and checklists for things that need to be completed in order, and sometimes I get to combine them.
My team at OnShift has begun to discuss growing an existing feature and how best to implement it. We had some questions about how the feature would change, how it would work, and if it would have the same usage as the initial version. Alex suggested that this would be a perfect opportunity to do some A/B testing, and we started to discuss our options.
Since Alex does a great job going through our technical and team considerations for A/B testing in his post, I’ll talk here about getting the business on board with the potentially expensive endeavor of A/B testing.
Culturing “Belief” in User Research and Testing
I went to a UXPA Cleveland talk a few months ago (April 2018) where Kirsten Butzow commented in her discussion of “Using the Wisdom of Ants to Build Successful Products” on the idea that belief > buy-in; if the team believes in the idea, they’re going to work harder to see that idea to fruition… and they won’t have to be sold the idea every time there’s a bump in the road getting to the finish line.
This idea of inspiring and culturing belief works better for me, especially in the user research and testing space because I don’t want to have to sell the idea that user research is valuable every time my team and I want to ensure we’re building the right product. I don’t have time for that, and if you’re reading — you probably don’t either.
My product manager and product owner had bought into the idea of user research and testing, but I wanted them to believe in it… and help stand up for the idea if we got push-back from upper management.
Finding Belief
How did I help my product manager and product owner find belief in user testing?
I engineered a template to tie user research goals (the things that UX is concerned about) to the business goals (the things that product people are concerned about).
The template addresses UX + business goals and takes into consideration timelines needed for the user research + milestones for the business to know when / if it should move from proof of concept → pilot → beta → early access → production. (And because I appreciate clarity, I include a glossary at the end so that anyone who looks at the document understands the difference between “proof of concept” and “pilot.”)
And Now… The Template
It’s important to remember that templates like these aren’t just for yourself / your manager, but for your team as a whole — including the folks who will be doing the development and testing of the product too.
I complete the template with my product manager and / or product owner. It fosters conversations and reminds us that we’re on this crazy journey together. If we are to get the developers and testers good product requirements, we need to be on the same page.
Because our template is stored in Confluence, it’s easy to make a clone of the template page and fill out the sections to create a project guide for each feature or project. I add the Table of Contents macro so it’s easy to come back to the project page to look at the actions selected or the established milestones.
Instead of walking you through a blank template, I’ve created a sample project so that you can see what sort of information to include and added some extra thoughts and instructions in italics. You can find the template without the extra notes on Google Drive here — just make a copy to your personal drive to use the template… or copy / paste to your internal tool.
Increase Interaction with Blog Posts — Proof of Concept / Pilot / Beta Project
The title starts as “Project (Template) — Proof of Concept / Pilot / Beta Project” and is what gets copy / pasted / re-named / filled out for each new feature / development project where some research needs to be completed.
Remember: I delete everything in this how-to-use section when I or my product owner is using the template for a project, but you can leave these details in if they’ll help your team.
Purpose of the Template
This template is designed to record information and decision points relating to proof of concept and pilot projects. This template can also be used to communicate decisions and milestones across the team.
How to Use the Template
Product owners should have an understanding of the problem and work with UX to understand recommended actions to test the hypothesis and set project standards for results.
The template should be completed for each project and posted in a public place / Confluence page so that it is easily accessible. The initial completion of the template for a project is expected to take about an hour due to the necessary discussion around the topics, key performance indicators, and milestones to move from proof of concept and pilot through to production.
Problem
I’m a big fan of the scientific method and like to use the framework, so my user research and testing methodology might sound familiar to some folks.
Completed Lean Canvas image:
We use a customized version of something called Lean Canvas to help with product planning / feature exploration, if this has been completed, I add the image here so it doesn’t get lost and so the team can see it / understand the value proposition.
Research and Testing Actions
Look at the user research / testing options below. You might have to go through the options more than once and explain to your Product Owner / team why you’re choosing one type of interview and not another, don’t give up — this is part of building belief in the user research — be calm and explain why in this instance your carefully considered actions are the best way to approach the stated problem.
Default Order and Timeline for Actions
Not all usability tests are required for each project, and the UX + PO team should work together to understand the business needs and available timeline for testing a proof of concept or pilot project.
Highlight the same tasks in the table below as the actions you checked in the actions box above — you’ll end up with a schedule of research tasks (yay Gantt charts ❤️ ).
Results
Use the following table to help define what results you’re looking to get from your user research and testing activities.
Follow-up
I think it’s important that everyone who has access to this document understands that if there are outstanding questions when this document is discussed, everyone knows who is going to follow up with whom and the questions they’ll be asked. Of course, it’s also important to update the document with answers to these questions so that they don’t get lost.
Lessons
The following are lessons that we hope to learn / the milestones that determine if a project can / will move through the process from proof of concept to production-ready code. (These milestones exclude approval from senior management and the business organization to create a new team or spend 💰 for UX, product owner, developer, QA, and ops time.)
If the results confirm and align with the hypothesis stated at the beginning of the project, then the project can continue through to the next step(s); however, if the results do not align, then it’s important to re-evaluate the hypothesis and restart with a new hypothesis + problem statement (and a new project sheet).
If you need to spend some extra time refining your hypothesis, updating what you’re testing, adding additional tests, or starting over with a new idea — that’s OK, it’s part of the process.
Glossary
(in order of development effort)
proof of concept (POC): This level of project is considered throw-away, or very low internal development effort. May use a 3rd party tool for initial development efforts to prove-out the idea to determine if it is worth continuing the investment of time and resources for the project.
This is a project that has some merit, though there is no official buy-in from the organization and this is not currently a sellable feature. The proof of concept is typically run with 1–3 customers; however, for a larger or more controversial feature, up to 5 customers may be asked to participate.
pilot / alpha: This level of project has passed the set of milestones required to move from proof of concept — whether that was due to internal development, research efforts, or a customer’s own efforts to enhance / customize the organization’s products; once the first set of milestones are passed, the project is eligible for consideration as a “Pilot.”
As a pilot, the concept has been validated and permission has been granted to continue to invest further dollars and development time to further validate the hypothesis. Once in this stage, the proof of concept development work may be discarded or built upon to get detailed feedback on design and user experience. Pilot customers are hand-selected for participation in the project because they meet pre-determined criteria and are highly likely to provide valuable feedback.
beta / early access: This isn’t a status we’re currently using; however, my team has had some discussions around implementation of beta releases.
This level of project has made its way through proof of concept and pilot milestones. The purpose of moving a project into beta status is to begin testing in a larger (but still limited) group of customers with increased scrutiny on interaction and user experience, with some focus on code performance.
production: Code is in production. The project is mostly development complete and can be marketed and sold as a feature. Additional development work and maintenance may be required as additional customer needs are discovered.
We’re just starting to work through our first template, so I haven’t had a lot of time to write about documenting / sharing / presenting results, though I promise it’s coming!
I hope that this has been helpful, and I’d love to hear how you’re using (and customizing) the template for your own needs!
