Conducting lean experiments
At Pivotal Labs I spend a lot of time helping companies understand their challenges and explore viable solutions. This involves uncovering risk and building the simplest thing first. Before investing time, money and energy we want to make sure our solution will be useful.
I’ll share a product idea with you that I recently worked on and walk you through some simple techniques that can help you apply lean methodologies to your own products and services.
Scenario:
For small business owners, engaging with legal attorneys is expensive. Most cost somewhere between $350 — $800 per hour. Knowing where to even begin can feel daunting. How might we create a service that helps small business owners who need legal advice on things like human resources, protecting customer data, payroll tax, or adhering to labor standards?
After doing some brainstorming and a business model canvas, we came up with the idea to provide some sort of concierge service that uses artificial intelligence to provide information on common legal scenarios small businesses often find themselves in. But building artificial intelligence is no small feat. Teaching a machine the ins and outs of the small business sector is complex. What could we do to figure out if this is even a viable solution?
The first step we’ll need to take is to identify what assumptions we’re making that impose risk. We can do this with a priority matrix that measures unknowns vs impact on success. On post-it notes we’ll write down any assumptions that we’re making and place them in the respective position on the axis.
Our riskiest assumption that we’ll want to focus on first is the one that has the most unknowns and the biggest impact on success.
Just because you and I might be comfortable with the idea of conversing with a machine doesn’t mean that other people will feel the same. There is a level of trust we expect when seeking legal advice. To assume people will feel equal confidence in a robot is a significant leap.
What experiments could we run to start testing the waters on this idea before we dive into machine learning and expensive software development?
Wizard of Oz testing
A Wizard of Oz test is essentially a human behind a screen manually performing the tasks of the product. When Zappos was starting out they didn’t actually have any inventory. They simply posted shoes on their website and when they received an order they ran out to the store and purchased them. This was an experimental way for them to figure out if they were a market fit.
In a similar way we could set up a chat service on our website that’s run by a human, but we’ll introduce ourselves as a bot. We’ll quickly begin to learn what kinds of questions people ask and how they interact with something they don’t believe to be human. We can even test their tolerance by making intentional mistakes. We can give people the option to speak with a real human and measure how often people opt for human interaction over the bot.
Qualitative interviews and contextual inquiry
By interviewing people and asking open ended questions about their current process for obtaining legal advice we can learn about their implicit and explicit needs. We’ll start to see patterns around where the challenges lie and what sort of solutions makes sense. Instead of asking people if they’d use something, we try to probe at their current behaviors and anticipate how those could inform habits they’ll be more likely to adapt.
Smoke testing
By putting up a website that offers the service before it actually exists we can start to gage interest. We could do this with a signup form or a crowdfunding campaign.
Solutions testing
We could create a prototype of our idea and put it in front of people to see how they respond. Even if they can’t interact with it directly, we could learn a lot about their initial reaction to the service offering and thoughts or concerns they may have with it.
Effective experiments begin with a hypothesis and should be conducted in a way that allows them to fail. It’s important to define what the measurement of success is. If we go with a Wizard of Oz test, we should identify what percentage of participants we need to positively engage, and if the experiment fails we need to know why so that we can course correct and change direction as we learn new information.
If we discover our hypothesis is too open-ended and our results are quite mixed, we can narrow it down. For example, we can go from stating “We believe that small business owners will trust a bot enough to sacrifice human interaction” to “We believe that small business owners will be willing to share sensitive information with a bot” or “We believe that small business owners will not give up and request a human the majority of the time”.
As we run experiments and begin to validate some of our assumptions we’ll begin to learn about the true needs of legal advice for small business owners.
At Pivotal Labs we offer an hour of free intensive product consulting over lunch. We’ll help you with a product challenge and make sure you walk out the door with actionable feedback to move your solution forward. If you and your team are interested in participating please visit https://pivotal.io/office-hours. We have offices across the globe and it’s completely free, no strings attached!
