How to Figure Out if Your Product Actually Solves Problems
This project could have been a complete failure…
It started with ambiguous requirements and competing visions from different stakeholders. But, this project taught me five valuable lessons along the way about the nature of risk and how to navigate ambiguity:
- Examine and call out your assumptions
- Validate risky assumptions before building
- Go back to first principles
- Research efficiently
- Create and tell stories
For context, the project in question is Shopify Flow, a simple automation tool that allows the most sophisticated Shopify merchants to build automated workflows that make decisions and take actions on their behalf.
Examine and call out your assumptions
In November 2016, 15 folks from Engineering, Product, and UX at Shopify Plus kicked off a project to build visual programming.
This felt like any other kickoff meeting. There were slides outlining the members of the team, and background on why this project was to exist.
The pitch? eCommerce Managers need and want the ability to customize commerce logic and workflows on their own to better meet their needs.
The project pitch sounded reasonable, but after a couple questions it was clear that our new project was coming from a place of, “We think this would be a great thing to build,” rather than proven user needs.
Gut and vision have paved the way for many successful projects. But one of the best things our team did was to immediately identify that the concept underlying this project was our largest potential source of risk.
We decided to focus on researching the problem statement before heading into any engineering work.
Side note: As a product manager, this was maybe my 2nd month at the company, so identifying and calling out this area of risk was hard — but as you’ll see in this post, it was 100% worth it!
Exploring our assumptions gave us a chance to recognize potential risk and build a plan to give us a higher chance of success.
We broke down the original problem statement into its core assumptions and discussed how we’d know if they were true. Here’s how we unpacked our problem statement:
“Ecommerce managers have business processes that they want to automate with visual programming” breaks down into:
- Ecommerce managers… (Who are they? Do they exist as a persona with distinct requirements? Are they the person who might actually do this?)
- … have business processes… (What even is a business process? Does the ecommerce manager own them? How do these differ across different businesses? Different industries?)
- … that they want to automate… (Do they want to automate them? Are they already automated?)
- … with visual programming. (What’s wrong with programming? Do they want a tool like this? Could they automate their processes using a tool like this?)
We did not have satisfying answers to these questions, which was a good clue that this was the right place to start.
How can you recognize that your project is based on assumptions?
1. Identify the root concept statement
2. Break it down into its component assumptions
3. Ask questions about the assumptions to see how well they’re supported.
Here are some potential questions you may want to ask:
- What lead us to believe this is a good idea?
- What other concepts have we explored?
- Do we understand who might use this or why?
- Do we have research or data to support these claims?
- What have we done in the past that we might learn from?
Validate risky assumptions before building
After identifying that the core concept was the most risky part of this project, as is often the case with ambiguous projects, our next step was to find more concrete evidence to support or deny our problem statement.
Building and prototyping MVPs is a nimble way to test an idea when potential user value is the riskiest aspect. In the case of Flow, we did not yet know what to build or test because we weren’t sold on our underlying assumption.
How do you know if you’re ready to start prototyping and testing concepts? Start by identifying what you need to mitigate the greatest area of risk, and then figure out the best way to test that assumption. It’s ok if prototyping is not the right choice.
For us, that was digging deeper into those core assumptions and heading back to first principles for some foundational research.
Go back to First Principles
To get on better footing, we used a hypothesis-based approach to quickly prove or disprove the basic assumptions we had identified as core to our concept.
To do this, we identified First Principles that together described the entire product idea. For each statement, we identified how to disprove them. Why? If we were unable to disprove our hypotheses, we could feel more confident in the core concept.
In Flow’s case, we quickly realized that the only reason to build any level of visual programming for automation was if things to automate (in this case: business processes) existed and were highly differentiated. If every merchant wanted to automate the same processes, we should simply build those features for them.
So, armed with that knowledge, here’s how we broke down our core assumption: Ecommerce managers have highly differentiated businesses processes which are not currently well supported.
Which can be broken down into:
- There exist ecommerce managers (This would be untrue if we couldn’t find a role that was responsible for implementing business processes).
- They have business processes they can articulate (This would be untrue if they couldn’t express or talk about “business processes”).
And fundamentally we should build automation for these processes if:
- The business processes are highly differentiated (this would be untrue if we found that everyone was using the same processes).
- The current solutions are not good enough (this would be untrue if they liked their current solutions).
- They don’t have the technical expertise or motivation to learn code (this would be untrue if they were already comfortable with code).
Rather than answering one large question, we could focus on smaller questions and then zoom out to see if they still add up to the whole.
After identifying the building blocks, we had a way to plan research to de-risk as much of the concept as we could in the fastest time.
How do you know you’re going back to first principles?
- Have you identified important assumptions? — If they were true, would you feel confident moving forward? If not, what’s missing?
- Can you break any assumptions into smaller parts? — If they’re large, it’ll be hard to tell what’s true or not.
- Do you know what would make your assumptions false? — If you can’t disprove them, you will be plagued by confirmation bias.
- Do any of these assumptions rely on each other? — Beware of ordering. You’ll need to start with the most basic assumption before moving onwards.
Research efficiently
After identifying the basis of our risky product idea, we tried to disprove it as fast as possible using data that was already available. We’d only invest in digging deeper if we were unable to disprove them.
To start, we gathered what already existed from other research and feedback, and talked to user-facing teams to hear about patterns they identified.
After being unable to disprove our hypotheses with easily available data, we created a plan to fill in the gaps.
We gathered the broader team and brainstormed all the questions we had and all the things we wanted to know. We put them on a wall and clustered them into themes. Now we could see where research was most needed.
We decided that first we’d answer:
- Who are our users?
- How and why are they customizing their business processes?
- Does that work for them? What are the pain points?
Then we’d:
- Cluster use cases, look for themes
- Evaluate based on criteria
- Isolate a problem statement and initial use case to explore
We conducted this research with a combination of interviews, surveys, more detailed data analysis, site visits, and more.
Our research taught us that ecommerce managers did exist and did have differentiated business processes. We collected unique examples of use cases, and identified patterns in how they were expressed.
We learned tasks were accomplished today with custom code, off-the-shelf apps, and manual work. And none of those solutions were working well.
Our users were not all technically savvy — something important when evaluating visual programming as a solution.
These findings gave us confidence to move forward with the product concept, and helped us form opinions about how we’d approach design and implementation.
Most importantly? The concept itself was no longer the riskiest part.
Create and tell stories
We identified and mitigated another unlikely source of risk: buy-in. Because we re-evaluated the concept and came out the other end with an edited concept, we had to make sure stakeholders and team members were on the same page.
To mitigate this risk, our secret weapon was storytelling. Stories are essential for ambiguous projects.
Stories gave us a way to validate our ideas with people who had thought a lot about the topic already. If we weren’t able to convince them of our plan, we had a chance to learn why and find out what we might be missing.
As a team, we told stories both informally and formally:
- Coffee conversations, presentations, meetings, and other face to face chats to pitch ideas and hear feedback
- Narrative about research findings which pointed people towards more supporting information if they were curious
- Narrative about the vision we were trying to achieve and why we thought it was worthwhile
To start, we told a story about why we were going out and doing research and what we were looking for.
We had key phrases that we repeated over and over again, like “If we see everyone doing the same thing, we should build that feature.” or “If the processes our users want to automate are extremely complex, we will be looking closer to the code end of the complexity spectrum. If the processes are simple, we want to help them program without even realizing they’re programming”. These phrases evolved over time as we learned, but there were always clear and consistent themes we tried to drill into peoples’ minds.
Typically the best and easiest story you can tell about a project is what most users will use it for.
We called this the “hero use case” — the powerful use case that expresses the ideal use of the product and the mention of which grounds teams in real user empathy.
Somewhat uniquely, our project did not have a “hero use case.” One of our foundational concepts was that merchants would all use it in differentiated ways. We had to make clear that the project was inherently about the differentiation and not about the hero use case, and refused to give only one example in any context.
Stories gave us a way to validate our ideas with people who had thought a lot about the topic already. If we weren’t able to convince them of our plan, we had a chance to learn why and find out what we might be missing.
Few projects are handed off in such a way that all information is transferred from the originating brains to the executing brains. This was a context we could use to pull more of that information.
We also got lots of new brains thinking about a complicated problem space. Being new in a company like Shopify meant many of us didn’t yet know who we should collaborate with or who even had the information or data we needed. By telling compelling stories, we created a sort of distributed brain of all interested parties to think on our behalf.
Finally, we needed advocates. This was a project that wouldn’t follow established patterns at Shopify and challenged preconceived notions about what it would do or look like. By telling our story early and often, we created buy-in at different levels and in different areas of of the company. We had advocates who not only clarified what others said about the project, but also cleared roadblocks and gave us space to experiment.
What do you need to consider to make storytelling your greatest weapon instead of biggest risk?
- What is the most likely failure mode for this project? How might you fail?
- What kinds of stories will help prevent these failures?
- Do you need to convince people of the vision, or of the logic behind it? Both?
- Do you need to counter old stories that might continue to come up?
- Do you need more help with finding places for collaboration?
- What’s the right medium?
- How will you let people know when your stories have changed?
To be continued…
With a clear story and validated concept in hand, we were ready to explore how to build our solution. Originally, the riskiest part of our project was whether there was even a need for it. We tackled this head-on by identifying our assumptions, going back to first principles and validating through research that there was a great reason to create automation for Shopify Plus. Then, the risk became whether we could get buy in and shape conversations about this project. Now, our riskiest part was twofold: Could we actually express automation logic in a way users could understand, and could we build a system that would scale for the use we envisioned?
This is the first in a two-part series on how we built ecommerce automation at Shopify Plus. In the second part of this post, I’ll explore how we went about building Flow and the lessons we learned from prototyping to launch.
Check out reiterate to learn more about the craft of product.
