Starting Products: An Introduction and a Lesson
Startups are nuts. And I mean that in a good way! Mostly. There’s a ton to be learned, and the experience is like dog years — 1 year at a startup is like 7 elsewhere.
I’ve learned a lot, and I want to share what I learned with you. I’ve spent the past 8ish years working at a variety of startups (hardware, software, B2C, B2B) as a product manager. I’ve also consulted/advised other startups and helped friends with their projects. I’ve largely been at pre-product or pre-product/market fit startups, and that’s a completely unique experience that many people don’t have and can’t truly imagine what they’re like.
I’m planning on sharing what I’ve learned through a series of blog posts I’m calling “Starting Products,” covering topics such as “You’re (Probably) Not Going to Sell a Million” and “Value Over Vanity.” I might even do one called “You’re Not Steve (Jobs).” If there’s a particular topic you want to make sure I write about, find me on Twitter (@joshanon). I’ll largely focus on the experience from a product-management perspective and what PMs encounter, but hopefully more than just PMs will find these posts useful. Also, some of the learnings — like in this post — apply to building all products, not just at early stage startups.
For context, I have a computer science degree and spent the first 10 years of my career at Pixar Animation Studios, doing a range of technical and creative roles. I was there before and after the Disney acquisition, and it was an amazing place with a lot of success. On the side, I built and sold products, some with millions of customers. And for better or worse, the startup bug bit me and I dove into it full-time. I also wrote a book about how to be a product manager that’s turned out to be a popular beginner reference.
Across all these experiences, the standout single biggest thing I’ve learned is to get your product — even in a very rough form — in front of customers as soon as you can and listen to what they say. To be clear, I’m not saying they know exactly what they want (a car vs. a faster horse), but listening to their feedback will ensure you’re building the product they need and will buy.
The most successful application of that principle I’ve seen of this was at Pixar. Pretty much all of our movies were a disaster in their first draft — even if they had a great table read. We learned to accept that the first version was going to change a lot and accepted that was OK and part of the process. Lightning McQueen wasn’t in the first concept for Cars (The Yellow Car). Up started as a story about two princes who lived in a floating city on an alien planet. Knowing the story would need to change, we focused on storyboarding them out as quickly as possible — making a prototype — so that we could watch them in the theater with an audience, just like you would watch a movie. The director and team would take feedback from the audience (notes came from everyone, not just the Brain Trust). Then the team would go address those notes, and a few months later, there’d be another screening. As Pixar co-founder Ed Catmull discusses in Creativity, Inc., this constant iteration and willingness to throw away ideas that aren’t working is what made the stories great. At least in my experience there as we worked on a lot of tools for process improvements, movies always took about the same amount of time to make but the process improvements let us iterate more so the final quality was better.
Testing early, iterating quickly, and testing again sounds really easy and obvious, right? Every startup should be able to do this with no issue, right? Well, not in my experience. The issues I’ve seen fall into two and a half camps. The first is when people focus on the flaws of the prototype rather than the potential value and refuse to prototype/test early, the second is when people don’t listen to the feedback — this comes in a few flavors, and the half is when people decide the feedback’s too much work and ignore it. Let’s dive into each.
Sometimes, people focus on the flaws of the prototype rather than the potential value, and refuse to prototype early out of fear the flaws outweigh the pros. I worked on a conversation-based product where some folks didn’t want to prototype as soon as possible because all the early prototypes would involve a human in the loop. They had valid concerns that a human could respond in ways an autonomous system couldn’t, and we found ways to mitigate some of them, but we couldn’t solve all of them. The net result is we didn’t really prototype any content until we had an autonomous system up and running. That made it incredibly hard to test a breadth of ideas to see what the customers reacted to best as we just didn’t have the time to do so in our master “need to ship by X or run out of money” schedule. Don’t get me wrong, there’s good content, but I firmly believe that if we’d started testing sooner, the product would be stronger. Unfortunately this can be a no-win situation for a PM, because either you accept it and don’t test or if you independently find ways to put a proxy in front of the customer sooner — unless it validates what the team wants to hear — you’ll encounter the second issue. People won’t react well to the results of the test because in this case, they’ll question its validity.
The second major issue arises in how people handle the results of a test. In an ideal world, you’d know what input is valid at this stage, and if all your customers are saying something you don’t want to hear, you change — or at least re-assess — what you’re doing rather than assuming your customers are dumb. I’ve found it rarely works this way in practice.
I’ve seen a few common reactions to the results of a test. The reason you’re testing is to see if what you’re doing is valuable for customers and if you’re on the right path. The most unfortunate reaction I’ve seen is when people just don’t like — and don’t want to accept — what customers do/say. If this is due to a shortcoming in the prototype, OK, this test wasn’t fully valid. But otherwise, if you want to be successful, listen to what the customer is saying and don’t be afraid to change what you’re doing to address their needs because otherwise, they won’t buy your product.
This non-acceptance takes many forms. I will never forget one discussion where the user testing team was reporting the customers were unhappy because there was so little content to use on a device but the device was fine and they liked it, and the founder was saying “no, the problem must be the prototype device’s form.” In this case, I think the founder disliked these tests because the device we tested on was nowhere near his true vision — and he believed our core value was in the physical form that vision took. I believe that getting user feedback that the value was in the higher-level software on the device challenged his vanity.
I will also never forget a conversation at a different company where after learning about what was going on with sales, the market, and the tech’s possibilities, I asked, “why are we focused on making a consumer product that consumers aren’t buying — and those who buy it don’t use it — when we could repurpose the tech for an enterprise market where they’re actively interested in what we’re doing?” I was told it’s because we raised $X million to build a consumer product. Suffice it to say, they’re out of business.
The learning here is that you shouldn’t be afraid of getting negative feedback: early-stage startups are often successful because they have the right people, not necessarily the right idea on day 1. Slack was originally an internal tool at a game development company. It’s OK to pivot/change what you’re building if you find what you’re doing isn’t useful or you can get to where you want to be faster with a different and easier/faster/cheaper approach. And the earlier you find that out — which is why you want to put your product in front of customers ASAP — the better. This can be really hard for a PM to make happen because the idea is often the founders’ baby, but if you see an opportunity, suggest it, and if it’s dismissed, move on. (Which is always easier said than done when you see the customer isn’t best served by what you’re building.)
Another reaction I’ve seen with testing early is when sometimes, customers react to something that’s a known issue but is exacerbated by the prototype nature of the test and not currently the priority to address with your limited resources yet internal stakeholders focus on the known issue. For example, if you’re testing a new hardware device, in the early stage, a lot of the components are large to make them easy to work with and they’ll get replaced with smaller components later. A prototype smartphone generally won’t fit in your pocket. If the test goal is around say image quality and you get notes on that but also a lot on weight/size, well you already knew weight/size was an issue. If the priority-setter hears customers disliked the weight/sizes and focuses on that, internally, what results is a mis-alignment of priorities between whomever’s building the product and what they need to be successful and whomever’s setting overall priorities (generally the CEO, not the product manager, in early startups).
The best case for handling this issue is when “the boss” trusts the team to prioritize correctly. For this to go well, you need a rational leader, good project management, stakeholders setting priorities together, or all of the above. Unfortunately startups often have limited project management, the trust isn’t there yet, and I’ve seen many founders just be unhappy that everything wasn’t done simultaneously yesterday. The worst reaction I’ve seen to this is when it spirals and people internally stop supporting user testing because they know that while valuable, they can’t react to the issues coming out of it right now given other priorities and don’t want to deal with the frustration of being asked why they didn’t address said issue(s). This is quite bad because it later becomes an uphill battle to get user testing incorporated again, and in the short-term, the team’s missing out on the opportunity to test what they’ve built.
A good way to handle this situation is to have clear goals for what you’re looking to learn at this testing phase and to work closely with whomever is communicating the results of the test to focus the communication on what you learned around those goals. To be clear, I’m not saying you should hide an issue, but if there’s a known issue that’s not impacting or part of the test’s goals, make a slide that says “Known Issues to Be Addressed in Later Tests” later in the presentation and put the users’ issues there.
A third major reaction to the results of a test is when a PM, CEO, etc. sees a solution to a problem the customer raised and is then prescriptive about the fix to the team. I’ve been guilty of that, and I’ve found I’m much more effective if I communicate the issue and then phrase my suggestion as, “one thing that would address the problem is if we did X,” clearly making it a suggestion and trusting the team to address the note. This is important because often the person being prescriptive isn’t close enough to the implementation to provide the right answer and sometimes, the right fix might be a broader change that eliminates the issue. Of course if the issue remains unaddressed — or not addressed successfully during subsequent tests with whatever fix the team tried — then sometimes someone needs to be a bit prescriptive about what to try next.
The final issue I’ve seen arise from putting a product in front of customers is when someone declares that it’s too much work to fix the problem the customer’s having, especially if it involves redoing something big, so they don’t want to address the problem. Sometimes, that’s OK especially if the issue isn’t core. However, I encountered a time where every user test showed one clear interaction change that would make the customers far more successful. The person in charge of doing that change said, “This won’t happen. It’s too much work.” This is a hard situation because as a PM, you can’t make the person do that work, and in startups with tight budgets and timelines, you have to balance the potential customer value with the time cost of redoing the work or doing the hard work balanced against everything else you want your team to do. I’ve found that the best people usually don’t hesitate to do the hard work or throw away work they’ve done to re-do it better, and usually as the PM, with those people, I’m the person pushing to not necessarily do it right away. In the example I mentioned, there was nothing I could do, and this issue kept coming up in testing until the founder told the other person that they had to address it.
Whew! For something that seems so easy — put your product in front of customers — it’s amazing what can go wrong. But I will say even though I’ve seen ways a company could react better to customer feedback, 100% of the time the ones that put the product in front of customers early do better than the ones that don’t, as they have some information to help guide decisions and priorities.
So how do you establish a culture that gets your product in front of customers early? The best way to start is with a solid internal demo culture. If you’re following scrum project management, there’s a sprint demo each sprint. Have the demo! If you’re not using scrum, just set regular time frequently to show off progress. Don’t worry about only showing sexy things or being over-prepared for the demo with a presentation that took a week to make…show your work. The benefit to a startup is that you’re small and should be able to move fast!
If you’re the person giving the demo, seek honest feedback even if you don’t want to hear it. The criticism really is about the work, not you. One attribute of a great artist at Pixar is always having another idea. I personally worked on shots where we did 10, 20, or even 30 versions until we found the right one. Animators have “dailies” at 9am each day, where they show off what they’re working on and get sometimes brutal feedback. And of course people celebrate when a shot’s approved.
And if you’re the person receiving the demo, please make sure your comments are appropriate to the state of what’s being demoed. I’ve seen a startup where the CEO’s expectations were never aligned with what work-in-progress looked like, and the result was the dev teams started keeping everything secret. This was bad because the CEO felt like there was no progress, the team felt like the founder was out of touch, and it killed the demo culture.
Don’t be afraid. Show your stuff.
Continue on to part 2: The Early Days or Lots of Free Time With Too Many Meetings
