Building a Great Product — Startup Founder Panel #4

Panel discussion at Rise New York on August 26th, 2017

What can startup founders learn from product managers? What are ways founders can measure growth and make data-driven decisions? How to handle and prioritize customer feedback?

These were some of the questions we discussed in our 4th Startup Founder Panel. We had the following panelists:

Here’s the transcript! 🎉 (audio at the bottom of the page)

Tom: The theme is building a great product — the intersection between product management and entrepreneurship. When people start a startup, the role of “product manager” doesn’t come in until the team is larger. Because of this, people have to wear many hats in a startup, so startup founders need to have understanding of these concepts.

As both entrepreneurs and product managers, what is your process to develop a product? What process do you use now, and how has that changed over time?


Cody: So I came into leading product for organizations from marketing. So when I started on product, the history of knowledge that I had behind me was interacting with the consumer: pitching a lot of things out and gaining knowledge from them.

And before I even knew what I was doing I was doing user experience research. It was lazy and not very academic. And I didn’t really know it. But in the end what I was doing was learning from my audience what we needed to build.

And then later in my career when I ended up being in charge of product for early stage companies, being the first employee or among a small team, I really relied on that background of knowledge as the foundation for how I thought about product.

So, when I started in product it was about user experience and creating designs and creating features and user stories, which maybe we’ll talk about as we talk about the artifacts of product management and what it is.

But if there is something that has evolved now which is important for both founders and product people it’s that my definition of product has evolved to be built upon three sort of facets. That is product and process and project.

And if you are the person in charge of product for your organization, you’re likely going to be responsible for all the things that make up product — the vision and the design and the roadmap and the features.

You’re also going to need to be responsible for the process, depending on the size of your team. How do we work? How do we keep track of work? How do we organize our efforts? Do we work in sprints? How do we share information across the team? Who else is going to do that other than you, or who else should care about it more than you?

And then finally it’s project. That’s the nitty gritty of a sprint and engineering estimation and other components like commitments and you know management tool artifacts. Are you using Trello or are you using Jira? Are you watching your burn down chart?

So if you look at all three of these things.. What I thought of product in the beginning was designing features. When I think about it now it’s, how does my team work and how do we build this thing?

Laney: To continue on that, let me give a little backstory about myself. I’ve had a unique route to product management. After I graduated from college I joined a four person startup here in New York. We build a personal assistant that schedules meetings powered by AI.

I didn’t come into this with a product background. So, early on my role was understanding the customers and what they were looking for in scheduling meetings — what were their expectations?

As the team grew with more engineers, it became a matter of translating those expectations into something we can build and scale from an AI and machine learning perspective. A year into that process I moved into a formal product role.

As the company grew and the team grew, my role changed. I was working more with the engineering team and the data scientists to figure out how can we translate what we want into something that we can build. And so it’s definitely something that changes as the team grows.

Andrew: At Brainscape our backlog of features and things that we would like to build is so much bigger than one might think. You might think, “Oh, it’s a flash card app. How hard could it be?” But there’s 10 years of stuff that our users would love us to have. I think Tom actually may have been a user at some point.

Tom: That’s how I learned Spanish.

Andrew: And so we have to work backwards from startup long-term goals and then look at medium-term goals and short-term goals. The long term goals are a quarterly or semi-annually off site where we really understand our North Star and where we would love to be as a company and a product 2+ years from now.

And then we work backwards from there. What are the capital needs? What type of resources would we need to get there? And what could we do in the medium-term that would allow us to create those resources.

Whether it’s, we need to raise a whole bunch of money. And so we need to create traction and hype and all those types of things that would attract VCs.

And it might be a different feature set then if we think we can generate the revenue to get us there and hire more people. Than that might be a little different than what’s attractive to investors in the medium term.

And then finally you get down to the short-term. And I think when you’re comparing the medium-term and the short-term, there is a big tradeoff between metrics — focusing on what’s going to move the needle more — vs. technological pre-requisites.

A lot of times there’s this one thing that would move the needle the most and it’s the most exciting thing that we could possibly do. And technically we could do this next and start next week. However, it will make a lot more sense and be a lot easier to do this one thing if we’ve done these three months worth of other things.

It would just make design easier. It would make it easier to integrate. If we’re going to upgrade to the latest version of Ruby on Rails or refactor the database or whatever… So sometimes you have to do plumbing things product-wise. Sometimes it makes it a lot easier to do next products after you’ve done plumbing catch up.

So it’s a constant trade off in the short- and medium-term between tech prerequisites and metrics. But it’s always in the service of, are we moving toward that long-term vision and not straying off on some tangent because some user asked for a random feature.


Tom: I think that’s interesting, because I see product as two parts — the vision, or finding product / market fit, and then the project side — managing a team and working with them towards that goal.

How do these two things inform each other? Andrew, you’ve grown your team from very small so you’ve probably worn both of those hats, so I’m curious what you think about that.

Andrew: There’s a lot of getting the team to drink the Kool-Aid that you need to do. And part of making sure that everybody is really bought into that long-term vision so that they’re making the right sort of medium-term decisions with you, is that they need to feel like a lot of that long-term vision is their idea.

If you are the entrepreneur and head of product — which sometimes is one and the same, sometimes it’s two people — and dictating to the people who are building it… “ I want you guys to build this.” It’s not always as engaged when the boss says, “This is a job” vs. if they feel like I actually contributed to this vision.

And of course when you start, it’s one person in a garage or two or three people in a garage coming up with that vision themselves and getting other people to drink that Kool-Aid. And upfront, people will say, “OK I’m in.” But over time how does that vision end up getting shaped?

And I think the most important thing for that is to fall in love with the problem and not the solution. So the vision is we’re going to solve this problem. The vision isn’t, we’re going to build this set of features to solve this problem.

If you’re so married to that early on then you’re probably not going to be as agile as you need to. I don’t know if that exactly answers the question that you’re getting at.

Cody: Can we do a hand scan here? How many of you are founders that are in charge of your company? How many are product managers at a company that is not your own company? How many of you are in charge of a product team, with product people under you? Not a lot of people worried about training others yet, which is it’s own thing to talk about.

You all have a different job depending on where you’re at and where your company is at, and there’s this delineation between project, product, or operations. It’s so varied.

If you are a solo founder and you have decided that to create your product you’re going to hire a dev shop in India, then you’re now a project manager no matter what. Because your primary focus is going to be on making sure that you get value from that team.

By the way, that’s not a very wise thing to do. Don’t do that. Maybe you’ll be successful but you’ll probably have a lot of headaches.

Otherwise if you are maybe Andrew at this point, you can really be about the vision and being the CEO, which is pretty important. It’s good that you can offload those tasks, and depending on where each of you is, what is most important is going to split.

The most interesting thing is that you’re all here for slightly different reasons and product is what you all do, because it’s a very ambiguous thing.

Tom: Andrew brought up the ability to sell your vision. The CEO of X.ai, Dennis, is very good at this. I’m curious from your perspective, how does your experience resonate with this, with getting the vision from the founder early-on, and seeing the decision-making process grow over time.

Laney: Yeah, so let me talk a bit about how we work and how that’s changed. When you’re five people in a room, it’s really easy to understand what the vision is. And then you reach different inflection points as the team grows where somebody doesn’t quite know what it is.

So the way that we handle some of that is always grounding on, “What are we here to do?” And we’re here to schedule meetings. So we want Amy and Andrew, who are our intelligence agents, to schedule meetings and take all of those meetings off of your plate.

So that’s the very top high level vision. There’s no confusion in the office on what that goal is. And then we also break it down into quarterly goals, between 3 and 6, depending on the quarter.

We may have goals on acquisition or customers experience — different aspects of the product that then trickle down into the team getting to figure out what are the projects that we can take on that will help us meet those goals. So the goals are never prescriptive in the sense of, we need to build X, Y and Z feature set. Instead, it’s up to the team to figure out what do we do to get there.

So it’s a very inclusive process across all of the different teams. It’s not just product or just the founders coming up with ideas on what should we build. It’s engineers and data scientists and customer experience teams contributing as we work through what can we do to reach those goals.

So I think that really helps with getting the whole team to feel a part of the process of meeting those goals, rather than a more top-down approach. That is something that has worked really well internally at X.ai.


Tom: Andrew alluded to metrics, and I’d like to bring that up. In our discussions we figured out that we have different ideologies about metrics. We live in a time where so so many tools are available that five 10 years ago were not. It’s not just Google Analytics. There’s all kinds of Web analytics where you can know about your customer acquisition.

So how do you use metrics in your company? And how do you get the most out of these tools that are available to drive the product forward?

Andrew: I think in the same way that you set your long-term, medium- and short-term goals, you look at your company in a similar fashion. So if you’ve raised money or plan on doing so, I hope that you’ve been working on a growth model — some kind of a spreadsheet.

It would ideally be a centralized Google spreadsheet that people in your management team and potential or existing investors could all look at and dig into.

And that typically is your meta metrics — right from the very top of your funnel, a number of page views or new companies in the pipeline. It’s not always online metrics, for example, if you’re B-2-B.

So once the business is making revenue you can look at things like churn and retention and repeat buying. Then you can look at how to grow that revenue, and use marketing to add fuel to that fire of growth. That’s the typical kind of growth model that you want to show to a VC.

And when you look at that on a quarterly or semi-annual basis with your board or with your team, that’s going to help you drive the one or two important metrics. Whether it’s a revenue goal, a user goal or a retention goal… It’s testing whether our model is even going to work, or if you’re looking at a viable corporate development, how does a certain metric fit into that.

And then once you pick that, then there are probably tools that go beyond a spreadsheet. Once it gets into a granular level of detail, then you don’t need to be bringing it into a spreadsheet. You can be using Mixpanel, or Looker, and make analyses.

Then you’re looking at all the different group stages of your funnel. There are hundreds of little micro behaviors that are involved with just conversion, to get a user to sign up on your web page. It’s even as granular as how far down the page a user is scrolling.

I mean you can get really granular with a lot of these tools. But you should also pick qualitative data points and ensure that you’re bringing in usability testing and real, live feedback. Bringing those together is really important.

I know a lot of times people think it might be overanalyzing by digging into metrics. You really have to understand the user story. But if you play the dance right and make it tell your company’s medium- and long-term story with numbers, I think you’ll be able to create more predictable success with your product and marketing.

Cody: So I have two different opinions on this matter.

One is that all the metrics that are related to the actual success of your product — user acquisition and growth — I think they’re interesting metrics. There’s another set of metrics that you probably are going to want to care about and that’s predictability and velocity.

Predictability and velocity are how much work are you releasing and how great you at releasing what you thought you were going to do. If you’re not measuring those two for the engineering work that’s coming out of your organization, your road-mapping is false. You’re just guessing at how much you’re actually going to get done.

So, predictability and velocity — super important. Think about how you are going to measure them, with engineering estimations or something similar.

On the other hand, I kind of hate a lot of metrics. Because I have been in more than a few early stage startups where metrics drove the organization a little crazy.

There’s a scenario where a lead investor comes into the room and says you must have X amount of revenue by this point. And what happens then? Your team is done. That’s all you care about and you’re totally demoralized. And that’s it.

And maybe they’re right to make that request, but up and down, you know? It’s a very challenging thing to deal with certain metrics and how you go about accomplishing them.

Moreover, I have seen the charm of metrics and analytics woo a lot of people into caring about things that don’t matter.

Andrew: Yeah, that’s important. And just to piggyback on that, when you’re building a model, build it as if having the data wasn’t an issue.

As if you could just snap your fingers and know any number whatsoever and then fill it in after the face. Because if you start with numbers first — “Let’s use what the cool kids are using. We’ve got Mixpanel, we’ve got GA, whatever.” Then you’ll be like, “Okay, great, what do we do with this?”

Then you’re going to have analysis paralysis and you’re probably digging into the wrong thing. But if you start with the model, “What do we actually care about? And what, if we knew it, would actually change our decision making?” Then fill that in with numbers, then you’re probably going to make much better decisions.

Cody: You know the viral YouTube video of the teacher packing a container with rocks, then stones, then sand, then water? Your metrics are like that. You need about four and then forget all the rest.

Laney: Just to add to that, picking which metrics to focus on is very important, and that’s something we tackled early on.

We have the traditional metrics, same as everyone else, on revenue, churn, new customers that have been acquired, etc. But as we look at our product, a conversational interface that schedules meetings via email, there’s not GA page views to look at for that. So we had to figure out what are the metrics by which we can compare one conversation in our product with another.

What makes one conversation better than another? And that’s a really hard question to get out of. It’s something we are constantly talking about and looking at — is this the right thing to be looking at? Because it’s easy if you focus on one metric. And if you do that, you can sometimes miss the long-term vision. So you have to make sure that the metrics you’re looking at are the right metrics.

Tom: Now, at X.ai, your team has focused on the amount of time it takes to schedule a meeting, is that right?

Laney: So there’s a handful of things that we look at. If I want to set up a meeting with you it might take us a few days to set that up. Maybe when I e-mail you, you’re in a meeting and it takes you a few hours to respond. Then I’m at a meeting. We all know that.

Whereas with Andrew and Amy, they’re a machine. So when you respond to them they respond back to you. So, what we look at and what we think about is, how many emails back and forth does it take for this? Because we all know that scheduling a meeting can be so painful. 15 e-mails in and you’re like, maybe we won’t meet for coffee after all.

And so, how can we make sure that those conversations are efficient and how can we balance all of the dimensions? So it’s these things that we’ve had to figure out how to measure and what are the different pieces of data or metrics that we can put together to have a view of theproduct. It’s hard to find just one number that does that.


Tom: The next question I have — there are many people in this room that are running a company or want to start a company. What advice do you have for them? For someone who wants to start a startup from just an idea, what would you tell them to avoid?

Cody: Don’t have a bad idea. Do not start that company. It seems like a joke but we interact with a lot of companies. We help people solve a lot of different sorts of organizational problems — engineering help, user experience and design, or a round of testing. There are so many different types of problems you can have. But there’s one problem that you can’t go back and fix it and that is that you started a company for something that nobody wanted.

Tom: Although you have a lot of YC companies, Reddit, Twitch, that started as something completely different from what they became.

Cody: One in a thousand. One in ten thousand. How many people have started that company and gave two years, which could be $600k and so much stress because you had an idea that wasn’t rooted in a problem.

Andrew: Try desperately to fail as hard as you can, as soon as you can, before writing a line of code. So try as hard as you can to disprove that this thing could ever work, as if you are hired to be the invalidator. And if you can’t then start the company.

Cody: I know 90 percent of you have read Learn Startup and you know who Eric Ries is and this is all stuff you’ve heard before. But people are still starting these companies.

That’s really risky. It’s not that I care. If the founder is willing to assume that risk then it’s fine. What really hurts my feelings is when you bring in four other people and you sold it to them and then they did it for six months and lost their jobs and didn’t get much out of it. Man, that sucks. I’ve been there.

Tom: It’s interesting. As an engineer, my first inclination is to code something right away. But one recent thing that I learned while working with a product manager from Huge was what he told us. “Don’t do mockups. Don’t do any code. Just make index cards.”

And apparently this is a thing — card sort. It’s brilliant. You take 20 of your features, put them in front of people, and ask them to sort what their top 5. We did this a month ago and it was more useful than anything else that we’ve done. So, as you said, before you code, before you do any designs, talk to users.

Cody: One set of really good product exercises is the New York Times Product Activity Guide. It explains a set of 20–30 exercises that product managers are generally responsible for. It includes card sort, personas, etc.

Laney: Once you’ve validated that you have a good idea, make sure that you keep talking to your customers. Keep a close relationship with them, because the customers you have in your first days are not the customers you will have a year later. You need to continue the conversations so that as you’re looking to build new features, you can have those conversations as you grow.

Keeping that relationship with the customer is so important, and that’s something that falls onto the product. You have to go in a couple levels, too, because someone might say that they have a problem. But there are 15 ways to solve that problem. It’s not until you dig in and discover, “Why is that painful for you?” “What is it about this?” Only then you might see the connections of one customer with another and figure out how we can solve that.


Tom: Last question. What is a product other than your own that you think is a really good example, and why?

Andrew: Google Spreadsheets. The last 7 years, there has been so many startups. I think the existence of Google Spreadsheets is remarkably undercited in this huge uptick in productivity.

What teams can do in terms of collaboration… how many teams here don’t use Google Spreadsheets? It’s so simple. They were classically Google in releasing it with so few features.

When it first came out, in 2007 or 2008, you were like, “Wait, this is a spreadsheet? But I can’t copy and paste!” or whatever. But they were so deliberate about adding only what was really necessary.

There are 10x more features that Excel can actually do, but Google Spreadsheets’ simplicity and convenience is amazing.

Cody: Earlier we were talking about Twitch. I’ve worked with eSports. I would pick Twitch — not because they’re worth about $1Bn.

What makes Twitch exceptional to me, is that it was a different video platform originally. And the users made it Twitch. They started doing something that video gamers used to do, watching someone play a video game. There was an entire generation of people who were doing that. And then they did it online.

Now, by almost themselves, they have built an entire industry in regards to eSports. That is a cultural shift that will have an impact of traditional sports and eSports and gaming and online content consumption. That’s happening right now, and those lines are going to keep diverging over the next few years. Super interesting.

Laney: Tough question, but I’ll give an example of something I won. I won an Echo Dot at work and kept it at home. And, it’s been a lot of fun playing around and testing the boundaries of it, and seeing how Alexa handles these weird questions that she gets asked. And there are handy things like when cooking, I need to set a timer, so I’ll call for Alexa to do it for me.