Lean Analytics — Ben Yoskovitz
Ben Yoskovitz, who some call one of the best product minds in the game. He’s going to introduce himself briefly, and then we’ll get right to the questions
Hi folks — I’m currently VP Product @ VarageSale, previously @ GoInstant (which was acquired by Salesforce). I ran a startup accelerator for ~1 year (called Year One Labs). Been an entrepreneur for 16ish years now.
Blog at http://www.instigatorblog.com/
And the book of course, which I co-wrote with Alistair is at:
Q. How did you develop your product chops? What are some of the biggest pieces of advice you give to aspiring PM’s? — Erik Torenberg
Lots of trial and error. ☺ I started a company back in 1996, learned a lot there. Then another one in 2007. So early stage CEO is also the product person (usually).
PMs can come from anywhere, which makes it interesting. I started as a startup founder and then slotted myself into the PM role. But PMs have a broad set of skills. They have to be both strategic and very tactical.
If you want to be a PM, I’d encourage you to go build something. Anything. On the side. Learn to code and hack away. Learn design.
Learn marketing. You’ll need all those skills in various degrees to be a successful PM.
Q. What would you say is one of the most important nuggets of wisdom regarding product you wish you could tell your 1996 self? — Jennifer Fox
Oh crap. Pretty much everything :simple_smile: First thing that comes to mind is Lean-based I think — don’t just build and build and build for the sake of building (or because customers want it all). Have a hypothesis, validate it with a minimal product, and then go from there.
Q. How important is the “ One Metric That Matters” to the entire lean process? Can you tell us a bit more about that? — Eric Willis
Sure — the One Metric That Matters is both theory and practice. By that I mean it’s meant to get people thinking about simplifying the number of metrics and the complexity of the metrics they track. People get buried in the data and overwhelmed. So that’s the theory part if you will, in the sense that you’re not obligated to track a SINGLE metric!
The practice part though — you should really try and pick a single metric at a single point in time and use that as the barometer for how things are going. It creates really intense focus — which is often a huge failing of PMs and startups (a lack of focus). If you don’t know what metric matters how do you know if anything you do will make a difference?
Q. What would you say is the most important thing in developing new metrics for new mediums? How are ideas best generated? — Jeff Needles
Great question. My first thought is this: What behaviors do you want to encourage? You have to understand those behaviors and the “loop of behavior” you’re trying to encourage (because you think it’ll create a real business) and that helps inform what metrics matter.
A good metric has to change your behavior as a company — so if you look at a metric and you don’t know what you’d do with it (as it changes) it’s not worth focusing on (at least too much)
Q. Who are the best product minds in the game right now? Who do you most admire ?— Erik Torenberg
HHhhmmm…product minds. Evan @ Snapchat is one. I honestly don’t get Snapchat (I’m too old I think) but I read an article recently about it and loved it. There are some real instincts there about a target audience that are important. Paul Adams @ Intercom is great too.
I follow Laura Klein a lot b/c she’s awesome. Focused on Lean + customer development. Super funny. Straight to the point (some might describe her as rude haha)
Brandon Reti (@brandonr182) at Breather is solid too. (Disclosure: I’m an investor in Breather). He doesn’t have a big following or anything of the sort, but I think Breather is a very nice product.
Q. What advice would you have for PH as we grow from your vantage point?— Erik Torenberg
PH is an interesting beast. It keeps growing, but I think many of us wonder where it will end up. I know we’ve had some private conversations around that. Scale + engagement are so tricky — you have to keep growing w/o sacrificing (too much) engagement. Then at some point you have to monetize.
Q. What do you do when you hear a great product idea that’s not being implemented? — Jeff Needles
I think almost every idea is being implemented right now, but hundreds of people. So my first reaction is, “Who else is already doing this?” But I do get easily excited about a whole bunch of ideas.
Q. What are the top 3 books you feel made you a better Product guru? — Jennifer Fox
4 Steps to the Epiphany blew my mind. I read it and said, “Holy fuck, I made every single one of the mistakes Steve Blank described.” Lean Startup by Eric was also huge for me, as was Running Lean. I think those are all pretty obvious choices though. Traction was a great book too.
Q. Are you learning anything new these days or is history repeating itself? — Jonathan Ma
I’d like to think I’m learning all the time. I don’t think you can survive in a startup without that mindset. Like most people I also like to think I know what I know, but that gets challenged regularly too. Having said that it’s important to look at history also, because there’s plenty of context there.
Q. I’m doing a bit of consulting and have been brought in to a couple larger 150+ people who throw the terms “lean” and “agile” around as buzz words, not really understanding what they are saying. What are your 2 favorite companies that have actually implemented successfully? — vbarnett323
I can’t really speak about companies I’ve worked with on the Lean front in terms of success stories (for disclosure reasons) but I think there are quite a few solid examples out there of companies genuinely putting efforts into place. There’s going to be a lot of “buzzword bingo” type nonsense in big companies re: Lean, there always is around pretty much everything in big companies.
Q. Many talk about having a true north when it comes to metrics but for staring startups, that is not easy to find. Any tips on this? — abe238
Most early stage companies (depends on how early we’re talking) need to focus on engagement as their #1 thing. So the metric they have to focus on is around DAU, WAU, MAU…something in there; something that tells you “Yes in fact, people are using this product, and at a frequency that’s important/material.”
Q. Favorite product idea that you think died too soon? — Jeff Needles
Oh man…I need more softball questions! ☺Honestly I find it more interesting how many ideas are now coming back that previously weren’t possible. Groceries online for example. Fact is, timing is so bloody important but almost impossible to manage. One of my startups that didn’t pan out (Standout Jobs) was in the recruitment space back in 2007–2010. Trying to build a recruiting company during a recession is a terrible idea.
Q. What new product areas do you see being opened up with the new emerging platforms like Docker, Wearables, etc.? — Michael Buckbee
That’s a 20,000 foot view type question! Truth is I think we’re nearing a point where almost anything can be built at any time (short of really hard stuff like teleportation). And since that’s the case the real question is, what SHOULD be built?
Q. What’s some advice around getting early traction that you think people should focus on? — Michael Buckbee
Focus on a tiny “string of behaviors” or “loop of behaviors” that you think matter; that you want to create, that you want to inject yourself into. So for example — I need people to do A, which leads to B, which results in them getting C, which encourages them to do A again. I know that sounds like it’s in the hypothetical, but if you can create that type of tiny loop in your product that’s what will get you early traction (or not; in which case you’re not solving an important enough problem)
That’s a bit more expansive if you want to take a look.
Q. What has been the book that has influenced you the most, not necessarily tech related? — vbarnett323
I’ve read a lot of Roald Dahl books to my kids. I don’t know if they’ve influenced me specifically, but it’s amazing when you read to your kids what they ask, how you see them think about it, the discussions it creates. That’s been extremely meaningful to me, if not influential.
Q. A product manager has to rally the entire company around their vision and ensure that other departments execute on their end to get there. Do you have any advice as to how to foster the type of leadership skills to accomplish this? — jenhau
I’m loud. Loud helps. I’m not sure you can really have quiet PMs. But I think it comes by leading from example (so doing as much as you are talking/planning) and also being extremely good at communicating high level vision and how that relates to the tactical. Everyone has to understand how the little thing they’re doing actually has an impact/is relevant to the vision.
Actually, @jenhau “loud” is a bit facetious; really it’s about being an excellent communicator. Over-communicating. I also think PMs have to be hyper responsive so everyone feels like they’re receiving the attention they need. A PM is there to un-block people.
haha yes, although the loud thing does help too, I find. ☺ Amen on the hyper-responsiveness. Sometimes I feel like the only person in the company that does not get to have a “Do Not Disturb” sign to get things done, but its essential for me to be available.
Agreed. I’m hyper-responsive, but I also force myself to block calendar time to actually build product. You can’t be exclusively in responsive mode to people’s inquiries.
Q. What product captures your attention the most these days? — Jonathan Ma
Interestingly, Timehop is one of them. After many years of posting on Facebook (and elsewhere), the stuff it’s surfacing starts to be really meaningful. I never shared anything out of Timehop until a few months ago.
I posted NewsGIF recently on PH too, which is news via animated GIFs. Silly, but there’s something there.
Q. Do you have a different approach to Enterprise vs Consumer product developments? — Michael Buckbee
There are some differences; I think that question warrants a lot more thought + a blog post (I’m writing notes for myself…hold on…)
I don’t think there are HUGE differences between the two. But in enterprise you have to be much more responsive to customers (assuming they’re paying), whereas with a consumer product you just can’t respond attentively to every single inquiry/comment. Now, that doesn’t mean you should build every single request from enterprise customers either, but it’s a different way of bringing in and interpreting feedback.
I think for consumer apps the “problem” is also harder to understand. It’s more nuanced b/c often consumer apps are solving “how should I spend the next 5 minutes while I’m sitting on the toilet”, whereas Enterprise apps are solving “real” problems around business processes, etc.
Q. is there’s anything as a PM that you purposely don’t balance? — Jonathan Ma
I think you always have to ask yourself when you’re building something what you are comfortable sacrificing/balancing and what you are not. Can you assume a bit more tech debt this go around? Can you refine the copy a bit later? Those are regular questions a PM has to ask. Same goes for whether you include a feature or not in a release. One thing I try very hard not to sacrifice/balance is whether something we’re doing fits within the brand / values / vision of the company.
Q. What’s your thoughts on whitelabeling a product? — Michael Buckbee
If you’re asking about whether I think it’s OK to sell a whitelabeled product, I think the answer is, “It depends.” It’s not my go-to strategy, because you lose a lot of control (potentially) of customers. You want to, particularly early on, have insanely close access to your customers/users. Anything that takes you away from that is a risk.
Whitelabeling could be a reasonable Scaling strategy though; it just wouldn’t be my first go-to market play if I could avoid it.
It also tends to lead towards losing product roadmap control; because feedback will come from a customer to your partner and your partner will instinctively say “Build X immediately!” And since you don’t have any data one way or another to tell you otherwise, you build X.
Q. Semi, related — any approaches you’d recommend for working in ecosystems? Like Varagesale works in ios/android etc. — Michael Buckbee
What do you consider an ecosystem? An AppStore? Something else?
Maybe Salesforce’s app store?
Well, my product only lives in PaaS App Stores currently (like Heroku, Azure, etc.) but yes, more similar to SalesForces
Got it. Generally I prefer products (if I’m thinking as an investor, for example) that have a direct to customer approach; again, because you have that direct link. You “own” the customer, which is extremely valuable. That can be true of things like Heroku too, but I think of those as distribution strategies, and in that case you have to measure the effort (to get into those distribution platforms) vs. the reward. So are you getting quality customers through X vs Y ecosystem? Some ecosystems have a fairly heavy price to get into, and may not lead to the results you want.
Well, the typical take from selling through them (like Heroku) is 30% off the top
Also, if they are in fact distribution channels, you should have already proven that your product is sticky and valuable to customers — then as you get into the later stages (in Lean Analytics parlance that would probably be “Virality”) you think about distribution, where these ecosystems are channels for you, just like any other.
It’s the big version of the App Store, where Apple takes 30% of the price or IAP
30% is fine, if the value of the customer is super high — LTV, etc.
So like any distribution channel you have to measure Cost vs. Reward.
In enterprise I would be looking at Lifetime Value, but also Time to Recoup Cost of Acquisition.
Some ecosystems are both a distribution / marketing channel and an ease-of-use play too of course. So if you can reach all of Heroku’s customers that’s marketing. If people come to you through other channels and then they can launch faster / get value from your product faster because you’re on Heroku that’s a different thing.
If you see a channel works well, the next step is to try and build a relationship with that channel. Network your way into Heroku for example and see if you can get some form of special treatment (distribution through email newsletters, etc.)
Disclosure: I used to work at Salesforce, which owns Heroku, but nothing I’m saying is specific to Heroku ☺
It’s like you are reading my to do list
Q. Did you see the Heroku Salesforce Connector? It exposes SFDC data as a Postgres database in Heroku and it’s awesome — Michael Buckbee
I’ve been out of Salesforce for awhile, so I’ve been paying less attention, but they are getting better at connecting all the things over there.
Q. Follow up question — any ideas what I could build with that since it’s now about a 100x easier to do integrations? — Michael Buckbee
Ha! I’ll have to leave the idea generation to you and others, I’m afraid. But if you love Heroku, then dig up Matt Creager (he’s an evangelist there) and tell him I said hi.
No matter how easy something is to build, you need to start with a problem; not a solution.
Hell, I can start asking questions too!
Has anyone read Lean Analytics? ☺ (I’m still surprised when people say YES)
It’s definitely on my reading list. I just started a new company and so, I’m getting used to the new schedule. — Jennifer Fox
Anyone have a good recommendation for referral software (that helps you launch referral programs) in mobile/native apps?
Sumo app I think is the name of it — vbarnett323
Not referral, but I think VarageSale could use one of those “Enter your phone number and we’ll text you the install link” to boost Web -> Mobile signups — Michael Buckbee
We’ve looked at it…we may do so again in the future. (Can’t really share too much on that front.)
What’s the 1 metric you’re tracking right now that you think is bogus and doesn’t matter?
How many metrics are on your main dashboard? (Go ahead, admit it…)
Too funny, at Disney we had so many and now I track client acquisition cost and monthly revenue — vbarnett323
You asked for metrics. Heres my redacted dashboard — Michael Buckbee
That doesn’t look too bad. My suggestion would be to figure out what problem you have right now that’s paramount to your business and go solve that. Figure out what metric tells you if you are, and focus, focus, focus.
We have this concept in the book that a metric is like a squeeze toy — you focus on one area (squeeze) and it tells you where the next metric is / problem is to focus on
Well, so my 1 true metric is number of platforms (like Heroku) that I’m on — as that has by far and away the biggest impact
Hhhm…ok. I’m not sure that makes sense (I know very little about your business though) but that number just keeps going up. So if you’re on 5 platforms, that doesn’t tell me anything about the business. When you add the 6th, you’re just tracking that. And there’s no guarantee (I don’t think) that new platforms will perform like old ones.
My feeling is it would probably be a Number / Platform — i.e. # of signups / platform so yo could measure the efficacy of each platform. But even that’s an assumption that more platforms is better, whereas optimizing performance in 1 may give you more bang for your buck.
Good metrics (as I preach here…) are usually ratios or rates and they have to change your behavior. How does the # of platforms you’re on change your behavior?
It’s https://www.expeditedssl.com — and you’re right. I can’t disclose numbers, but it’s just been outsize marketing return from being listed in platforms and deeply integrated than through any other form of marketing
I was looking at it from the risk side of things, if I’m just in one then it’s substantially more risk.
OK. So platforms may prove to be the best marketing channel, and # of platforms may be impactful for the business, but it’s not a measure of success per se.
That’s fair. But really if you’re at the stage with your company where you’re focused on Revenue, then the One Metric That Matters has to be revenue-related. MRR for example.
In the book we talk a lot about knowing what Stage you’re at. And you take the Stage + the type of business you’re in and that’s how you find the OMTM
I think that’s a fair recasting of it — and suspect that I may have tapped out how many signups I get from Heroku
Signups is usually not a great metric either — because it doesn’t tell you if the people who sign up are doing what you want them to be doing
And it always goes up (albeit at different rates) — so early on if all you care about is getting people through the front door, then signups / time can be meaningful.
I thought this was like my parents bookclub — we’d just show up and drink white wine and devolve into a Jenga tournament
I’m in a weird spot where 90%+ of my signups go from not knowing I existed to paying for ExpSSL monthly in a matter of a few minutes.
There’s no real trial other than setting it all up and then cancelling.
I don’t know if that’s weird or a great f-cking business model ☺Nothing wrong with charging people money.
“PH is an interesting beast. It keeps growing, but I think many of us wonder where it will end up. I know we’ve had some private conversations around that. Scale + engagement are so tricky — you have to keep growing w/o sacrificing (too much) engagement. Then at some point you have to monetize.” If you have a sec, talk more about that ☺ and what you’d think about if you were running PH — Erik Torenberg
Sure…so first you have an MVP and it’s super engaging (ideally — certainly PH was). But you need more users otherwise there’s no real business, so you start making efforts to scale. PH was invite only initially, then started opening up, then raised $$ and the PR hit. So now more people pile in. That’s awesome. But if engagement drops, or if new users don’t behave as well (engagement-wise) as old users then things can crumble.
So the challenge for social / consumer products is often how to grow (and grow quickly because you can’t monetize with small #s of users), without losing the engagement. Engagement will most certainly drop, it just has to. Churn will go up. But if engagement drops too much then the product loses its value to new users AND old users.
In a UCG (user generated content play) like PH as you grow you also have to worry about Quality — if quality tanks, engagement tanks, and who cares how many users you have.
Q. I think Quora in many ways is a precursor to PH. What might you suggest Quora do (or should have done differently)? — Michael Buckbee
I don’t have any intimate knowledge of Quora, but it had a problem of moving from early adopters to later adopters. Engagement rocked early on when it was small, people knew each other, tech celebrities were answering questions — but they had to grow. And I think that’s been a struggle; emerging out of the “tech bubble”. I still see a lot of Quora activity (based on the emails I get), although I rarely use it.
PH will have a similar challenge.
One approach is to go into other verticals (which we see PH dabbling in, e.g. games) but then the question becomes whether or not those verticals/communities want to engage in the same way. That’s a tough question.
I keep head nodding in agreement — Erik Torenberg
So for me, the big challenge is always GROWTH GROWTH GROWTH w/o killing engagement.
Turns out it’s freaking hard ☺
Btw, what this has made me realize is that there are multiple stages of Product/Market Fit. PH hit Product/Market fit very early on with a very small crowd. But as it grows there will be other Product/Market fits that it has to hit. It’s a continuum as opposed to a point in time.
That’s a great point Ben, we are trying games, then books and these are new PH market fits (secretly , ha), new product market tests
This has been a ton of fun — does anyone have any last questions before I call it a night? Plus I have a pile of requests from my team that I have to respond to! haha
That was awesome thank you for hanging out with us and thank you for you support throughout.
Absolutely. PH is great. And glad I could participate.
Lean Analytics is the missing piece of Lean Startup, with practical and detailed research, advice and guidance that can help you succeed faster in a startup or large organization.” Dan Martell, Founder Clarity.fm
The lessons in Lean Analytics apply to organizations of all shapes and sizes, from small business to big government.” Jennifer Pahlka,Founder & Exec. Director, Code for America
In Lean Analytics, Ben and Alistair have done a masterful job showing us how to use data and metrics to peer through the haze of uncertainty that surrounds creating new businesses and products. This book is a huge gift to our industry.” Zach Nies, Chief Technologist, Rally Software
The bad news is, there will always be people out there smarter than you. The good news is, Alistair and Ben are those guys. Using Lean Analytics will give you the edge you need.” Julien Smith, New York Times bestselling author of Trust Agents and The Flinch
Lean Analytics will take you from Minimum Viable Product to Maximally Valuable Product. It’s as useful for product managers at today’s multi-billion dollar companies as it is for entrepreneurs who aspire to build those of tomorrow.” John Stormer, Senior Director of New Products, salesforce.com
Entrepreneurs need their own reality distortion field to tilt at improbable windmills. But that delusion can be their undoing if they start lying to themselves. This book is the antidote. Alistair and Ben have written a much-needed dose of reality, and entrepreneurs who ignore this data-driven approach do so at their peril.” Brad Feld, Foundry Group Managing Director, TechStars co-founder
Available on Amazon
Marc Andreesen said “Markets that don’t exist don’t care how smart you are.” Whether you’re a startup founder trying to disrupt an industry, or an intrapreneur trying to provoke change from within, your biggest risk is building something nobody wants.
Lean Analytics can help. By measuring and analyzing as you grow, you can validate whether a problem is real, find the right customers, and decide what to build, how to monetize it, and how to spread the word. The core idea behind Lean Analytics is this: by knowing the kind of business you are, and the stage you’re at, you can track and optimize the One Metric That Matters to your startup right now.
Repeating this process helps you to overcome many of the risks inherent in early-stage companies or projects, avoid premature growth, and build atop a solid foundation of true needs, well-defined solutions, and satisfied customers.
