Measure What Really Matters
The Good, the Bad and the Ugly of John Doerr’s OKR Book
Update: I have included sections about Christina Wodtke's Radical Focus and also the issues with quantitative objectives. Both were part of the original draft but somehow got cut off in the editing process.
This post was originally published at felipecastro.com.
John Doerr’s book on OKR, Measure What Matters, arrived this week with a big launch that included a TED talk, the opening bell at Nasdaq, and praise from Bill Gates, Jim Collins, and Sheryl Sandberg. Doerr, the “Michael Jordan of venture capital,” is one of my personal heroes. He was an original investor and board member at Google and Amazon, and I make a point of sitting in the front row every time I hear him speak, as you can see from this photo.
I always quote Doerr during my OKR workshops, so my expectations for the book were quite high. I had pre-ordered the book a year ago, and I jumped right in when it arrived. Many people expected that Doerr would write the definitive book on OKR, but unfortunately, Measure What Matters fails to deliver on that promise.
Let me say that, for the most part, the book is very good. Doerr’s talent shines through when covering OKR both through theory and real stories. I could have called this review “the excellent, the bad, and the ugly,” but, that is not as catchy.
A Dash of Salt
The devil is in the details, and while Measure What Matters does a fantastic job of presenting the philosophy behind OKR, it has serious flaws when it comes to translating theory into practice. One of the things that I learned after training thousands of individuals in OKR is that you have to be careful with what you write, because readers may take it literally. Some people will quote Measure What Matters as the holy scriptures of OKR, and instead of focusing on the principles behind it, they will blindly copy its examples, no matter how bad they are. I hope Doerr can fix those flaws in a new version, but as it stands, the first edition of the book has to be taken with a few dashes of salt.
After reading the book, I found myself in the awkward situation of having to criticise the OKR pioneer. But as someone to teaches OKR for a living, it is my job to both recommend the book and to provide that side order of salt. I hope to add an “OKR from the trenches” view to Doerr’s “OKR from the Silicon Valley boardroom” perspective. You may think I sound arrogant by going against the teachings of the OKR prophet. If that is the case, I ask you to give me a better argument than “But Doerr said so.” Seek to understand the principles behind OKR instead of blindly following authority.
Without further ado, here is my review of the Good, the Bad, and the Ugly of Measure What Matters.
The Good
Measure What Matters unveils lots of new content on OKR starting with tales from its origins with Andy Grove at Intel. It also includes first-person stories from the CEOs of Google and YouTube and the playbook that Google uses to train its employees. Only Doerr could deliver this breadth of content, and he does it brilliantly. He also reinforces the importance of having the right culture and the need for leadership buy-in. “In implementing OKRs, leaders must publicly commit to their Objectives and stay steadfast,” Doerr writes.
Doerr does an incredible job introducing OKR’s philosophy while at the same time addressing some of the most common questions.
From Cascading to Aligning
Doerr attacks the traditional approach where “goals were handed down the org chart like tablets from Mount Sinai,” explaining the adverse effects of cascading, including loss of agility and flexibility. In doing that, he updates the football example from his first OKR presentation to Google in 1999 which is more important than it seems at first. In 2013, Google Venture's Rick Klau decided to include the football example in his OKR video, and since then, it became the perfect excuse for those who want a top-down management approach.
Measure What Matters reinforces the need for bidirectional goal setting, and states that “high-functioning teams thrive on a creative tension between top-down and bottom-up goal setting,” based on dual cadence OKRs — annual for the company and quarterly for the teams.
While cascading requires goals at every hierarchical level, OKR enables a different approach. Doerr explains that employees can align their OKRs directly to the company OKRs, skipping “multiple levels of hierarchy.” This practice makes the OKR process lighter and faster while increasing alignment and engagement.
The book reinforces that leaders should no longer dictate the work through a top-down cascade. Instead, they “set the context, ask the big questions, and furnish relevant data,” giving employees the autonomy to innovate.
Stretch Goals
Setting ambitious goals is, at the same time, one of OKRs most essential tenets and the least understood. Doerr introduces two OKR variants: Aspirational and Committed. Aspirational OKRs are higher-risk, and teams aim to reach only 60–70% of the targets in aggregate. On the other hand, Committed OKRs are more predictable, and teams aim to achieve them in full. In my Beginner's Guide to OKR, I use the terms Moonshots and Roofshots, but Doerr’s taxonomy is more intuitive, and I have been using it in my workshops.
In the book, Susan Wojcicki, YouTube’s CEO, explains what stretch goals are about:
While Google aims for a grade of 0.7 (or 70 percent attainment) on stretch goals in aggregate, and there are times people totally fail, no team goes into an OKR saying, “Let’s settle for 70 percent and call it a success.” Everyone tries to get to 100 percent, especially once an Objective seems within reach.
The book also explains that you don’t have to start with very ambitious goals and that you can evolve with time:
In my view, it’s better for leaders to set at least a modest stretch. Over time, as teams and individuals gain experience with OKRs, their Key Results will become more precise and more aggressive.
Tracking and Adapting
One of the things that I love the most about the book is that it reinforces the importance of the OKR check-in, thus avoiding the “Set it and Forget it” mindset:
Without frequent status updates, goals slide into irrelevance… At quarter’s end (or worse, year’s end), we’re left with zombie OKRs, on-paper whats and hows devoid of life or meaning.
I also love that Doerr explains that "OKRs are adaptable by nature," and that you can modify them during the quarter if needed.
Compensation
Doerr dedicates the second half of the book to continuous performance management and its relationship with OKR. In it, Doerr talks about the "amicable divorce" — I loved the term — between OKRs and compensation, reinforcing another crucial tenet of OKR:
Divorce compensation (both raises and bonuses) from OKRs. These should be two distinct conversations, with their own cadences and calendars.
I wish he delved deeper into that, as this is a recurring topic.
Doerr talks about reinventing the performance review with CFRs (Conversation, Feedback, Recognition). It is an interesting framework, although it seems less mature than OKR.
As you can see, Measure What Matters has lots of good stuff. Now, for the bad part.
The Bad
The book can be inconsistent and confusing. Doerr seems to contradict himself a few times, and I can see myself saying "what he meant here was…" for years to come.
Shared OKRs
Although the book mentions the need for cross-functional collaboration and the use of cross-team OKRs, in different sections it is apparently against the use of shared OKRs, a powerful tool to create alignment that many Googlers use.
Doerr states that OKRs "create networks" but at the same time the book includes a quote from Mike Lee, MyFitnessPal CEO, stating that "co-ownership weakens accountability." Lee seems to believe that all work is done by individuals, while modern organizations are now networks, filled with people working in team-based environments — which require shared OKRs. We don’t want a product team where the designers, product managers, and engineers are all going in different directions.
High-performance teams have shared accountability over the outcomes they want to achieve. They create healthy peer-pressure and do not accept any team members that are not committed to it. If you listen to Mike Lee, you may never create a real team — only a bunch of individuals.
The only tool mentioned in the book to create cross-functional alignment is to get the department heads in the same room to identify dependencies, which of course does not scale. Alignment at the C-Suite is not enough.
OKR Rollout
In one section the book states that "the team deploying it — whether a group of top executives or an entire organization — must adopt it universally," while in another section it recommends an OKR pilot. Should the reader adopt a universal rollout — which is a terrible idea — or use a pilot?
I believe Doerr meant that after a team or unit has adopted OKR, there should be no exceptions. Adopting an incremental rollout with a good idea but the teams that have already adopted OKR must commit to using the same language.
The Ugly
There is a question I always ask on my OKR workshops: If you deliver all your tasks and nothing improves, are you still successful?
Of course not. To measure what really matters, we need to focus on the outcomes we want to achieve.
Ironically, Doerr’s examples do not focus on measuring what matters. Instead, they present OKR as a project management tool mostly used to track deliverables. Although Doerr quotes Marissa Mayer, (“It’s not a Key Result unless it has a number”) the majority of the examples included in the book fail to meet that simple test.
Ugly Examples
The examples are where things get ugly. Out of the 60 Key Results listed, 32 (53%) lack numbers. They include things such as "Create a retirement plan for all legacy technology," and "Focus on hiring A player managers/leaders." Even Doerr's own OKRs from his days at Intel include Key Results as "Develop a Demo."
Good Key Results measure outcomes: the value and the benefits you deliver to your customers or your company. In modern OKR we call the examples above initiatives: the deliverables and tasks associated with achieving your OKRs. Initiatives are usually binary and thus fail Marissa's test.
"Launch features to enable instructors to create more engaging videos" is an initiative — and a bad one. A good Key Result would be “Increase video watch time from X to Y.”
How about the other Key Results listed as examples? 18 of them are counting initiatives, measuring the volume of tasks instead of the business outcome. For example, "Deliver five benchmarks." Where is the value?
Of the 60 Key Results listed as examples, only nine measure outcomes including “Reach 80M total registered users.” The remaining example includes both an initiative and a Key Result: “Launch YouTube VR experience and grow VR catalog from X to Y videos.”
Outcomes Thinking
A major benefit of OKR is moving from project management to outcomes thinking. It is a significant mindset shift, but every company can make it. Outcomes thinking is crucial to any organization that is serious about software or digital transformation. As Ron Kohavi and Stefan Thomke wrote for the Harvard Business Review, most ideas (and thus projects) fail:
The vast majority of [ideas] fail in experiments, and even experts often misjudge which ones will pay off. At Google and Bing, only about 10% to 20% of experiments generate positive results. At Microsoft as a whole, one-third prove effective, one-third have neutral results, and one-third have negative results.
What amazes me is that although Doerr lists my friend and colleague Christina Wodtke’s Radical Focus as an OKR resource, he does not seem to have heard her message. Christina makes a clear point that Key Results need to be about results, not tasks.
If you are still not convinced, several authors outside of the OKR community have written about the need to focus on outcomes, including Marty Cagan, Jeff Gothelf and Josh Seiden, Jeff Patton, and Barry O’Reilly.
Before Measure What Matters, the Google Ventures OKR video was by far the most accessed OKR material in the world. And it also included bad examples of Key Results, such as "Launch a Monetizing Tab." Luckily, Rick Klau tweeted about what he learned after the video, including the need to focus on outcomes:
I would like to thank Klau for clearing this issue.
Quantitative Objectives
This next issue may be a minor one, but it can lead to bad behavior: three of the example Objectives included in the book are quantitative, including “Reach 1 billion hours of watch time per day.”
We have to be careful not to mix Objectives and Key Results. Objectives are qualitative themes that give purpose and meaning to that set of Key Results. Quantitative Objectives are confusing and tend to make people focus on the how: the initiatives they are going to do to achieve them. Qualitative Objectives may also lead to top-down OKRs, as a manager may take one of his Key Results and cascade it as the Objective for a direct report.
Hidden Outcomes?
Doerr, Grove, and Google are famously metrics-driven. So how did we ended up with such bad examples?
Some of the stories mention metrics, even though their OKRs do not reflect that. They are hiding the outcomes. Maybe Doerr assumes that companies are already evidence-based and are all measuring outcomes. This may be the norm with his portfolio companies, but that is far from the truth for traditional organizations. The examples are misleading and, unfortunately, the readers that follow them will be setting themselves up for failure. Adopting OKR is an entirely different ballgame when you are not an agile, data-driven Silicon Valley startup.
A second cause is that some companies mentioned appear to be stuck in an IT mindset. The stories about Remind and Intuit revolve around which features to deliver, instead of the outcomes you want to achieve. They seem to treat teams as mere feature factories with no focus on delivering value.
Key Takeaways
Measure What Matters is a must-read for anyone who is serious about learning OKR. Unfortunately, the book leaves readers in the strange situation of having to ditch almost all of the examples presented in it.
In the past months, I have been working on my upcoming book by Sense and Respond Press, titled Goals for Agile Organizations: Scaling OKR from Teams to the Enterprise. I have learned that having the right editors can drastically improve a book. Vicky, Josh, and Jeff would never let me get away with inconsistencies such as the ones presented in Measure What Matters. I hope that Doerr decides to publish a second edition that addresses the issues above, maybe with a different editing approach. The book would probably benefit from beta readers, as the OKR community would poke holes on such bad examples.
As this quote from Measure What Matters states, adopting OKR is a journey. Maybe writing a book about works the same way:
“You’re not going to get the system just right the first time around. It’s not going to be perfect the second or third time, either. But don’t get discouraged. Persevere. You need to adapt it and make it your own.”
Thanks to Vicky Olsen and Alberto Caeiro for reading drafts of this article.
Did you enjoy reading? Please clap below. It’ll let me know you’d like to read more articles like this and will help people find it. You may also follow me on Twitter @meetfelipe or visit my website: felipecastro.com.
