Goal-driven Product Management: How we use OKRs at Oscar

By Duncan Greenberg

As anyone who has ever worked in tech knows, setting the right goals is a painstaking process, and justifiably so: start-ups flourish or fail based on what they prioritize, and building your plans around a small set of well-crafted goals is the surest way to avoid wasteful investments.

Many years ago, Oscar adopted a goal-setting framework known as Objectives & Key Results, which was originally developed by Andy Grove at Intel, embraced by Google in its early years, and has now become widespread in tech. We’ve co-opted them into our own planning process we call our “Vitals.” In a nutshell, once a year our leadership team defines a set of company-wide goals with “objectives” (what are we trying to achieve?) and “key results” (how will we measure our progress or impact?). These are cascaded down to the different departments, and eventually to the individual product “pods.” At each layer of the org, the goals become more targeted and specific but ultimately ladder up to the company’s mission and business priorities. Teams refresh their Vitals every few months to account for new insights and initiatives.

If it sounds easy, it’s anything but. Goal-setting is really strategy-setting disguised as a mathematical exercise. It involves an articulation of your team’s purpose as well as a dissection of the company’s long-term aspirations into step-by-step achievements. The goals you choose will helpfully prune the tree of possibility but could send you down a fruitless path.

While much has been written about the theory of OKRs, teams often run into challenges when they put them into practice. So we’ve compiled answers to some of the most common questions we’ve wrestled with in trying to craft meaningful OKRs to move our mission forward.

What is an Objective, anyway?

The Objective is the thing you’re ultimately striving for. The headline that will greet your success. More specifically, it is a meaningful but achievable goal that your team can directly shape over a 3–12 month period that clearly ties to the priorities of the business. Objectives can be qualitative or quantitative.

Here are some Objectives from our Vitals over the past few years:

• Objective: Meet members where they are and reach our less digitally-engaged audiences by making SMS available as an option to 100% of members.
• Objective: Support Concierge in meeting our variable cost reduction goal of 10% through efficiency gains targeting 2% of the reduction.
• Objective: Mitigate major sources of member (and broker) friction in onboarding leading to a 50% reduction in negative pulse points

Each of these had a set of Key Results with metrics tied to work slated to launch during the OKR interval. You’ll notice that all of these happen to be quantifiable — we sometimes like them that way (they’re a bit more tangible) and it helps with nesting KRs as Objectives for sub-teams, but both kinds are absolutely okay!

How is an Objective different from a Key Result?

While objectives may be quantifiable, key results must be. Think of them as metrics with targets that let you determine whether your team’s work was successful in making progress toward the goal.

Aim for writing Key Results that are:

• Measurable. They can be expressed, say, in SQL query form. Little or no judgment is required to compute them.
• Meaningful. They significantly impact the business outcomes your team is focused on and of course feed into the Objective.
• Controllable. Your team can directly influence them and if your work is successful you expect them to improve.
• Intelligible. People outside of your team could read them and understand what they mean (OKRs are not just a tool for intra-team alignment, they’re a way of communicating priorities to other teams too)

We’ve found that OKRs are most effective when defined cross-functionally with product and stakeholders co-developing them. That doesn’t mean that all teams will play an equal part in each key result–some may be more product-driven, others more operationally led. The idea is to use OKRs as a tool for alignment from the get go and for negotiating prioritization conflicts throughout the year.

While not always possible, the best OKRs are formulated in such a way that completing all of the KRs means collectively you’ve achieved the Objective.

How do I evaluate my Key Results?

At Oscar, at the end of each OKR interval, we grade our Key Results on a scale from 0 to 1. If your goal was to increase sign-ups by 20% and instead they rose by 10%, in the simplest way of grading, your score would be a 0.5. Our Vitals grades are published for transparency and leaders are periodically asked to unpack them (good or bad) in our weekly All Hands forum.

My Key Results are kind of meh. What advice do you have?

A lot of goals flop because success was poorly defined from the start. Here are some of the tell-tale symptoms of iffy Key Results to watch out for:

• Too vague.

Example: “…feature is 20% more usable…” What does “usable” mean?

• Too meager.

Example: “…increase sign-ups by 0.2%…” If this is the most you think you can impact it, is it worth the investment? Sometimes tiny metrics like these are a sign that you should actually be targeting a particular part of a conversion funnel (and shrinking the denominator to a more appropriate size).

• Too fuzzy in terms of business impact.

Example: “…ship new check-out flow…” How can you be sure that the redesign will actually benefit users or the company? What are you really aiming to accomplish?

• Too noisy i.e. influenced by lots of other factors beyond your control.

Example: “…reduce company costs by…” Which company costs exactly? Be precise or your performance may be drowned out by other initiatives and you won’t know if your work made a difference

• Too far out i.e. they’ll take much more than a quarter (or a year) to realize.

Example: “…launch new product line by 2025…” Is there a milestone you can measure in the next 12 months? How will you know if you’re on track?

A few pro tips…

• Launch as many new features as possible as A/B tests (against the prior version). This will make the results you deliver unequivocal (no squinting at trendlines).
• Write the query to measure the KR before you get going instead of waiting until the end of the quarter only to realize that the data to measure it doesn’t exist. Good hygiene around instrumentation will ensure you always have the user data needed to set proper targets.
• Plot your KR metric over time before you settle on it to see how noisy it is
• Use KRs to capture research activities. For example: “…interview 5 users about…” or “…identify 3 opportunities to…”
• Steer clear of binary KRs (or at least use them sparingly!)

What if I can’t actually quantify my goal?

Like it or not, sometimes you’ve got goals that are binary (“…complete migration to Python 3…”). In these cases, consider setting a measurable quality bar that answers the question “how well did we execute on this project?” and “did we do it in a way that wasn’t disruptive to the business?” For example, “Migrate to Python 3 with 0 critical bugs.”

If your team builds internal tools for operational users, true A/B tests can be a challenge. Some user groups need to be trained on new features before they can use them. It’s usually possible to construct randomized groups of pilot users to roll a feature out to. Barring that, you can use before/after comparisons but this will only work for metrics that are fairly stable and you’ll need to pay close attention to seasonality.

Having a lot of “did I do it or not?” KRs could be evidence of bloat. OKRs are not a substitute for work plans or checklists. Do use them to highlight your most essential “build” work. Do not use them to goal-ify all of the day-to-day “run” work that must happen regardless. Fewer OKRs is better. If you must have many, try to assign priorities to the KRs or let the aggressiveness of the targets dictate which ones will command your attention throughout the year.

OKRs in action!

Suppose your objective is to “Launch a digitally native Diabetes program that measurably reduces medical costs,” which key result(s) would you choose?

1. 30% of eligible members adopt the program

• A good measure of top/middle of the funnel performance, which for a new program may be the only thing that’s readily measurable. But is this something the Product team will own or will adoption be a marketing-driven effort? You may want to define an additional KR that product can rally around.

2. Members enrolled in the program have an NPS >50

• Obviously user satisfaction is an important metric, but given the focus on cost reduction, is this the main thing you’re optimizing for? Also, this metric, as written, could be confounded by lots of variables outside of your team’s control (e.g. the bedside manner of the diabetes coaches who support members throughout the program). Perhaps it can be teased apart? Or framed in terms of retention in the program?

3. Reduce Medical Costs by 1% on an annualized basis for enrollees in the program

• This one is ideal in that it’s directly tied to the business value invoked in the objective, but it may not be realistic to achieve (or for that matter, measure) within the OKR interval and it will be difficult to attribute to a single team’s work. Break it down further!

4. Members enrolled in the program see a 10% improvement in their A1Cs

• This is an example of an intermediate metric or leading indicator that may help bridge the gap between user behavior and business outcomes when the latter is too hard to directly tie to your work or will take too long to materialize.

The bottom-line

Which KR you pick will depend on what you’re optimizing for, the opportunity you’re targeting, the role your team is playing, the stage of investment in the initiative (pre- or post launch), and how long you have to wait before evaluating its impact. In fact, you’ll definitely need multiple (though ideally not a multitude) to cover all of the relevant dimensions of the work, especially if you take our advice and team up with stakeholders to write them. The important thing to ask yourself is: at the end of the OKR interval, if you achieve all of your key results, will it fulfill the objective? To claim success, you need to convince yourself of that. And you need to feel confident that the objective was achieved because of the work you did, not incidentally.

Once you’ve got your goals, hold yourself accountable for them. Schedule a monthly OKR review with your immediate team or with partners from the business. Cite them in your monthly team communications to stakeholders. Don’t just set them and file them away.

No goal is perfect, but if you invest the time to make rigorous goal-setting a mandatory part of your process, you will find that it makes prioritization easier. It will focus your energy on product areas and use cases with outsized opportunity, and fend off the kind of project-driven product management that so often results in a race to irrelevance.

Duncan Greenberg is the VP of Product, Care & Coverage Experience at Oscar where he oversees the product teams responsible for Oscar’s digital member experience, customer service technology, virtual care platform, and automation engine.

Want to talk more tech? Send our CEO, Mario, a tweet @mariots