Improving the Way We Improve — Measures of Success

Below is an excerpt from Chapter 1 of the book Measures of Success: React Less, Lead Better, Improve More, by Mark Graban.

Most organizations are under pressure to perform better. How do we increase sales and production in our family-owned manufacturing company? Can we reduce infection rates in a hospital’s intensive care unit? Will we get our startup on the growth trajectory that we promised the venture capital firm?

People often feel like they’re on a proverbial performance roller coaster. There are a lot of ups and downs. There’s anticipation, excitement, and sometimes yelling — whether from excitement or fear. The emotional roller coaster of metrics and the way leaders react (or overreact) to them can be exhausting. This book is meant to help you get off the performance roller coaster, both stabilizing and systematically improving your results instead of ending up right back where you started.

Leaders might be under pressure to judge performance on a daily (or even hourly) basis. Color coding performance as “red is bad, green is good” can lead to a lot of overreaction, which then wastes the time of managers and their employees. Does any of this help us improve?

In an age of “big data,” we are too often drowning in numbers and information. Using the methods in this book, we can turn a flood of data into a controlled flow of knowledge and insight that allows us to evaluate performance better, focusing our efforts on sustainable improvement instead of knee-jerk reactions. When do our reactions, no matter how well intended, end up hampering improvement?

The desire (or need) to improve doesn’t mean that an organization knows how to do so. It doesn’t mean leaders know how to look at their performance measures in ways that determine if they are actually improving. Do they know how to look for meaningful changes in performance, or are they guessing or relying on gut feelings? Do they use rules of thumb, such as “you must investigate and explain every data point that’s below the target” or “you must find a root cause and give a corrective action plan for every below-average week?”

Leaders might ask the following questions:

  • Are we reaching our goals or targets?
  • What do we expect our future performance to be?
  • How do we know if a change has led to a meaningful improvement?
  • Can we discover if a system’s performance is degrading before it falls back into the red?
  • How often are leaders pressured to make metrics look good instead of improving the system and its underlying performance?

Measures Matter, but Don’t Forget About Managing

An expression that’s often shared on social media or in email signatures is:

“What gets measured, gets managed.”

A statement like this emphasizes the importance of measurement, but it’s frustratingly vague about how to manage or improve what is measured. Focusing on “what gets measured” is the reality in modern organizations. Measuring is easy; managing is hard.

Organizations often measure too many things, losing sight that the “K” in the common “KPI” acronym means “Key Performance Indicators,” not “Ka-jillion Performance Indicators.” More measures might mean more work — and possibly more overreaction — instead of more improvement.

When under pressure to improve metrics, leaders and employees will pay attention. They’ll talk about the metrics. They might assign somebody to be responsible for each one. They might form teams. That doesn’t mean they know the best way to manage the metric, and it doesn’t mean they know how to improve the system that generates those results.

Can we measure everything in life (or our workplace) that matters? No. As the late W. Edwards Deming said:

“The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them.”

The reality is that we have to do our best to choose measures that matter (or our boss tells us what to measure) — and then manage those measures in the best (or least dysfunctional) way possible.

What Are the Right Metrics?

Leaders are often told what to measure, causing organizational harm and dysfunction. Sometimes, we get the opportunity to choose metrics or have some input. Hopefully, we have metrics that matter instead of things that are easy to measure.

While there is a risk that the methods in this book could be used to “better manage” the wrong metrics, the focus of this book is to best manage and improve the metrics that we have. For a deeper treatment about what metrics to choose and how to set targets, books on the following methodologies could be helpful (see Appendix C):

  • Balanced Scorecard
  • Strategy Deployment
  • Lean Startup

A “balanced scorecard” of metrics helps protect against the dysfunctions that can result from focusing too much on any single metric. For example, if cost is the primary metric, managers might be pressured into actions that hurt safety, quality, or other important measures. Think of all of the problems caused by companies that focused solely on growth at any cost.

Strategy deployment (or “hoshin kanri,” in Japanese) is an important component of the “Lean” management approach. Lean organizations in various industries often use a balanced scorecard of metrics such as safety, quality, delivery, cost, and morale. These metrics should be relatable to people at all levels, so they feel like they can initiate improvements that contribute to their local metrics and the success of the organization as a whole.

From “Vanity Metrics” to “Actionable Metrics”

The “Lean Startup” movement asks important questions about what to measure. Eric Ries coined the phrase “vanity metrics” to describe our measures that “give the rosiest possible picture.” Instead of looking at the things that paint a picture of success, we should look at metrics that are truly the key performance indicators for our organization.

If we’re using metrics to evaluate the success of an improvement initiative, are we choosing metrics or manipulating them to tell a story of success, no matter what actually happened? Or, are we using metrics honestly to evaluate if we are getting better, getting worse, or if we’re in the awkward in-between state of having a metric that seems to be fluctuating?

Ries suggests we replace vanity metrics with what he calls “actionable metrics,” where “data must demonstrate clear cause and effect and be related to changes” to our product, our services, or our system. Otherwise, we’re just randomly trying a bunch of improvements, and that’s no way to run a business.

One classic example of a vanity metric is the number of visitors that come to a website. This number is easily measured, and we would hope to see a trend that increases, always going “up and to the right,” as entrepreneurs often say. But, higher website-traffic numbers might be meaningless if that does not translate into increased sales. What’s easy to measure isn’t always what’s meaningful to our business.

Other examples of vanity metrics might include the number of “Lean Six Sigma Green Belts” we have trained or certified or the number of improvements and projects that are completed. Those are easy to count, but it’s also too easy to lose focus on the measures that matter, such as quality, market share, and profit.

As Ries wrote in his second book, The Startup Way:

“The fact that your site has seen an uptick in visitors doesn’t mean your product is more popular or you’re more successful.”

KaiNexus, a startup I have worked with, does measure website traffic. But, more important metrics include profit, which is driven by revenue, which is driven by sales, which are converted from qualified leads, that often start as website visitors.

If KaiNexus lost sight of the real objective, they could publish really popular “clickbait” type articles that might attract a large audience. Instead, the company’s inbound marketing process is designed to attract people who are likely to pursue buying their type of software.

The Dangers of Arbitrary and Unachievable Targets

Where we see a metric, we usually see a target. That’s the reality of modern organizations. Leaders spend a lot of time debating whether this year’s target should be something really specific, such as the possibly insignificant difference between 34.17 and 34.634.

As an aside, some people use the words “goal” and “target” interchangeably. In this book, we’ll adopt the other convention that says a goal is a longer-term ideal objective, such as the goal of “zero preventable harm” in a hospital, while a target is shorter term and helps us gauge progress toward the ultimate goal.

I’ve seen too many cases where an organization didn’t hit their target one year, only to then set the same target the next. Or, they’ve optimistically set an even-higher target. The implication in most organizations seems to be, “If we choose the right metrics and set challenging targets, then improvement will happen.” If it were only so easy, everybody would be hitting their targets, whether that’s increasing sales in a startup or reducing falls in a hospital. This is why Deming would always ask an important and challenging question:

“By what method?”

It’s not enough to set targets and demand better results. Too many people believe that empowerment means setting aggressive targets and then leaving people alone to figure out how to meet them. Collaborating with staff doesn’t mean a leader is micromanaging. Working together to improve our systems and processes will lead to better results.

In the wrong kind of organizational culture, setting arbitrary targets can become very dysfunctional. One of Deming’s famous “14 points for management” reads:

“Eliminate slogans, exhortations, and targets for the workforce asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the workforce.”

I’ve worked in the type of culture that Deming warned about. So, I understand how an “us versus them” environment creates unbearable stress when leaders demand performance that’s unrealistic and then blame employees for not meeting that impossible standard.

There’s a difference between an arbitrary target and one that’s a “law of nature,” a term used by Donald J. Wheeler, Ph.D. to describe a target that has a scientific basis, such as a 60-minute target for the “door-to-needle” time for stroke patients to get treatment. An example of an arbitrary target might be “We need to increase sales by 25% this year.” A target is still arbitrary if it’s based on a competitive benchmark, last year’s performance, or an organizational target that has been passed down from senior leaders.

As Brian Joiner wrote in Fourth Generation Management, there are three things that can happen when people are pressured to hit a target without having the proper support and an effective improvement methodology. The first two are dysfunctional and are too often easier than the third, and preferred, alternative:

  1. Distort the numbers
  2. Distort the system
  3. Improve the system

We see many examples of this in the news, and we might see them in our own workplaces. In recent years, we’ve seen the CEO of Wells Fargo set an arbitrary target of “eight is great,” meaning each customer should have eight different accounts. Since the target was unreasonable, thousands of tellers signed customers up for accounts they didn’t need or didn’t know about — and then got fired for being “unethical.” Eventually, the CEO was forced out into retirement.

In the United States Veterans Health Administration, local clinic managers were put under pressure to keep waiting times for patients under 14 days. Even though the Congressional Budget Office called the target “unrealistic,” people in dozens of offices created secret waiting lists (a paper waiting list to get on the official waiting list in the computer) or other such distortions to make results look better than they were. Again, some employees and local managers were fired for what were arguably very systemic problems.

I recently heard a funny story about a fitness center that asked departing customers to push one of four buttons that rated their visit as one of the following:

  • a very smiley face
  • a somewhat smiley face
  • a somewhat frowny face
  • a very frowny face

The gym’s manager and staff were promised an incentive if a certain number of customers hit the “very smiley” button each month. They might have feared being punished for not hitting that target. What did the employees figure out? They learned that they could hit the “very smiley face” button a few times each time they walked nearby, which boosted that metric. Problem solved?

However, it’s possible to have a culture where a meaningful goal or target is shared by all. An organization like that tends to have supportive leaders and a spirit of “we’re all in it together.” Effective managers don’t just set targets; they work together with people to hit those targets.

Paul O’Neill, former CEO of Alcoa, set an audacious goal for an important measure: zero employee harm in their workplaces around the world. Such a goal could have been demoralizing if it seemed impossible, and people feared punishment. However, his leadership style made it clear that accountability started with him and that the company would work together toward that ideal, without blaming or punishing anyone for not reaching that ideal target. During his tenure as CEO, Alcoa reduced “lost workdays per injury per 100 workers” from 1.86 to 0.2 (a reduction of 89%). After O’Neill’s retirement, thanks to the culture and methodologies that he left in place, the rate fell to 0.125 by 2012.

See… not all metrics get dysfunctional. When we react less and lead better, we can improve more. Read more in the sample PDF chapter of Measures of Success, the book preview on Amazon… or buy the book! :-)