Measuring Hard-to-Measure Things

Chrissie Brodigan
Jun 23, 2015 · 6 min read

What? is not the same thing as Why? You need both, but why is usually two steps ahead of What?

In June 2015, I spoke at Monitorama, and it was one of the best experiences of my career. The community was accepting, supportive, and safe. Thank you for the warmest welcome, truly. Jason Dixon, you are the best! The following post provides a narrative to accompany slides and video.

Slippery Slope

I joined GitHub in March 2013, excited and nervous to be starting as the company’s first user experience researcher. Professionally, I’m trained as a historian who turned towards design to fund an academic habit. Not being an engineer researching engineers felt intimidating, but it turns out there’s plenty of overlap between disciplines like: curiosity, skepticism, and a compulsion for discovery.

We make informed design decisions by using the product we build on a daily basis. We keep what works, remove what doesn’t and improve what needs to be improved. I think formalizing that process and having a “UX Researcher” is a slippery slope.

Since I’d been hired shortly after this post, it was clear that we were on a slippery slope. However, my colleague’s insights were both honest and accurate, and GitHub had been wildly successful without user research.

GitHub was powered-by luck …

Without research all you have is luck. –@sboak

Early product success is exciting and terrifying, because if the growth path is up-and-to-the-right your audience will broaden as will your challenges. Newcomers will arrive with experiences less like your earliest users and less like your own, and may begin using your product in ways you didn’t intend. This may leave you puzzled, scratching your head.

Who are these people? What do they want to achieve? Why do their goals matter? How do we help them?

In GitHub’s case, there was a distinct shift in growth and audience. By 2013 developers experienced with version control and git signing up for accounts were far outnumbered by newcomers with limited experience, hungry and expecting to learn both. The caché of a GitHub profile and membership to its community brought many types of people through the door and into a product that is frankly pretty hard to figure out.

Image for post
Image for post

Why would you request someone “pull” in your changes? What would a person think if you tried to change their project?

As your audience broadens early luck can start to run out.

Blind Spots

All products have blind spots, and no amount of analytics and monitoring can account for the hidden variables “waiting to screw you.” Even the best systems can gather data on unusual behaviors for a long time without signaling that something is amiss. The sheer volume of data we have now can make it hard for talented analysts to utilize information, and powerful tools and visualizations are rendered less useful.

Gap from metric/graph to insight can be huge. –@randommood

Graphs are impressive and addictive. They depict what is happening (e.g. year-over-year growth, anomaly detection, etc.). However, in isolation, graphs can also be misleading and even create a sense of overconfidence in data. No amount of gorgeous graphing or intricate analysis can make up for fundamentally flawed data.

What v. Why

What? is not the same thing as Why? You need both.

My rant on WhyWhy is hard:

  • Why is time-consuming and awkward.
  • Why can’t be automated.
  • Why requires you look at the world through someone else’s eyes.
  • Why breaks our product design egos down with radical amounts of empathy.
  • Why shows us what we don’t know and what we didn’t want to see.
  • Why confounds and humbles us and is usually two-steps ahead of what.
  • Why can sink a product or push it over the mountain into success.

Watching someone use your product will change the way you see your product.

You are a Human Instrument

With regards to engineering challenges, data collection, and analysis we tend to think in terms of technical instruments. If you want to surface blind spots, you will have to get out from behind the numbers and in front of people. Think of yourself as a human instrument. You are handcrafted, possibly artisanal, and definitely brilliantly designed to measure socially meaningful questions with an open and compassionate heart.

  1. Motivations — People are driven towards achievement, they have underlying reasons for behaving in particular ways. People want to succeed.
  2. Workarounds — When goals and motivations are out of sync with a product’s user experience, people can be very effective at using bad design.

Vocal Minority

When you’re designing a product, especially one that needs to scale for thousands and millions of people, you must be increasingly deliberate in your design decisions. If you introduce complexity, you must also consider its effects on both current and future behaviors; it’s hard to take features away.

There is always a vocal minority.

Listening closely and reacting to insights from the wrong audience might encourage you to head in the wrong direction. Sometimes the vocal minority are not the best representation of your customers, they just happen to be the ones generating the most feedback and are often easier to reach.

Guarantee

At a time when we are awash in data, if you think in terms of human behaviors and get out from behind the numbers to talk with people, I guarantee you will become better at discovering, measuring, and solving hard problems.


Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store