What’s Obvious vs. What’s Interesting
With research we’re always on the lookout for patterns–some are obvious, but the ones we care most about are interesting. However, obvious and interesting don’t usually fall into the same group. This post will cover an obvious insight, an interesting insight, and a technique to help you rethink your graphs.
An obvious insight
In 2013, we conducted 20 field interviews with new users and realized that there was an increasingly gap in newcomer git knowledge–on the qualitative side it looked like more people were signing up for GitHub without much or any git knowledge. In 2014, we used field insights to design a short survey instrument to ask new users signing up from within the United States to rate their familiarity with git on a 3-point scale:
This visualization presented our teams with an obvious takeaway — 76% of our newest users signing up from within the United States were brand new to git. Holy cow! We had a clear signal. However, the longer we considered what people didn’t know (obvious), the more it became clear that we needed to learn what people did know (interesting). This was the inspiration that drove our development of GitHub’s annual Tools & Workflows survey.
An interesting insight
Below is a stacked bar chart taken from the GitHub 365 survey, which depicts a relationship between categorical variables. Note: Both the colors and order are arbitrary, the only thing that matters is the relative surface area that each color holds.
We can see a very strong pattern with the yellows and greens, which represent “nothing” (no prior experience with a version control system) and Subversion (SVN). People who code very little or not at all are generally not using any other VCS. However, what’s less obvious and more interesting is that as programming experience increases, people become more likely to be using a VCS that’s not GitHub.
Cover up the obvious insight
Thinking back to the first graph in this post that depicted peoples’ experience with git, we refined our approach to ask a more elegant question, which produced more accurate insights in subsequent research efforts:
“What is your greatest challenge getting started with GitHub?”
What’s obvious is that learning git is the greatest challenge for a global audience that includes both tenured (3+ years) and new (less than four months) users. But, cover up the most obvious insight on the visualization and look for what’s interesting.
What we see is that there’s an interesting shift from the second most frequent challenge, “Fear of making a mistake in front of an audience” among tenured users (17%) to, among new users, “learning programming skills” (19%). This insight supports a thesis statement we have that over the past three years people are increasingly coming to GitHub for practical reasons like skill-building and code hosting, rather than in search of the open source community. Wow, that’s really interesting.
—
Research is better when you get to discover new things about the world with someone who has a different perspective — someone who is willing to challenge you.
