“No causation without manipulation.” –Paul Holland

At GitHub, we frequently use experimental research designs to study the effects of new features and models. This post describes when and how we use experiments, and why they are such a powerful tool.

tl;dr: experimental designs are the most efficient means of establishing causal relationships, but causal inference is hard!

To discuss randomized controlled experiments, in this post, we’ll use the example of the Golden Ticket experiment. …

There’s an infamous story and that often comes up in product debates to help justify not listening or reacting to customer requests. It’s inaccurately attributed to Henry Ford:

“If I had asked people what they wanted, they would have said faster horses.”

Customer requests are an important input in product development and the role of research is to listen to and surface the how, why, when, and where, behind them. We’re looking for patterns, use cases, and the human motivations and goals driving the conversation.

Customer Request: We need Branch Permissions

At GitHub, in 2015, customers told us they needed a complex feature called “branch permissions.”

Muir Beach Overlook, which I overlooked for four years

The past few days I’ve been thinking a lot about what a team of two was able to accomplish in three years at GitHub, spanning some pretty difficult times for the company. It’s a pleasure to look back and even to see what I overlooked and some of the challenges I personally faced.

What follows are 10 lessons I learned while building GitHub’s early research practice. To be sure, I didn’t practice all 10 perfectly all of the time, in fact, some lessons are aspirational. In spite of some bumbling, I also experienced an incredible amount of learning.

Truth of table stakes

The one…

We recently launched the second wave of a study internally named, The New Account Creator Longitudinal Study or “Project NAC.” We’ve been following roughly 3,200 new users since September 2015, checking in periodically to ask:

  • How they plan to or use GitHub (purpose for signing up).
  • What their biggest challenge at the moment is (technical or social).
  • How they are/aren’t solving that challenge.
  • How and where they discover and learn about new developer tools.
  • What are the considerations that shape their Git/GitHub workflow.

A single picture vs. a baby book

Longitudinal studies are complex and, at times, difficult to complete. A person in the cohort may respond…

I truly hate using word clouds to present open text data, which is a research lovers pot of gold. Word clouds –well, I feel like they cheapen the richness and elegance of open text. However, every once in a while, a project comes along where word clouds make just enough sense to use as a visualization tool –they are especially accessible for diverse audiences across an organization.

As part of the annual GitHub Tools & Workflows survey we asked an optional open-ended question at the end with a single open-text field:

How would you describe GitHub in three words?


“However small the chance might be of striking lucky, the chance was there.”
Roald Dahl, Charlie and the Chocolate Factory

GitHub.com’s business model charges for private repositories with two types of plans –personal and organization. In 2015, we launched a randomized controlled experiment lovingly named “The Golden Ticket,” giving coupons for free private repositories to 39,800 people.

What follows recounts some of the high-level insights we drew from this experiment that might help you research your product’s value or price.


Experiment design

We used a classic experimental design with 40k tenured users on free accounts assigned randomly to treatment and control…

With research we’re always on the lookout for patterns–some are obvious, but the ones we care most about are interesting. However, obvious and interesting don’t usually fall into the same group. This post will cover an obvious insight, an interesting insight, and a technique to help you rethink your graphs.

An obvious insight

In 2013, we conducted 20 field interviews with new users and realized that there was an increasingly gap in newcomer git knowledge–on the qualitative side it looked like more people were signing up for GitHub without much or any git knowledge. …

GitHub began as a tool for hobbyists and evolved with organizations and on-premise enterprise products into a tool for professionals.

We’re going to share 9 key insights from GitHub’s annual Tools & Workflows survey, welcoming debate about “learn git v. less git,” text editor and IDE popularity, and shining a light onto some blind spots.

One finding, in particular, “Professionals solve their problems with support from human peers” surfaces a critical discovery about human behavior: the distinguishing factor in solving challenges getting started with Git and GitHub is that professionals have greater access to people who can help them solve…

Photo by Nemo, San Diego 2015

Most web services experience seasonal lift in new user sign-ups and traffic. This is especially true in January when people return to work and school, and resolve to make changes to behaviors. Specifically, many new users arrive to apps with their highest potential to learn something, do something. However, new users readiness is often quickly lost when your product’s experience doesn’t present them a compelling path forward.

RealTalk™ … Over the past three years as our newest users arrived to GitHub, who were different than our early adopters, we failed to engage the majority and lost them.

This large leak…

Q: If we could have done one thing to improve your experience what would it be?

End-of-year (EOY) is an excellent time for a retrospective on your app’s growth –specifically looking at where you didn’t grow. At GitHub we conduct a short annual survey, dubbed “The GitHub 365.” We use the 365 to examine what happened with people who signed up for an account, but at some point ceased to return.

Let’s talk about your product:

  • Who are your newest users that don’t come back?
  • Why did they leave?
  • What’s one thing you could have done differently to help them…

Chrissie Brodigan

UX Researcher. Today: Googler on Firebase. My exs: Airbnb (2018), GitHub (2016) & Mozilla (2012). Human interface to Paisley pug.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store