We recently launched the second wave of a study internally named, The New Account Creator Longitudinal Study or “Project NAC.” We’ve been following roughly 3,200 new users since September 2015, checking in periodically to ask:

  • How they plan to or use GitHub (purpose for signing up).
  • What their biggest challenge at the moment is (technical or social).
  • How they are/aren’t solving that challenge.
  • How and where they discover and learn about new developer tools.
  • What are the considerations that shape their Git/GitHub workflow.

A single picture vs. a baby book

This post discusses longitudinal studies, and we hope you’ll learn more about:

  • The difference between a longitudinal and cross-sectional study.
  • Why longitudinal studies are valuable and worth the investment.
  • How a longitudinal study is put together (why are there so few of them?!).

Newcomers & git for version control

What is a longitudinal study?

We tend to opt for cross-sectional studies to help us move quickly. Studying people over time is a special treat for researchers (my Halloween candy). If you’ve ever been sucked in by a “where are they now” magazine headline, you know the intuitive appeal of finding out how things turned out. The enduring popularity of the UK-based Up series, the profound insights from the Harvard Grant Study (following a cohort of 268 men for 75 years as they died and aged into their 90s), and the critical policy findings from the Panel Study on Income Dynamics demonstrate the cultural and scientific power of this type of study.

However, these studies require an investment of time, in which researchers are highly patient, deliberate, and willing to age alongside their subjects.

Understanding Cross-sectional vs. Longitudinal Studies

Cross-sectional studies are (relatively) straightforward: choose a population to study, create a survey instrument, select and recruit a representative sample, collect and analyze the data. An example of this would be our annual GitHub Tools & Workflows survey.

In contrast, longitudinal studies are complex and resource intensive: Go through all of the steps above, for each of multiple waves of data collection.

Among longitudinal studies, there are two general types:

- Cohort studies select a population based on some shared experience (e.g. birth year, exposure to a vaccine, etc), and repeatedly sample from this population to study how the experience impacts the subjects over time.

- Panel studies begin with a single sample and study that same sample over time at repeated intervals. This is a particularly resource intensive method, because the same subjects must be located again, contacted, and convinced to continue participating in the study. At each wave, the recontact rate is typically around 10%.

Bias

GitHub’s Project “NAC”

From our initial sample of 90,000 accounts, we received about 3,200 responses. Last week, we recontacted most of the initial sample. From the set of respondents to the first wave, between 12 and 19% participated in the second wave, as well as over 2,300 users who did not respond to the first wave.

With new users we’re looking at them in three phases for the next 12-months:

  1. The beginning (inception) — When newcomers sign up, poke around, & experiment. It’s harder to find them after they leave (we rely upon email outreach).
  2. The messy-but-sticky middle — When newcomers are regularly active; in GitHub where the workflows & workarounds happen (they imprint onto & are imprinted by the product experience).
  3. The end — Where newcomers have abandoned the product; GitHub “inactives.”

Longitudinal Studies are rare in our industry (both of apps and our efforts are babies ourselves), but they are essential if we want to assemble and learn about human lives –the effect that an application like GitHub has on skills advancement and career narratives for example.

This article was written as a partnership. Research is better when you get to discover new things about the world with someone who has a different perspective — someone who is willing to challenge you. I’m grateful to the people I worked with at GitHub. We were a small-but-mighty team.

UX Researcher. Today: Googler on Firebase. My exs: Airbnb (2018), GitHub (2016) & Mozilla (2012). Human interface to Paisley pug.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store