Photo by Nemo, San Diego 2015

Most web services experience seasonal lift in new user sign-ups and traffic. This is especially true in January when people return to work and school, and resolve to make changes to behaviors. Specifically, many new users arrive to apps with their highest potential to learn something, do something. However, new users readiness is often quickly lost when your product’s experience doesn’t present them a compelling path forward.

Made by my brilliant colleagues at GitHub (Bill, Pam, Liz, and the comms team)

What’s hard to measure

Dashboards and metrics power many decisions in product development. I find these tools to be impressive and useful, they tell us some things, but I can almost never get what I need to know about our users, as humans, by looking at graphs alone. Ines Sombra eloquently shared a memorable statement in her Monitorama 2015 talk:

The gap from metric/graph to insight can be huge.

Example: Do you know what’s going on here?

  • Motivations — Drivers that power people towards accomplishing tasks and reaching goals.
  • Workarounds — Patterns people create to accomplish tasks that the current design doesn’t surface.
  • Behaviors – Core (current/engrained habits), Future (incremental workflow changes with new design to accomplish goals), Aspirational (leaps into the future).
  • Emotions – How people feel about a product and its user experience. Some companies measure this with Net Promoter Score (NPS).
Learning to surf, by Nemo, San Diego 2015

New User Journeys & Demographics

Our research approach to studying new users and their journeys is backed by three survey instruments listed individually below. The approach is portable, so you can recreate it almost anywhere. In addition to required questions about product experience, we’ve slowly added optional personal demographic questions (human age, education, sex/gender). When you’re working on a growing app/community that has reach into both hobbyist and professional communities, you can learn a lot from human age. For example, many people on GitHub transform over time from hobbyists into professionals. How do people age with your product?

Responsibly gathering and using personal demographics wisely is a game-changer.

Three New User Studies

GitHub’s user research team is tasked to go beyond machine reporting in dashboards, connecting the what with numbers to the why with human stories. We take a mixed methods research approach with closed-answer and open text fields, and interviews. Three instruments power three studies:

  1. Longitudinal New Account Creators (NAC) Survey – 1-year of panel data collection with a single cohort, delivered continuously to 90,000 new account holders.
  2. 365 Inactive Accounts Exit Survey – one-time cross-sectional study with 3,000+ respondents (snapshot).

1.) The New User Quiz

Let’s talk about you:

  • What questions would you ask your newest users upon sign-up?
  • How would those questions help you predict success/failure?
  • Where would they uncover intent/motivation?
  • What would you be able to learn about people signing up for your service today as opposed to your early adopters?

2.) New User Panel Data “NAC” Longitudinal

Graph from the survey tool, raw data
  1. Panel studies select a single sample and study that same sample over time at repeated intervals. This is a particularly resource intensive method, because the same subjects must be located again, contacted, and convinced to continue participating in the study. At each wave, the recontact rate is typically around 10%.

3.) New User Exit Study, “The GitHub 365”

  • Why did they leave?
  • What’s one thing you could have done differently to help them succeed?
  • How (and should) you try to bring them back?

Inactive, but not abandoned

When we think of inactive users we think of abandoned accounts. In the 365 study we reached out to 100,000 inactive accounts created from December 2014 — December 2015. More than 3,000 people responded.

Q: Which best describes why you stopped using GitHub?

People shared thoughts and experiences through closed-response options and open-ended responses, depicting that more than 50% of inactive account respondents (who are humans and not bots) indicated that they intend to return to GitHub someday.

Wrapping up

Studying new users is a multi-phase approach:

  1. Go back 365 days and send an Exit survey to learn why people left.
  2. Design a longitudinal study with continuous research into today’s newest users of your product.

UX Researcher. Today: Googler on Firebase. My exs: Airbnb (2018), GitHub (2016) & Mozilla (2012). Human interface to Paisley pug.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store