A Product Nerd’s Approach to Volunteer Retention
Combining Surveys, Slack and GreetBot to measure engagement
“Did she come back?”
“Have you seen him lately?”
“What ever happened with that project?”
If you work in a volunteer organization, these questions probably summoned a face or a name in your mind — that new recruit who showed up for some meetings or events and had so much potential. But for one reason or another they drifted away.
Sustainability for our organizations means not just recruiting new members, but also developing a pool of strong members to sustain and grow the organization in the future. It all starts with engaging new recruits.
At Hack for LA, we found that we only knew why someone disengaged if they had already taken on some responsibility. For the vast majority of recent joiners, or for people who kept a low profile, we didn’t know why they drifted off.
We decided to take a Product Management approach to understanding our own organization. Looking at Hack for LA as the product, we could track users’ progress through the funnel, from awareness to multiple points of conversion.
This approach has limitations, of course, as it’s hard even to pin down the conversion points along the funnel. “Attend a meetup” is a clear measurement, but how do you define “took on leadership responsibility”? We could define “became a regular attendee” mathematically, but what data would we use? We don’t check people in at the door — maybe we should — and how do you account for people who contribute remotely but don’t attend meetups?
As a product nerd I’m excited to try to define the user journey through the funnel and define criteria for subsequent levels of engagement with our “product.” But in the meantime I’ll share our approach to measuring the very beginning of the funnel:
- Why do our “customers” make their first “conversion,” i.e., attend a meetup? (Or do we need to consider an alternative first conversion as well?)
- Do they feel engaged after attending a meetup? Are there clues in their behavior during the first week that correlate to continuing to engage or dropping out?
Question #2 is a little tricky to measure — disengaged people are harder to survey. But we might capture what poorly-engaged new members are experiencing in real time.
Implementing a Survey for New Joiners
We already had a qualitative understanding of why people join Hack for LA (the workshop output above is one example). Regulars had of course shared their stories with us. And at the beginning of each meetup our meetup hosts often ask people to share why they came. But we had no formal data on it. Since we needed a quantitative view, the obvious approach was an online survey.
Hack for LA was already using Google Drive and G Suite (Sheets, Docs, etc.), so it seemed like a no-brainer to implement our new joiner survey using Google Forms. We knew the features would meet our simple needs, and the team was familiar with how responses would be posted to a Google Sheet for data analysis down the road.
(You can check out the survey in its final form.)
To be sure that my survey made sense and captured the data I wanted, I created an MVP version and tested it out. First, I sent it to some recent joiners who had become regulars quickly. They provided great feedback on wording and responses to add. They also urged me “not to be apologetic about asking people to fill this out. Just ask directly.” So I took out the polite-speak, thank-yous and we-would-appreciates. In addition to making the language more confident, it had the added benefit of reducing friction through streamlining — desirable attributes when considering our survey as a product.
With these early results in hand, I expanded the pilot by posting the survey link in a couple Slack channels where new joiners tend to subscribe. The data flowed and I didn’t receive any significant new feedback.
Now I needed to find a distribution channel to get results at the scale I wanted.
Distributing the survey
The ideal distribution method for a survey depends on what stage along the funnel we’re trying to measure. For this survey we wanted to reach people in that critical time period right after they first engaged directly with us.
A few options we considered:
- We could have posted the survey in public: on our website, social channels, or even meetup.com itself. But these seemed too early in the funnel. Going back to Question #1 above, we want to target people who have already engaged with us in a meaningful way (i.e., “converted”).
- In-person - We had tried asking people to take surveys during announcements at our meetups, but barely anyone followed through; the timing is just awkward, with the night beginning and lots going on.
And then we realized: what if we could use Slack? Joining our Slack team is the step that nearly everyone takes at their first Hack for LA meetup. We warn them at the start that our teams use Slack for all of their collaboration, even sharing links with people sitting across the table from them. And of course this sets people up to engage with their teams remotely or between meetups. (We don’t ask people for personal details for our roster or mailing list — although this is something we should work on!)
It turns out Slack offers some great tools to engage with team members through Slack apps. The classic example of an app is “Slackbot,” who greets all new Slack users (not just at Hack for LA). Slackbot presents itself as your guide to Slack — if you use Slack’s built-in reminder feature, for example, it’s Slackbot who will pop up on your Direct Messages list to remind you.
Slackbot is just an example of an app, and a relatively simple one at that. Slack provides an extensive developer ecosystem of web hooks and APIs that let a developer program their own “bot” to do all kinds of things. It’s an incredibly flexible toolkit, and if you’re a developer you can build your own apps. For the rest of us, there is a marketplace with hundreds or thousands of apps to choose from. At Hack for LA we use Slack apps from GitHub and Google Drive, both of which make it much easier to talk about work that’s occurring on those platforms. [An illustrative example: I think of the GitHub app as a “bot” that notices when a GitHub link is included in a Slack message, pulls some pertinent information from GitHub (e.g., the title of a pull request), and adds that to the message automatically.]
Our survey use case is so simple, I thought I might have a reason to build my first Slack app. Put simply, I want a Slack app/bot to send new joiners a direct message with a link to our Google survey. How hard can that be? I even hoped I would find a tutorial out there covering exactly that use case.
Alas, I’ll never know, because someone created GreetBot and made it super easy to add it to our team for free. They had already implemented a customizable greeting and even added a time dimension that I immediately wanted! Greetbot lets you create custom messages for different times since joining a Slack team: immediate, a day later, a week later, and a month later.
I thought a day later would be too quick, because I wanted to give the person a little time to settle in and engage with their new team post-meetup. At the other end of the scale, a month later seemed too long to capture “disengagers,” which is a key constituency for this survey. I picked the “week later” option.
Of course I implemented “immediate” and “next day” messages while I was at it, just with general info instead of the survey link.
Next steps
Our data won’t be randomized, obviously — people are self-selecting for some personal characteristics in choosing to respond to the survey. But this bias may play into our intentions. We most want to hear from people who are interested or conscientious enough to take a survey in the first place.
So what can we do with our new data?
- Validate our qualitative understanding about why people came to one of our meetups.
- Get some visibility into whether people join our Slack channel at times other than their first meetup. (I.e., if they’re on Slack but have never been to a meetup.)
- Learn more about the prevalence of different new joiner constituencies — e.g., how many people are hoping to learn more about data science vs. web development or user research?
- Examine how different segments answer questions about engagement.
It would be nice to track engagement levels over time and see how they respond to our engagement efforts. But we probably won’t have the volume of data necessary to run experiments so scientifically, there are just too many variables.
We also need to find ways to measure engagement as people mature with our organization. Do regular members see the right opportunities for growth? If people came to gain new skills — the most common reason we’ve heard in qualitative research — are they successfully learning those skills? Do reasons for participation change over time?
I’ll look for opportunities to write more about this as we learn more, and I’ll share more of our experiments. In the meantime I hope you found our approach useful.
