Making Low-data decisions

Imagine you’re designing a new program. It’s a little different than what your organization usually does, so you don’t have much internal data to work with. You’ve got some big decisions to make and not a lot of information to work with. Now what? (Actually, you probably don’t have to imagine this scenario — you’re likely living it.)

Here are some thoughts to get you making smarter (or at least more confident) decisions in these low-data scenarios.

1. 👋 Talk to people. None of us want to look like we don’t have the answers. But don’t let that keep you from talking to people — your teammates, program participants, community members, even the funder. Share what you’re thinking about and why, and ask for advice. Being open to input can unlock insights on your audience, the service you’re offering, and the sustainability of the program.
2. 🤖 Create a persona. So you don’t have any real participants yet. But you do know who your intended audience is. Recently, I had a data request for “a map of theater-goers.” I don’t think the Census asks about that, but we were able to quickly develop a couple personas of who we want to serve and map their attributes. Consider their demographics, economic status, health conditions, location, etc. to get a better starting point. You can replace the personas with real people later. [1]
3. 💭 Think probabilistically. Even if you don’t have empirical data, you can make informed estimates and update your thinking as you go. Most outcomes — especially in our field — aren’t binary; they don’t simply succeed or fail. Start with a hypothesis: “I think this program has an 80% chance of improving the lives of most participants.” As you add or tweak a component in your design, and as you get real data, revisit that hypothesis. Do you think the probability is higher or lower? Keep revisiting this throughout and you’ll see how your decisions are becoming smarter and your participants are better off for it. [2]
4. 👩‍🏫 Focus on learning. “All news is good news when it results from explicit learning about the pertinent uncertainties.” If you’re working with a lot of uncertainty, then you’re probably innovating, which means you should calibrate your expectations toward learning rather than impact. For instance, you may have a great understanding of your potential participants and still not know if this new approach will be impactful. Take the pressure off yourself to know everything before it happens, and direct it toward learning something new. That will help you eventually modify, scale, or end the effort. [3]

We don’t always start with solid data. But we can find ways to think more logically and create information along the way. That’s how we advance the social impact field!

___________

[1] Crossing the Chasm is a classic with great tips on bringing a new product or service to market.

[2] Nate Silver’s The Signal and the Noise is an amazing intro to probabilistic thinking and bayesian statistics (you don’t have to learn the actual formulas of it for it to be useful)

[3] The first two chapters of Innovation and Scaling for Impact lay out some simple frameworks for understanding if you’re innovating (high-uncertainty investments) or scaling (getting to bigger and/or better impact)