Closing the gap between behavioral science and business

It is a strange time to be a behavioral scientist in business. When I left my PhD program almost ten years ago to focus on real-world applications, I spent the majority of my first few years explaining over and over what behavioral scientists actually did. Now, I regularly get inbound requests to speak at large companies and books by Adam Grant, Dan Ariely, Jonah Berger, Angela Duckworth, Amy Cuddy, Barry Schwartz, and others sell millions of copies. Executive confidence that behavioral science is both valid and interesting has seemingly never been higher.

And yet hiring behavioral scientists to explicitly apply what they know has remained somewhat rare. When I left Microsoft, I got far more recruiter inbound based on my background in startups and venture than I did in behavioral science, despite being considerably better at the latter. And while there are a few corporate behavioral science groups, likeOm Marwah’s burgeoning team over at Walmart, Steve Wendel’s at Morningstar, Charlotte Blank’s at Maritz, Prasad Setty’s at Google, and Jeff Helzner’s at AIG, by and large the Fortune 1000 seems to love behavioral science without actually applying it.

There is a fundamental gap here. How can executives believe in behavioral science, espouse its virtues and recommend its writings, and yet not be investing in its application to their own companies? Surely it can’t be all TED talks and no desire to see it work?

And we need it to work. We live in a world where ~70% of people are fundamentally disengaged with the work they do (and that number hasn’t meaningfully changed in 15 years) and despite record highs in health and wealth, personal happiness is actually declining in the United States. We are doing better but feeling worse and our current approaches to addressing that aren’t working. If we want change, we need to employ people trained in and focused on behavioral outcomes.

So why isn’t that happening? To me, the gap isn’t a failure of business but of behavioral scientists, myself included. If we accept that executives across industries want to incorporate behavioral science into their companies, then we need to remove the barriers to implementation until it is applied repeatedly and at scale. We need to demonstrate the kind of problems to which behavioral science can be applied and create clear processes for that application.

We’ve done a bad job so far. When I was preparing to write this post, I asked some of my fellow scientists if they wanted to contribute examples of their work and methods. The overwhelming response was “we wish we could”. Because talking about human behavior and its explicit modification remains an area of corporate secrecy. We trumpet the results but not the methods.

So I want to give an example from my last job at Microsoft and to call out the potential for a replicable process that could underlie the application of behavioral science at scale.

Because I believe every large company should have a Chief Behavioral Officer and team actively applying behavioral science to identify and improve human experiences, both internally and externally. And I believe that showing how we do it makes it easier to do.

When I first came to Microsoft and looked at Bing, one of the things I heard anecdotally was that search volume was lower than expected in schools. The working theory seemed to be that kids simply weren’t curious enough and didn’t know that search could be used to satisfy curiosity, and there were vague plans to run a marketing campaign encouraging curiosity-based search in schools.

So I got my hands on query volume logs and ran some regressions to confirm the deficit with data, creating a metric of “searches per student per day”, then went to go visit some classrooms. Unsurprisingly, there didn’t appear to be a problem of inquisitiveness: anyone who has interacted with kids for more than a few minutes knows there isn’t a lack of curiosity. Instead, three big inhibiting pressures stuck out, inherent in teachers’ concerns about search: a vague and unspecified fear about how student data was being used, advertising in the classroom, and potential exposure to adult content. Plus teaching search itself was a grey area. Was it a media skill, best left to librarians? How did it really fit in the daily rhythm of a class?

After the classroom visit, I got a few people in a room and used a Competing Pressures model to design Bing in the Classroom: ad-free, safe, private search that was free to schools, coupled with daily mini-lessons that fit in existing classroom models. We helped Engineering to build it, acquired a small budget to publish lesson plans, and convinced large districts to sign up in advance of the launch.

The net result? A 40% increase in student searches at participating schools and an additional 15% at home, larger than any single previous product innovation at Bing.

Bing in the Classroom isn’t the only project I’ve built that changed behavior at scale. One of my startups, personal finance tool Thrive, got bought by LendingTree because we proved that we could increase people’s credit scores by 20 points in six months through behavioral finance. And GetRaised, a tool we built that helps women figure out if they are underpaid and what to do about it, has helped tens of thousands of women earn over $2.3B in raises by changing the approach from encouraging women to “lean in” (as if they don’t already want fair pay) to simply making it easier to act. I can rattle off a laundry list of behavioral science interventions that have worked.

But of all the things I’ve built, Bing in the Classroom feels most like a process that can be repeated inside corporations to change behaviors of value.

And that’s the key. I believe behavioral science can be used to affect a myriad of business problems that have proved resistant to other efforts: the hiring and retention of the underrepresented, the modification of products and services to work across cultures, entrenched user experience problems like payment and support. But if we want the chance to address them, we need a replicable process for applying behavioral science to these problems.

It starts with theory-based observation. Behavioral scientists, like all scientists, are trained to look for the systems that underlie outcomes. And all undergo statistical training, often fairly advanced, in order to try to extract the signal of behavioral patterns from the noise of the human condition. For Bing in the Classroom, it wasn’t enough to believe something generally about kids and search; we needed data to confirm it and observation to understand it. Without those, we would be stuck running brand campaigns about curiosity.

Then there was the design process. I use a Competing Pressures model and that is only one of many options, but what the methods share in common is a firm grounding in the science of intervention and behavior change. Behavioral scientists are fundamentally interventionists, rather than operators, because the experimental process itself is about taking how people normally behave and then proving that a change in environment or interaction or situation results in a replicable change in behavior. The goal from the outset is to alter the status quo.

That orientation is very distinct from the processes we teach to those whose primary job is operational. Think of it as the difference between a test (atheoretical evaluation, which tells you that the green button is better than the red one) and an experiment (theoretical evaluation, which tells you why the green button is more likely to result in an outcome than a red one, and thus allows generalization to purple, blue, and orange). Because behavioral scientists don’t just care about what but also why, they can better create interventions that change behavioral outcomes in more permanent and integral ways.

This focus on the why of human behavior also brings an openness to a wider range of potential interventions than most operators. Unsurprisingly, when asked to generated solutions, people gravitate to those within their locus of control. Marketers tends to find marketing solutions, like a curiosity campaign, and the engineers find technical ones; for Bing in the Classroom, they wanted to build out a special desktop client to be individually installed on each student machine (school district IT people, you can thank me for vetoing that outright). Behavioral scientists bring something novel to table because their domain is behavior itself; all levers are fair game.

A removal from operational components also creates another uniqueness for behavioral scientists: The ability to let go. There is a part of the Bing in the Classroom story that is rarely told. Once it was launched and scaled to around 10M users, with the query volume increase proved, I stepped away and handed the program over to Marketing. Indeed, the two marketers who took the handoff were named Marketers of the Year by the CMO; I was the one who nominated them.

While many at Microsoft thought this was odd, my scientist friends instinctively understood, because this is a unique facet of scientific research. When a theory is proved, it becomes part of the corpus of knowledge that belongs not to the researcher but to the field as a whole, and researchers immediately transition to the next area of investigation.

This is fundamentally different from what we instill in operators, who are generally rewarded for retaining ownership when something is successful and thus change only when forced. By constantly working toward solution and transition, behavioral scientists can be a potent force for change. And politically, HR, marketing, and other disciplines have everything to gain and nothing to lose, because any positive lift created remains with the operators and the behavioral science team moves on. This is why I like internal NPS as a potential measure of the efficacy of a behavioral science team: if you are doing your job correctly, other teams will recommend working with you.

Taken collectively, there is a replicable process here: theory-based observation coupled with data science leading to intervention-focused behavioral design, with a build-test-refine cycle and then a handoff of an empirically validated program that is ready for scaling and operation. It can be applied to novel human experience problems that operational teams have had difficulty solving because it brings an entirely new perspective and is distinctively apolitical. The team and expenses are small because your only costs are human capital and the potential upside can be directly linked to profit.

And that is an important word: profit. In a world where businesses at scale have done everything in their power to maximize earnings, behavioral science is an untapped well. Imagine the efficiency that comes from addressing the 70% of the workforce that is disengaged. Or reversing the negative consumer happiness trend in the United States.

For the majority of large businesses, improving the human experience of a company is where the next massive increase in profit will come from.

There will be a first CEO willing to commit a few million dollars to hiring a CBO and team to drive change in their business. But it is on behavioral scientists to make this work easier to embrace. To show structure, process, and results. Not a skunkworks of product ideas thrown against a wall, but a disciplined approach to behavior change. Not tests but experiments. Science, well applied.

Side note: One of the chief difficulties of enlarging the presence of behavioral science in business is the tendency toward secrecy. Governments, including both the US and UK, have behavioral science teams that are transparent about their process and success. Yet business lag behinds. At Microsoft, we discussed my hiring but never noted that Bing in the Classroom was a direct result of a behavioral science experiment. If we want to close the gap, we need to do so in public. So if there are other behavioral scientists willing to contribute case studies to a followup post, send me a note.

Editors: This post would not be possible without the input of both other behavioral scientists and businesspeople, all of whom I am lucky to count as friends. Thanks to my wife Stefanie Sugar and Dominic Price, William Leach, Kevin Brilliant, Lauren Woodman, Kara Silverman, Jonah Berger, Adam Grant, Om Marwah, Steve Wendel, Jeff Helzner, Josh Wright, Michael Butera, Val Tsanev, Charlotte Blank, Dan Storms, Erik Johnson, Avi Karnani, Bill Cromie, Carson Miller, Andres Glusman, Michael Norton, and Kelly Peters.

Originally published at Matt Wallaert.