Step-by-Step Evaluation Replication Guides

Adapt and adopt successful evaluation projects and trials from The Behavioral Insights Team’s work with WWC cities

By The Behavioral Insights Team

Over the past 5 years, we at the Behavioral Insights Team (BIT) have worked, to date, with 67 cities across the U.S. and have run 120 evaluations to recover hundreds of thousands of dollars in revenue, triple enrollment in new city programs, and to help residents better navigate services. In addition to working directly with cities to run evaluations, BIT also has supported these cities to build the skills internally to run successful evaluation projects on their own. As part of What Works Cities (WWC), we’ve leveraged the social sciences to help cities design rigorous and practical evaluations — like randomized control trials and other quasi-experimental designs in order to generate data and evidence on their programs and services.

[What is an evaluation and why do they matter? Learn more about this foundational best practice that data-driven governments are using to get results!]

We’ve learned a lot along the way, finding and testing what works, what doesn’t, all while fine-tuning what it takes to successfully implement a rigorous data-driven evaluation in the context of a city. Now we want to help cities learn from and replicate the successful projects conducted by their peers in order to get results for residents and make the most of taxpayer dollars.

What are the Evaluation Replication Guides?

In this suite of replication guides, you will find examples from cities — like Newark, NJ and Tulsa, OK — unpacking what these evaluations can look like along with detailed instructions on how to administer similar efforts in your city.

The projects selected for these actionable resources have a few things in common:

• A simple, low-cost, evidence-based intervention to target a prevalent municipal issue, drawing from the behavioral sciences;
• A robust evaluation plan using experimental methods to determine their success (and help you make progress towards WWC’s evaluation assessment criteria);
• They all worked! In each project, cities saw measurable positive impacts for residents.

Specific guides that we are offering:

• Provide More Affordable Housing by Increasing Landlord Unit Registration: Letter campaign in Newark, NJ
• Increase Support Service Uptake via 211: SMS campaign in Tulsa, OK
• Coming soon! Switching to Paperless Billing: Mailing insert campaign in Scottsdale, AZ
• Coming soon! Reduce Fines for Household Waste Disposal: Email reminder campaign in Charlotte, NC

You can use these off-the-shelf, step-by-step project guides to learn from and build on the successes of your fellow cities, and to effectively tackle similar challenges in your own city. Each resource is broken up into three sections — the challenge, the solution, and the evaluation — to take you on a deep dive into the work the city conducted to solve a specific pressing challenge.

If we already know these projects work, why do we need to evaluate them?

These guides are based on actual projects that real cities have conducted. Although other cities had positive findings, it’s important to test to confirm that the intervention will have a similar effect in your city. The context matters because each city has its own unique culture and processes. While we may have rigorously evaluated an intervention in at least one city, we typically would want to see multiple cities replicate their success to be sure an intervention really works.

By conducting evaluations, you will make sure that the interventions work as intended in your city’s unique context, build upon the evidence base to inform other cities’ efforts, and build capacity to conduct more evaluations in the process.

In addition, when you run a project from one of these guides, you are directly helping to build the evidence base for what works in cities (as these evaluations tackle projects that address issues that are common in many cities around the country), and are contributing to our shared body of knowledge in this growing movement!

Choosing the right evaluation project for your city

Each project in our replication guide is low-cost to implement, uses communication channels that typically exist within most cities, and can be conducted by existing city staff.

Before you select a project, make sure your team consists of personnel who can provide:

• Access to the communication channel (email, letter campaign, etc)
• Authority/approvals to make changes to the communication from relevant or impacted department(s) within your city
• At least one team member with evaluation or data analysis experience (he/she/they doesn’t/don’t have to be a data scientist — but someone that can operate Excel, understands descriptive statistics and testing differences in means (i.e. t-test)).

Additionally, it’s important to confirm you have access to the data you will use for pre-trial randomization and accurately measuring outcomes.

Completing these projects will require time and effort by your team which will enable you to demonstrate the impact of the change you are making and build new valuable skills for you and your team.

Use the table below to help you choose the right project for your city.

Explore the projects:

• Provide More Affordable Housing by Increasing Landlord Unit Registration: Letter campaign in Newark, NJ
• Increase Support Service Uptake via 211: SMS campaign in Tulsa, OK
• Coming soon! Switching to Paperless Billing: Mailing insert campaign in Scottsdale, AZ
• Coming soon! Reduce Fines for Household Waste Disposal: Email reminder campaign in Charlotte, NC

Launched by Bloomberg Philanthropies in April 2015, What Works Cities (WWC) is one of the largest-ever philanthropic efforts to enhance cities’ use of data and evidence. Cities across the country are more effectively driving change and delivering results for residents by participating in our What Works Cities Certification program, the standard of excellence for data-driven local government. The program’s 45 criteria outline the people, processes, and policies foundational to a well-managed city.

Want to learn more about What Works Cities? Contact certification@whatworkscities.org.

The Behavioral Insights Team (BIT) is a social purpose organization that helps governments around the world boost their impact through the application of behavioral science and rigorous evaluation techniques. As part of What Works Cities, BIT helps cities develop and rigorously evaluate ideas for improving government services. We define rigorous evaluations as evaluations that utilize experimental or quasi-experimental methods. We believe this is the best way to help understand the impact of a program or policy.

Want to learn more about the Behavioral Insights Team and services available to cities? Contact info@bi.team