Avoiding the Black-Box Effect: Tracking the Impact of Flood Analytics

by Sri Ramesh, Monitoring and Evaluation Advisor, Cloud to Street

First responders and flood-affected people in Yatagampitya, Sri Lanka show Cloud to Street CEO Bessie Schwarz the impacts of a recent flood.

Frequent catastrophic flooding affects millions of people every year. Those who are tasked with responding to these escalating crises—namely, first responders with the UN, IFRC, and within national disaster management agencies — must invest in good data in order to coordinate disaster responses, and ultimately, save more lives. “Good data” in flood response includes timely, accurate estimates of the number of flood-struck and at-risk civilians and the amount of roads and agrarian land destroyed by flooding. Such information is in high demand among first responders in charge of organizing response logistics and saving lives in an emergent situation.

Yet, data providers are frequently left in the dark when it comes to how their insights are used to drive decisions on the ground. Let’s say that we provide good data (in this case, timely, accurate estimates of flood-affected population, roads, and cropland) to end-users in a public sector organization in the form of a dashboard that updates weekly. How do we avoid the black-box effect around what happens next? In other words, how do we systematize the tracking of decisions and actions that end-users take on the ground?

Over the last five years, I have worked on providing data to governments both as a federal analytics consultant and as part of the flood analytics team here at Cloud to Street. I have also spent time in South Asia and Sub-Saharan Africa conducting randomized and quasi-experimental impact evaluations. These two experiences have exposed me firsthand to the challenge of the black box that so many data providers face when they attempt to track the impact of their work in the broader world.

At Cloud to Street, we take this question seriously by developing strategies to dismantle the black box. In other words, we systematically track the lifespan of the analytics we provide through the decisions and actions our end-users take in the world of flood preparedness, response, and recovery.

Here are a few insights we have learned along the way:

1. Our end-users in the humanitarian sector often triangulate different data sources together to evaluate their priors about a given situation. In periodic conference calls, our end-users explain to us that the power of the satellite-driven estimates of flood-affected populations, roads, and croplands that we provide through Cloud to Street’s Flood Analytics Dashboard has been in combining our data with existing data sources on flood impact. Our users use our satellite-driven estimates of flood damage to verify numbers coming in through the media, needs assessments, and other forms of ground truth. They are also able to get eyes on the ground faster with our satellite-driven estimates, and validating flood events faster in instances where extreme flooding cuts off access to flood-affected regions and delays their ability to collect ground truth information. In light of this reality, at Cloud to Street we track impact not by drawing a one-to-one relationship between our product and our end-users’ decisions, but by drawing many-to-one relationships between the end-users’ multiple data sources, of which ours is one, and a singular decision that end-users will optimize on the ground.
2. Our end-users need to optimize certain decisions, which means our impact-tracking process should include indicators for how those decisions evolve throughout the project lifecycle. Our end-users usually seek out Cloud to Street’s near real-time flood mapping services with one of a few goals in mind: improving their preparedness for future extreme flooding, reducing their response time to a flood emergency, or monitoring a specific region of interest for flood events. With this in mind, we work with our end-users to identify two or three priority activities or decisions to optimize. Finally, we develop a unique system to define each activity in quantifiable terms, set goals within that activity, and measure progress toward each goal.
3. Tracking the impact of a data product is heavily reliant on strong working relationships with the end-users themselves. Our most important insight in the course of developing an impact tracking framework was the necessity of cultivating strong working relationships with our end-users. At Cloud to Street, we speak with our end-users on a bi-weekly basis to get an understanding of the lifespan of the data product in their world, to address any ad-hoc needs around additional flood information, and to track progress towards the end-users’ impact goals.

Cloud to Street is on a mission to protect ten million flood-vulnerable people worldwide in the next five years. To track our impact towards this goal, we must piece together what happens in our end-users’ world after we deliver rapid flood mapping services in near real-time. Success in this endeavor requires having open and consistent lines of communication with all those who consume the data, make decisions with it, and act on it. This approach ensures that the black box can be dismantled and that the data we provide results in actionable decisions on the ground.

Sri Ramesh is a Monitoring and Evaluation Advisor at Cloud to Street.