Everything all at the same time
Knowing which communities to prioritize for protection isn’t always easy. In Guatemala, UNHCR is innovating with data to make evidence-based decisions.
Although UNHCR, the UN Refugee Agency, does many things, it is at heart a protection agency — dedicated to saving lives and ensuring the safety of people forced to flee. But it’s not always so simple to identify which communities are most in need of protection. In Guatemala, as in many contexts, those decisions are primarily made based on the expert insights of UNHCR colleagues. While valuable, these insights aren’t always backed by robust data.
Diego Nardi (now working as an Inter-Agency Coordination Officer in Moldova) and Associate Information Management Officer Pedro (Peter) Martinez thought that was a problem. “So, we decided to try to understand what is happening, in terms of human mobility and socioeconomic issues, in every municipality in Guatemala through data,” Peter recalls.
With the support of the Data Innovation Fund, Peter and his team started working on a composite indicator — a tool that would analyze data on, say, livelihood opportunities, public services, gender-based violence, food security, and so on, to assign a severity score to each municipality — which could be used to help UNHCR more effectively solve the challenge of which communities to prioritize for protection.
One year on, the project has resulted in the Admin2 Severity Index Tool (A2SIT). A user-friendly, open-source web application, the A2SIT does what the team hoped it would, quickly visualizing the severity of specific socioeconomic conditions across the country in an easy-to-understand format. UNHCR colleagues can simply gather relevant data points and upload that raw dataset into the website; in response, it delivers a color-coded map, overlaying a severity score across distinct geographic areas.
But the process wasn’t straightforward. “It was a rollercoaster, this project, actually!” says Information Management Assistant Juan Francisco Verdugo, otherwise known as Juanfra.
A winding road
UNHCR teams supported through the Data Innovation Fund are guided on their project journey by the Data Innovation Roadmap — a methodology, outlined in an infographic, that details the different steps on the path toward effectively combining data science with innovation methodologies. The Roadmap is an essential reference tool and includes checklists to enable the transition of teams from one stage to the next. It also assumes a certain level of nonlinearity.
“That was really useful for us, to see that even in the Roadmap, our back and forth process was actually shown there,” says Peter. This process was shaped by several key challenges:
1. Deciding what to measure: Building the framework for the composite indicator — deciding which data should be included — was a steep learning curve. Partway through the project, the team had to rethink its approach entirely, based on the insights of experts in the field and of communities themselves. In discussions with forcibly displaced people, certain socioeconomic issues emerged as a key concern, resulting in the addition of more indicators focused on livelihoods and essential rights (to work, to health, services, education, and so on).
“It was kind of a mess,” Juanfra says, with a laugh, of this part of the process. “But the good part is we understood a lot more about what we wanted to measure, we did a little bit of research, we had to read a lot of academic papers. And we understood more about what the framework needed.”
2. Responding to data gaps: While Peter and Diego had initially hoped to look at the community level, data with that level of geographic specificity was simply not available. The team adjusted its focus to municipalities and, even then, the data was far from comprehensive. The solution? Bringing together information from many different sources — government, NGOs, academia, think tanks, and so on — and exploring ways to include qualitative data to plug gaps and add nuance.
“You can have an indicator that shows how many health centres there are in a given area,” Peter says, by way of example. “But if there’s no doctors, no medicine, no nothing, you can have tons of centres and they’re not working.” Or, as Juanfra notes, the medical staff might not speak the local language.
To include relevant qualitative insights, the team worked with experts across Guatemala’s municipalities, building a set of 12 new indicators. They’re now collecting information (using a set of questions for each indicator), which will then be operationalized — in other words, assigned a numerical value, so it can be included in the composite indicator. This process is being documented, and the indicators and questionnaires will be included as part of A2SIT documentation, so others can use them.
3. Managing multiple perspectives: The final challenge the team faced was also their greatest strength: many different voices. “One of the interesting things I’ve learned during this project is that including a lot of perspectives is really cool, it gives you a lot of tools, and helps you to develop the project in some new ways you didn’t think about,” Juanfra reflects. “But when you have many voices, it can be difficult to decide where to go.”
The team sought to balance their priorities against those expressed by others, and against the needs and strategic direction of UNHCR more broadly. Although such wide consultation made the trajectory of the project challenging to maintain, it also resulted in rich learnings. “A lot of people had really good ideas,” says Juanfra. “At the end of the day, the project wouldn’t be what it is right now without them.”
A simple, flexible tool (and a few surprises)
A2SIT is the result of this work: a composite indicator builder housed in a web application. UNHCR colleagues and others can input the relevant data into an Excel sheet containing the framework of indicators, upload that file into the A2SIT website, then press go. In response, the tool produces a color-coded map, with each municipality assigned a score based on what the data says about the severity of its situation. Juanfra says:
For me, the most important thing is that for someone technical or nontechnical, it gives you a really good insight into each municipality according to the data.
A2SIT is, of course, intended as a complement to, rather than a replacement for, human decision making. It provides additional insights to inform UNHCR’s protection decisions. And its results aren’t always intuitive. “We can think of a thousand communities in need of help,” says Juanfra, “but to narrow this discussion to UNHCR’s mandate has been really difficult.”
Many early testers of the tool expressed surprise that communities they knew were experiencing great hardship weren’t assigned a higher severity score. The reason: although those municipalities had many problems, human mobility wasn’t one of them. “We have a feature in the tool where you can work with the weighting of different areas,” Peter explains. “The area of human mobility, when you turn it down or off, the map looks like what people expect. That’s one way to prove the tool is actually working.”
Once users understand this rationale, the results make sense, and feedback has been positive so far. A2SIT is also customizable, so users can build their own frameworks, adjust the weighting of each indicator, and select different aggregation methods (to address perceived bias in the data). “From the start we wanted to build something useful not just for Guatemala,” Peter says. “I think it could have a big impact across UNHCR, if people use it.”
An ongoing journey
The team has traveled a long way — from not fully knowing what a composite indicator was, to building one that could have an enduring positive impact on the way UNHCR prioritizes communities for protection projects, thereby ensuring services reach the people who need them most.
The perspectives and priorities of the people UNHCR works with and for have been, and continue to be, crucial to shaping A2SIT — to ensure that the tool’s results encompass the factors that displaced people have identified as being foundational to a good life in the place they are living. Although these communities aren’t the intended users of the tool, their insights ground its results in lived experience. Peter says:
Working with communities and discussing what they think about the tool has been super useful for us. … We’re now conducting user-interface, user-experience testing with UNHCR colleagues, so we’re asking technical and nontechnical people to use the tool, to check whether it’s intuitive.
They’re also starting to test the tool across new contexts, with interest from several country operations in the region (and from entities beyond UNHCR). This user testing will enable the team to iron out any software wrinkles before the website goes public. As with any data tool, A2SIT’s output is only as good as the information it is fed. But the team is confident it could be scaled up to become an important addition to the agency’s decision-making toolkit, and to inform interagency coordination exercises.
What’s more, the process has been personally and professionally enriching — and fun. Juanfra reflects:
I come from an academic process, where everything is slower, there are clear steps. With innovation, it was like everything all at the same time. I knew how data can help to publish a paper in a journal, but I didn’t know how data could help an organization do humanitarian work. Thanks to building this tool, I finally understood what UNHCR is — what the mandate is. It’s been really interesting to understand that part: data for the humanitarian world.
The A2SIT will be made publicly available in the coming months. In the meantime, explore the detailed documentation behind the tool and find out more about UNHCR’s Data Innovation Fund.
