Listening is not enough
Participatory video and multi-stakeholder dialogue for humanitarian aid evaluation
by Marleen Bovenmars, Head of Training & Design, InsightShare
Within the humanitarian aid sector there has been a growing interest in participatory evaluation and ‘communicating with communities’. Many organisations and funders are keen to promote accountability and better understand the needs of disaster affected communities. Many NGOs have however been struggling to find evaluation methods that are accessible, time and cost-effective and that bring them concrete insights for future programmatic improvement. For many people working in the sector, evaluation has become a tick-box exercise that is often felt as a distraction from their actual work.
In the spring of 2017, InsightShare partnered with ActionAid Bangladesh to explore if and how a participatory video approach to evaluation could tackle these challenges. Their pilot programme aimed to enable two flood affected communities to play a central role in the evaluation of the 2016 flood response in Northern Bangladesh. Six youth from the two communities were trained to form an evaluation team, alongside them two ActionAid staff members were also trained and worked with InsightShare’s facilitator to guide and coordinate the process.
This article summarises the methodological conclusions from this pilot programme. Guided by a series of key questions, it presents how participatory video can facilitate the kind of cost-effective, multi-stakeholder dialogue that is necessary for community perspectives to inform the design of future aid programmes.
Understanding how an aid intervention worked
More and more funders emphasise the need for rigorous impact assessment processes. These assessments can provide them and their grantees with essential quantitative insights. However, in order to go beyond knowing to what degree past humanitarian responses were appropriate and successful, we need to develop a clear understanding of how programmes supported the recovery of crisis affected communities, how experiences were different for various sub groups within these communities and how future programmes could be (further) improved.
There already seems to be a general understanding in our sector that stories can provide an easy and engaging way to access this type of rich information. Stories place the experiences of individuals within the complex context of their lives and environments. However, it can be challenging for NGO staff or consultants to get beneficiaries to open up and to talk about their experiences in the form of a story. In addition, it can be time-consuming to identify how people’s individual stories compare to the collective experience of a (sub)group and to select whose stories to highlight in an evaluation.
So what if community members could facilitate the story collection and selection process?
During the first stage of the pilot programme in Bangladesh, the 6 community representatives and AAB staff received training to enable them to facilitate one so-called ‘story circle’ with each of the four sub groups in both pilot communities. This process enabled the participants to share their stories and ideas with each other in a safe and comfortable environment. After the story sharing, the trained youth used a quick analysis tool to help each sub group to identify whose’ story was a strong example of their collective experience and to find out if that person would be comfortable to share their story on video.
The 85 collected stories provided valuable insights into people’s experiences, values, opinions and relationships, whilst providing social and environmental context. The stories illustrated (unexpected) factors that hindered or helped people along their journey of recovery. They showed for example how households that were most in need didn’t receive any or not enough aid, how many people didn’t receive the kind of aid that they felt they most needed and how some aid that was delivered was useless. The stories highlighted a need for greater consultation, information and coordination between government representatives and NGOs. They also identified the scarcity of distribution points and the great distances involved in reaching them and the impact of nepotism as some of the main barriers to effective aid provision.
Identifying concrete ideas for improvements
More and more organisations collect the perspectives of their beneficiaries during their evaluations. However, the collected data is often analysed by staff members or external consultants. In addition, it seems that few organisations ask their beneficiaries to come up with concrete ideas for how programmes could be improved.
In these complex systems, community members clearly do not have a complete understanding of all the complex factors that influence the delivery of aid at various levels. However they do hold an important piece of the puzzle and they are the ultimate recipients of the aid processes designed and implemented by the various international, national and local aid providers. What would happen if aid provision processes could be designed and changed according to the needs and ideas of those whom they are meant to serve?
What if these beneficiaries could be supported to analyse evaluation data and use the insights drawn from this analysis to ‘design’ improvements to the aid delivery process?
During the second stage of the pilot programme the trained youth team were supported to analyse all 85 stories that they had collected. With the help of simple tables and drawings, they extracted the key challenges, enabling factors and ideas from each story. The completed tables also helped them identify which problems, enablers and ideas were considered most significant. Other participatory tools were used to increase their understanding of the relationships between the various problems and enablers as well as the capacity of different stakeholders to limit the negative impacts of flooding in the future.
The youth team shared these findings with the wider community in workshops that drew on human centred design (design thinking) to enable community participants to are-design the aid provision process and their own role within it. Through this process they identified important ways in which the key challenges could be addressed as well as strategies to leverage the communities’ internal strengths and resources. Their re-imagined aid provision process includes some simple but effective steps that would enable more timely and accurate needs assessments and minimise the opportunities for nepotism. In their design, community volunteers play a key role in the needs assessment process which could reduce the NGO staff time that would be required for implementation.
Producing accessible and effective communication products
Whilst we see more and more evaluations being carried out, how many of the actual findings manage to reach the right people quickly enough to inform the design of the next programming cycle? How often are significant amounts of time and money invested in the production of high quality evaluation reports that stay on the shelf and fail to reach their audience?
What if stories and evaluation findings could be clearly presented in short, engaging videos that are easy to share?
Although many organisations see the value of producing video products that present case studies or that summarise the findings of an evaluation, the costs of hiring a video professional often provides a barrier. In addition, community members often feel nervous speaking to outsiders let alone to a stranger behind a piece of impressive video equipment.
What if evaluation videos could be shot and edited by community members themselves with minimal support from NGO staff as an integral part of the evaluation process itself?
During the pilot programme, the trained youth used drawing exercises to support participants to recount the key elements of their stories and to help prepare the storyteller to speak in front of the camera. Who was behind the camera? The local youths filmed each storyteller in a quiet place of their choice in their community context. This meant that they could talk in a familiar environment, with the support from their peers. InsightShare’s participatory video method ensures that the footage is watched back and that participants can decide to make changes or give their consent. They are in control and this changes everything. In the same way, the team filmed the participants’ ideas for how the aid provision process could be better in the future.
This approach ensured that the participants’ perspectives were captured in a way that remained true to their experience, and that effectively carried their voice and viewpoint, without being unduly filtered or interpreted by intermediaries.
Over the course of the programme the team learned to use editing software to organise parts of different people’s stories and to create videos that summarise the collective experience of the community according to the key findings of their analysis. The outcomes are five short videos on the key areas of flood impact as well as an summary video that brings together all the key findings and recommendations from each area.
Facilitating multi-stakeholder dialogues
It would be a great start if more organisations would invest in working with community members to capture their concrete ideas for future programmatic improvements. Organisations should however be mindful that previously top down approaches can’t simply be replaced by bottom up approaches. A structural change may be called for, underpinned by constructive dialogues between communities and other stakeholders; as together they could come up with the best designs and solutions.
What if an iterative process of learning and design could be facilitated between community members and relevant stakeholders?
The videos that summarise the evaluation findings of the pilot process in Bangladesh have been screened to stakeholders at the local, district and national levels. At each screening event the attending stakeholders were asked what opportunities they could see to take forwards the evaluation findings and the communities’ ideas. Stakeholders were also asked to present their key responses in front of the camera so that their feedback could be screened back to the community members.
“The PVM&E process is an effective advocacy tool to communicate to governmental duty bearers how humanitarian assistance can be effectively delivered to the communities in a more transparent way during emergency periods. It promotes two way communication with communities. In existing M&E systems, not all levels of stakeholders engage, but through PVM&E this has been ensured.” Abdul Alim, Manager ‘Humanitarian Response, Disaster Risk Reduction and Climate Justice’, ActionAid Bangladesh
Affordable capacity building and national-level collaboration
The pilot programme in Northern Bangladesh demonstrated how a participatory video approach can be used to facilitate multi-stakeholder iterative cycles of research and design. However, facilitating a complete participatory video training programme each time is costly and the evaluation and training budgets of many NGOs are limited.
What if the process could be implemented in a way that is efficient, thorough and affordable and that encourages the sharing of learning between NGOs working on the same type of humanitarian responses?
Based on the lessons from the pilot programme in Bangladesh, InsightShare is keen to offer national level, collective training programmes in the participatory video approach to staff members of different organisations that provide the same type of humanitarian aid. During the training programme the staff trainees would be trained and mentored whilst they manage their first complete participatory video evaluation process in one or several of their beneficiary communities. They would also come together with the trainees from other NGOs to share their learning.
The national level, collaborative training approach would be cost-effective. First of all, the costs of the complete training programme would be shared between all participating organisations. Secondly, only a limited amount of communities would need to be consulted, to collect enough concrete insights and ideas to inform programmatic design of one type of disaster response. As long as the communities are carefully selected to provide a relevant sample. Finally, once the trainees have carried out their first complete evaluation process with guidance from an InsightShare trainer they can continue to evaluate other types of disasters independently. In Bangladesh, the trained team has already carried out their second Participatory Video evaluation, with communities affected by landslides.
The global alliance for the Core Humanitarian Standard has expressed an interest in our training model. The Core Humanitarian Standard (CHS) makes nine commitments to communities and people affected by crisis, supported by quality criteria and a set of key actions and organisational responsibilities needed to meet each commitment.
“Humanitarian workers must be accountable towards the communities they serve and, conversely, communities can hold humanitarian workers to account. InsightShare’s community feedback initiative through participatory video seems a very effective, promising approach as it allows for genuine, inclusive story-telling.” Rezaul Chowdhury, CHS Alliance’s board member and Executive Director of COAST Trust, Bangladesh.
For more information or if you would like to participate in future participatory video trainings feel free to get in touch.