User research case study: Developing Effective Altruism in Asia

How might we improve mutual understanding between Effective Altruism communities in Asia and the West?

Note: some images have been intentionally blurred to protect confidentiality.

Network map of Effective Altruism stakeholders in Asia

Summary

Objective

To improve understanding and communication between Effective Altruism (EA) communities in Asia and Western countries.

Methods and Tools

• Documentation review
• Database aggregation
• User interviews
• Stakeholder mapping (geographical and network)
• Stakeholder case studies
• Evaluation framework design
• Theory of change mapping

Results

Improving understanding of the Asian landscape

• Finding: Most stakeholders underestimated the amount of activity happening in Asia, including those from Western and Asian countries, possibly due to the lack of public information about EA activities and actors in Asia.
• Output: I created two stakeholder maps (geographic and network) to provide the global EA community with an accessible overview to the Asian ecosystem. The network map also highlighted emerging hubs in the region.
• Output: To foster a deeper and more empathetic understanding, I also developed case studies, illustrating EA development in the emerging hubs, i.e. Singapore, the Philippines and India.

Providing a common language for strategy evaluation

• Finding: None of the interviewed stakeholders had a clear view on how EA should be developed in Asia and there were mixed views on the region’s importance to the overall movement. Some Asian stakeholders felt confused about how their activities were being evaluated, and faced challenges in developing their local strategies.
• Output: I created an evaluation framework that compiled the key considerations of Western stakeholders when assessment EA development in emerging locations. Local organisers have subsequently used this to inform their strategy development.
• Output: I also developed a map of different theories of change held be different stakeholders, showing connections and divergence, and possible implications of different views.

Problem

Effective Altruism (EA) is a social movement that advocates using evidence and reasoning to determine the most effective ways to benefit others. Originating in the UK and US in the late 2000s, the movement has expanded globally in the last few years. There seems, however, to be strong differences in opinions as to how EA should be developed in non-Western countries.

I started exploring the movement in 2018, and had the opportunity to speak with several leaders of EA organisations, as well as local group organisers in Europe and Asia. This is when I discovered the discrepancy of views and sensed that it may lead to potentially negative outcomes, including miscommunication around strategy development, suboptimal prioritisation and resource allocation, and increasing misunderstandings and frustration between stakeholders.

In 2019, I applied and was accepted to investigate this situation as part of a research fellowship with the Future of Humanity Institute at Oxford University. While I started with the ambitious question of “designing a thriving EA ecosystem in Asia”, I quickly homed in on a major practical obstacle to achieving this: accurate, mutual understanding between stakeholders in Asia and the West. I pivoted my research focus to creating a basis of shared knowledge that would enable key stakeholders to collaboratively answer the bigger questions such as ecosystem design.

“How might we improve mutual understanding between effective altruism communities in Asia and the West?”

Project overview

Role

I conducted this research as part of the Summer Research Fellowship co-organized by the Centre for Effective Altruism and the Future of Humanity Institute. I designed, executed and presented the research. This included conducting stakeholder interviews, creating stakeholder maps, building case studies, writing a longform forum post, and speaking at an EA conference in Singapore. I also brought on Vaidehi Agarwalla, sociology student at the time, to provide research assistance.

Project period

3 months, from July to September, 2019

Process

1. Secondary research
2. Stakeholder interviews
3. Communicating research findings

1. Secondary research

Documentation review

I started by reading through the existing literature on the EA movement, including its history, key ideas from major stakeholders (mostly Western, given historical geographic development), especially around movement and community building, and different models or frameworks that were currently being used.

Constraints of EA development in Asia

• Most of the literature around movement building was generic (i.e. not geography-specific) and recommendations seemed to be largely based on lessons learnt from EA development in Western countries
• The most recent writings seemed to focus primarily on the risks of movement growth, and generally discouraging broad outreach in new countries
• There was very little public information about EA activities and actors in Asia. Asian stakeholders rarely published writing on major EA communications channels like the EA forum.
• There were several different articles and forum posts that discussed the criteria used to evaluate EA group organizers, activities and the impact potential of a country.

These findings informed the focus areas of my subsequent stakeholder interviews. How different stakeholders viewed:

• The EA movement’s theory of change
• The state of the EA Asia ecosystem
• The way of evaluating EA development in a new location

Limitation: Asian Stakeholder Database

Next, I needed to find out who the Asian stakeholders were. There was no specific database for this information, so I compiled my own database by aggregating information across several sources:

• Spreadsheets managed by individual group organizers
• Databases managed by EA organizations
• Annual surveys conducted by EA organizations
• Local EA Facebook groups (including collecting data on their group size, age and activity)
• Organizations recommended by EA charity evaluators or job boards (e.g. GiveWell, Open Philanthropy, 80000 Hours)
• People and organizations mentioned in the interviews — this information additionally allowed me to identify connections between different stakeholders (data which I then used to develop a network map)

2. User interviews

I interviewed experts in movement and community building in Europe and North America, as well as local EA group organisers and donors in Asia.

Methodology

I conducted 21 semi-structured interviews, most lasting 60 minutes on average, covering the following questions:

The EA movement’s theory of change

1. How does EA make good things happen in the world?
2. How do you/your organisation fit into this picture?
3. If things went great, what might EA look like in 5–10 years?

Perceptions on the current and future state of the EA movement

1. What is preventing EA from getting to the ideal state now?
2. If in 5 years, nothing much has changed, why do you think that might be?

Perceptions about EA development in Asia

1. How do you feel about the development of EA in Asia?
2. How might Asia fit into the theory of change previously discussed?
3. How might you evaluate the effectiveness of EA development in a new location?
4. What questions, if answered, might change your mind about how to approach EA development?

Additionally, for Asian stakeholders, I asked:

1. Can you describe your journey in EA?
2. Can you describe your relationships with other EAs and EA organizations? (could be within and outside of Asia)

Interview findings

There were three main areas where I identified significant gaps in understanding between stakeholders in Asia and the West

Colour coding interview transcripts by themes during analysis

On the theory of change:

• People engaged quite differently with the questions around theory of change. Some took a more macro or theoretical view, some looked at culture and values, while others focussed on more concrete structures or activities.
• At times, it was difficult to tell whether different stakeholders actually agreed or disagreed. It was also unclear, at first glance, how (or whether) everyone’s views linked together to form an overall strategy for the movement.

On evaluating EA development in Asia:

• No one had a clear view on how EA should be developed in Asia and there were mixed views on the region’s importance to the overall movement.
• There was some confusion from Asian stakeholders around how their country and activities were being evaluated, some felt frustrated by the lack of support they had received so far
• There was uncertainty from Asian stakeholders around how to develop local strategies for EA development, especially on how to prioritise between different activity types

On the state of the movement in Asia:

• In general, stakeholders in the West underestimated the amount of activity happening in Asia, but acknowledged that they had limited information, and expressed a desire to get more visibility into the region.

3. Communicating research findings

Based on my findings, I put together several tools and publications to help fill in the gaps that I had identified.

Asia Stakeholder Maps

I created two types of Asian stakeholder maps (geographic and network) to provide the global EA community with an accessible overview to the Asian ecosystem.

Geographic Map

Based on the information I collected through interviews and secondary sources in the first phase of the project.

Lo-Fi prototype (Sketch on a whiteboard with post-its)

Hi-Fi Prototype (Digital map created on Kumu)

The geographical map plots all the stakeholders, and allows filtering by their type and cause area. It also allows for looking specifically at local groups and identifying relationships in their location, size, age and engagement.

Network map (created on Kumu)

I also developed a network map that showed connections between stakeholders and their relationship type. This helped identify key hubs (people and organizations) in the region.

Case studies

To help Western stakeholders understand what was happening on the ground, I “zoomed in” on the hubs and shared qualitative stories about EA development in these countries, often by mapping out the journeys of the main local organizer, including touch points with other stakeholders that influenced their journey.

Through these case studies, I found that EA development followed a different model in Asia:

• in Europe and North America, EA development often starts with student groups and organizers, then expand to city and sometimes national groups. Asian EA development is often reversed, with national groups starting first and city or campus groups emerging later.
• Also, in contrast to Western groups, several Asian groups grew by leveraging the audiences of EA or EA-adjacent organizations, or were started by staff at these organisations themselves.

Evaluation framework

On the other hand, I also wanted to help Asian stakeholders understand how those in the West may be evaluating EA development in new locations. Using information from the interviews and literature review, I tested out different ways of organizing this information that was both logical and easily applicable, say, for a local organizer to use when planning their group strategy.

Testing out different ways to visualize the evaluation criteria collected from research phase

I shared different versions of the summary table with a handful of group organizers to test and improve its “usability” before publishing on the EA forum.

Final version of the framework summary table

The complete documentation of this framework can be found in my longform article on the EA Forum: A framework for assessing the potential of EA Development in emerging locations.

Theory of change diagrams

Finally, I tried to tackle the most abstract area of misunderstanding: theory of change. I tried using different diagrams to represent and compare the mental models that were shared with me during the interviews.

Master white board where I tested different visualisations of theories and concepts shared from interviews

Eventually, I landed on a tree diagram (with some loose elements) to present different strategic approaches and connect them to potential practical implications.

I used a whiteboard presentation to co-develop the tree diagram with stakeholders in real time.

Results and reflections

Based on feedback from stakeholders, my research helped improve mutual understanding and communication across EA geographies. After the fellowship, I also received a grant from the Centre for Effective Altruism, to continue providing stakeholder research to EA organisations. I think this was evidence that I had narrowed on an important topic and this style of research had been neglected within EA. I also found some additional benefits for the overall EA movement and for my personal development.

Improving stakeholder understanding and communication

I presented my research at the Future of Humanity Institute (FHI) and was invited to speak at the Doing Good Better conference in Singapore. Here is some feedback from people who attended:

• From Matthew van der Merwe, a researcher at FHI
  “Jah Ying’s presentation on the EA Asia ecosystem was informative, well-structured, and covered a wide range of important questions. She is an excellent public speaker, and led a fruitful audience discussion.”
• From Varun Deshpande, Managing Director of GFI India
  “Jah Ying’s mapping out EA work and community building underway in these regions has already been very useful to bring together community members across Asia… Jah Ying’s particular style of interviews and analysis has been well received, and brought forward details and insights which may not have otherwise been known.”

Adoption of new tools for strategy evaluation

I published the evaluation framework as a longform article on the EA Forum: A framework for assessing the potential of EA Development in emerging locations. To my knowledge, at least three groups have applied the evaluation framework, e.g. one of the organizers of EA Spain immediately tried testing the framework to assess Spain’s strategy. Subsequently, EA Singapore and EA Philippines have also used the framework when developing and evaluating their local group strategy.

Acceptance of user-centred research methodologies

It was quite obvious as I embarked on my research at the FHI that my user-focussed approach differed significantly from the researchers around me. However, my methodologies and presentations were ultimately well-received, as illustrated by the feedback I received from senior researchers:

• “[Jah Ying (JY)] brings a somewhat different perspective to the problem — while many EAs tend to put a lot of weight on simple semi-quantitative models (the INT framework, IACPCs, various group impact models), JY’s approach is often much more user-oriented (interviews, maps of stakeholders, network maps)”
• “AFAICT, Jah Ying’s choice of an interview-based methodology was spot on given her goals, with this being particularly noteworthy given that according to my impression this and other qualitative approaches tend to be undervalued and underused within EA.”

As the EA movement continues to grow in size and diversity, it seems likely that the need and value of user and stakeholder research will increase, as inter-regional coordination and cultural differences become more prominent challenges. As a community that traditionally focussed on employing quantitative analysis to problems, I believe Effective Altruism would benefit from adopting qualitative research practices, such as user interviews, to inform and augment their existing evaluation methods (e.g. surveys). For example, this could mean conducting interviews with a sample of members and key stakeholders across regions prior to designing the annual survey, to elicit issues and questions that may not be obvious from the experiential context of the survey designers.

Personal discoveries

Finally, I also discovered some things about myself, namely an excitement and propensity for conducting stakeholder research. I think the latter was best captured in the feedback from a co-manager of the research fellowship:

“Based on observing some of the ways in which [Jah Ying] engaged with some ‘core EA’ stakeholders — including in challenging situations such as diverging views or skeptical attitudes toward her work — I believe Jah Ying is unusually strong at several communication skills… includ[ing] an eagerness of seeking out feedback, responding to it in appropriate ways such as by asking clarifying questions (as opposed to e.g. being defensive, or ignoring the feedback), and a general awareness of being in a situation where different stakeholders have different knowledge, interests, and preferred communication styles, and accurately conveying one’s views or questions in writing.”

Since the fellowship, I have been working with other collaborators in the user research space, exploring new methodologies, and applying them to serve a broader set of clients, including nonprofits, community groups and foundations.