Q&A with Grace Su on Measuring Social Impact in Myanmar
From field research to synthesizing data: a Myanmar social enterprise’s approach to learning what farmers gained
Grace Su, born in Yangon and educated in Singapore and Australia, came back to Myanmar in 2015, with one simple idea for her career.
“I wanted to work for impact, and also re-connect with Myanmar,” she said.
Previously working in the financial sector, Grace found ways to mash her wide set of skills — from being able to connect with people to making sense of a messy Excel spreadsheet — through her new position in Yangon. As manager of the Knowledge and Social Impact team at Proximity Designs, she often finds herself stepping into paddy fields, getting to know the humans behind data and hearing their struggles and aspirations.
It’s not always straightforward, or easy, to conduct a wide range social impact survey in a country as geographically and culturally diverse as Myanmar. Our conversation went deep into the process of a recent impact assessment project she led. But it started with what motivates her in her current job.
How do you see the value in the monitoring and evaluation work you are doing at Proximity?
Impact assessments are very, very important for anybody in development work. It helps the management make more informed choices. It also helps us to be more accountable. You cannot simply say that you as an organization are improving Myanmar farmers’ lives without going out there and speaking with them and having the data to prove your impact. If your work does have an impact, what attributed to this impact? And how is it beneficial?
If the farmers’ income increases, you can’t just stop there. First you have to go backwards and find out how these services are helping them increase their income. Is it because the farmers have more free time now, hence they can invest the extra time that they have to engage in a secondary business? Or is it because they are becoming more efficient and spending the right amount of input in the farm, therefore increasing their crop yield? How did they manage to have this kind of income increase?
Then you go further and ask them: what do you do with this income increase? Because it’s not as big of an impact if the farmers spend the extra money on alcohol and betel nut, right? If they invest in health and education, or repay their debt, or invest in their farms by expanding or buying more productive assets, these create bigger impact.
Going back to your question of why impact assessment matters, I think it helps our organization figure out what we are doing right, what we can improve. I think it adds very valuable input to how we design our services, which in turn could maximize our impact.
PART I: Ready, set…
The project to assess the impact of Farm Advisory Services took Grace and her team more than three months to complete. While Farm Advisory Services –– one of four business units at Proximity Designs –– goes around to provide low cost and easily-applicable techniques to help farmers farm smarter, Grace and her team wondered long and hard about how to more accurately depict the changes these techniques have brought to the people in need. The six-member team, plus a handful of volunteers, trotted their way through the Ayeyarwady Delta during a heavy monsoon season to talk with 176 sampled households to find out the answers to their questions.
Let’s talk about the survey process. When did you start? And what task did you start with?
For the first month, in July, I went through what I call “ deep understanding of the business unit.” I spent a lot of time trying to understand the function of Farm Advisory Services: What is its theory of change? How does it think the business unit is having an impact on the farmers? Then I spoke with different stakeholders. I spoke with the management. I spoke with the operation manager. I spoke with the data analysts to have a holistic understanding of why they do what they do. I also spoke a lot with the donors’ team, as donors are also one of the major stakeholders. I have to first understand what the management wants and what the donors’ needs are. I try to find that middle ground between both sides, and then design a survey that answers all the questions that both parties want.
Was it difficult finding that middle ground?
Surprisingly it wasn’t that hard because many of our donors believe in Proximity’s model. Instead of proposing to the donors that we are going to do a project, which in my experience is what many NGOs do, for us, our donors trust and support our ongoing businesses.
With farming being a volatile business, was it possible for you to have an assumption of the impact before going into the survey?
You always go into impact surveys after you understand the business and its theory of change: how they have designed the services to have an impact on the end customers. For Farm Advisory Services (FAS), it promises farmers cost-effective techniques that will improve their yield. These are the underlying theories. The impact assessment was designed to prove this assumption was true or not. I went in with the assumption that FAS is positively affecting the farmers, but I wasn’t sure exactly how much. Sometimes the farmers’ lives are very unpredictable. The can have pest problem and the whole season would fail.
What are some key elements that you were considering while designing the survey questions?
We go by two ideas. First, what are we trying to achieve through the questions that we ask? Secondly, the feasibility of the questions. Some questions sound nice on paper, but it’s not very feasible. For example, asking an interviewee whether he or she is a household head. It’s a very subjective thing. Sometimes, the man says he’s the household head, but it’s not necessarily true. Sometimes maybe the mother-in-law is the real household head.
Why do you want to know who the household heads are?
So we can categorize the households we surveyed based on genders of the household leaders.
So I can see some responses could be influenced by the local culture?
Exactly. Another example is if we ask the farmers how satisfied they are with our services. It sounds like a nice question, but it is not a feasible question, because the farmers would say every time that they are very satisfied and happy. Instead, we chose to ask them: how likely are you to recommend our products to others. That serves as a representation of how satisfied they are as our customers. We have to ensure that our questions make sense and are able to reveal informative answer, also whether they are culturally appropriate and feasible in the field.
So I’m still curious, how did you approach that household head question?
Usually we ask them who decided to adopt our services (chuckle). Many techniques that our Farm Advisory Services promotes are quite a change to the ways farmers normally approach their farming practices. It’s a fairly big decision. Whoever decides to adopt them is most likely the person who’s the household head. That’s our assumption. We look at the ones who make the calls.
How long did it take to decide on the survey questions?
About a week. First we drafted the survey before we went on the trip. When we were eventually on the trip, which was the second week of August, we did a pilot testing for the first two days in the field. We came back every night, regrouped and discussed if anyone had any difficulties interviewing farmers. If there was a difficulty regarding the question, we changed the questionnaire together. That happened quite often, as the way we framed certain questions and the way we structured the sections could be weird sometimes. These are the things we would only know when we actually carry out the interviews with real farmers. We did this at the start of the field trip. We changed on the spot and print the questions out again. It was a fairly fast and reactive process.
PART II: On the road
Here’s another thing: when you talked to the farmers, do you think the farmers understood the concept of impact? Or what your team was trying to do?
(Chuckle) Well, we told them that we were there to better understand how their farm business was doing. They knew well that we were there to learn how their crops were doing year after year.
How open are they to the idea of someone they don’t know at all asking them about their business?
They are welcoming to us. It is in Myanmar tradition to host guests, especially guests from afar, like Yangon or overseas. The farmers really enjoy being the hosts and will sometimes secretly prepare lunch for us. They are genuinely excited to meet us.
And the questions usually go well?
The way we interview is not just try to answer the question one by one. We try to engage farmers in very fluid conversations. We will pick out the information that is relevant to the questions we are asking. For the basic household demographic questions, we’d just ask: “how’s your family doing?” They would answer: “oh, my daughter is well. My parents are well.” We can actually pull out a lot of relevant and basic information that ways. Everybody does interviews in their own way, but we make sure for certain tricky questions, we are consistent in the underlining assumptions that we take. When we asked for income and cost, for example, we made sure the five major categories are filled out, like fertilizer, labor cost, petro cost, seed cost and pesticide cost. These things are very specific, so we try to make it very uniformed. We are OK with however we ask them and in whatever sequence. It’s up to how the conversations flow.
Individual conversations vary, but that’s to ensure the authenticity of the information you receive from the farmers. Is that right?
Yeah. If we ask them question after question, it might be too imposing on the farmers. The farmers might think that we are coming there to check on them, and the answers might be guarded. The farmers might try to give you the “correct” answers.
This three-week road trip for interviewing farmers, how much planning went into that?
We really couldn’t plan so much ahead. We had the skeleton of the trip planned, but we didn’t have the day-to-day planned until we get to the townships. We don’t have the village-level information. When we get to the townships, we work with the field staff, and the field staff will help plan the trips. They have a very good idea of which villages are located beside each other, so they can help plan more efficient routes.
PART III: Looking back
Interviews make up the important data-gathering process, but the work is far from being over. With the hundreds of pages of completed survey, the team regroups in Yangon to start compiling data and conducting analysis.
After the team came back to the office and synthesized the data, how did you decide on the metrics you used to showcase the impact?
First, we need to answer the questions of what do we care and what do we want to find out. When there’s an income increase, we need to go beyond “this is the financial impact on the farmers,” to where did they spend this money. Because we need to understand what their spending patterns are after they have this increase in income, which gives them the ability to make more choices. It would be very helpful if we can understand how are the farmers are changing the way they spend: Are they investing more in their health by buying more protein-based food, or in their children’s education? These are the indicators that Proximity as an organization cares. We want to know further beyond the financial impact, but more importantly the life improvement aspect.
Answers edited for content and clarity.
#ProximityPeople
