KaiAnalytics
Published in

KaiAnalytics

Design, Refine, and Implement More Ethical and Culturally Sensitive Surveys

Image of people on a busy street of a city in the Philippines
Kai Analytics conducting research on the coffee industry in the Philippines
  1. Respect for your local partners — Work with local partners every step of the way and adhere to a place-based approach.
  2. Respect for your participants — Make sure that your surveys are accessible and safe for your participants.
  3. Working together — Your participants and their communities are partners in research. Respect their time and knowledge rather than extracting from it.

SURVEY DESIGN

1) Finding a Representative Sample

Image of people walking on a busy street in black in white (probably somewhere in North America)

2) Check social norms and domestic laws

A woman in a hijab sitting outside and working on her laptop
  • Talking to locals — By talking to locals you can get valuable insight about your population of interest. They will be able to answer questions such as what is culturally appropriate to ask and what kind of topics to avoid.
  • Pre-testing — Once you pre-test your survey with your population of interest, you might discover that some questions do not make sense or are not culturally appropriate (we discuss this later in the blog). This is a great opportunity to communicate with your participants about how to improve your survey.
  • In the field — For in-person surveys using enumerators, you might realize that a question is not phrased properly only once it is live. At this point, you cannot go back and change the question, but you can make a note of which question caused confusion or was not well received by your participants, so you can take that into account when you are analyzing your results. It may be possible to rephrase the question and use simpler words but be careful not to change the meaning of the question.

3) Get permission from authorities to conduct the survey

Scrabble letters put together to form the word law
  • Participants might be forced to take the survey — In some cases participants might be forced to take the survey by the authorities or they might feel pressured to do so. There could also be some sort of benefit within the organization (e.g., career advancement, rewards) if they take the survey. This means that your participants are giving you consent that is not entirely voluntary (check the informed consent section below).
  • Participants might be discouraged from taking the survey — On the other hand, you might find yourself in a situation where the authorities actively discourage your participants from participating in your survey. In that case, you should inform the people of the views of the organization’s authorities and if they could face any consequences if they do decide to participate.

REFINE YOUR SURVEY

  1. Choose the right words
Wooden scrabble letters
  • Skipped questions — Skipped questions are one of the most telling indicators that your participants did not understand the question. People might feel embarrassed to admit that they do not understand what a question requires from them and simply skip it. This means that it is vital to use lay language, simple language that the general population is likely to understand, when designing your survey. When questions are missed, not only will you as a researcher not get the information that you need, but you might also be forcing your participants to give uninformed consent (see below). For reference, only 20% of adult Canadians read at a college level and when people are ill or stressed, their literacy level drops even lower .
  • A lot of “I don’t know” or “neither agree nor disagree” answers — If your survey has an “I don’t know” option, your participants are probably going to choose that if they do not know an answer to your question. However, if you did not include this in your survey your participants might be choosing “neither agree nor disagree,” as it most closely resembles “I don’t know.”
  • A lot of “Not applicable” or “NA” answers — If participants feel or believe that the question does not apply to them, they might be choosing the “Not applicable” or “NA” options. This might indicate to you that the way that you have worded the question does not signal to your participant that it is relevant to them.

2) Understand when “Neither Agree nor Disagree” is a Genuine Answer

A man filling out a document
  • Abnormally long answering times — If you have the option of tracking how long participants spend on a particular question, watch out for long hesitation times. If participants look stuck on a question, this might be an indicator that the question is too complex, confusing, and needs rewording. However, this could also mean that your participants are simply distracted with something else.
  • Abnormally short answering times — If you think that your participants are answering your survey too quickly, it may be because they do not understand the questions. They might also not have a genuine interest in answering properly and honestly. They could be in a rush, forced to take the survey, or might not recognize any benefit of taking the survey to them.
  • Question clarity
  • Length of the survey
  • Sensitivity and relevance of your questions
  • Test question styles — If you have asked the same question in two different ways (e.g., a Likert scale question or an open-ended question), you can also ask your participants which version of the question was easier to answer during debriefing. You may discover that your participants strongly preferred one style of a question to another, thus, giving you insight to design a survey that is better suited for a given population.

IMPLEMENT YOUR SURVEY

  1. Get Truly Informed Consent
A person signing a document
  • Allow participants to skip questions if they are feeling uncomfortable answering them.
  • Make sure that your participants know that they can withdraw from the survey at any time and write that into your survey (e.g., make sure that this is worded clearly and that there is an option to opt out on every page).
  • Explain what the survey is about, why you are conducting it, what type of questions are going to be asked, if you will ask questions about sensitive topics, a privacy statement, and if there are any risks to the participants.
  • Share the time and data that the survey will be completed and available.
  • Remind them that their identity is kept confidential, and clearly mention if otherwise.
  • Provide details on how the results will be collected and assessed.
  • If or how the results will improve the livelihoods of people that are participating in your survey.
  • Give your participants opportunities to raise concerns and address those concerns accordingly.
  • Monetary Incentives — Incentives are a great way to encourage more people to take part in your survey but be weary of offering large sums of money. Not only could that incentivize people to take a part of your survey for the wrong reasons, but it might also put pressure on people that are financially disadvantaged to participate in your survey. This could lead to risks for them participating that they might be willing to overlook due to the monetary reward.
  • Social Incentives — Avoid creating an impression that it is “cool” to participate in your study or that participating can in any way enhance someone’s social image. For example, if a student feels (and this might not be the reality) that participating in a professor’s survey can enhance their grade or on the contrary, their studies will suffer if they do not participate, their consent will be compromised.
  • Ability to consent — Some participants may not be able to consent due to limited or impaired cognitive abilities as they will not be able to make a fully informed decision whether to participate in your survey or not. It is possible that by law, these people will be allowed to consent to participate in your survey. Yet you should consider whether these people can:

2) Work around internet availability

A woman in a red sweater holding a survey on a clipboard
  • Carry out your survey in-person — To carry out a survey in-person in a hard-to-reach population you will need to rely on local partners. Find trusted enumerators that are experienced in socio-demographic research techniques that will deliver your survey to the right people.
  • Telephone survey — If your population of interest does not have access to the internet, you still might be able to conduct your survey remotely by phone. While it is a great option if you do not want to or cannot travel to the area where your population of interest is, sometimes the fastest way to get your results is to administer your survey in person.
  • Instant messaging software — In many communities Kai Analytics have worked in, Facebook messenger was the DeFacto way to communicate. Even with spotty internet connectivity, we found out the farmers we interacted with are still able to asynchronously communicate back and forth. Ensuring that you have the data privacy measures in place and the appropriate messaging software for the country context you are working are important considerations.

3) Demonstrate Empathy

An old man and a young woman hugging

--

--

We help businesses understand text-based feedback. We can clean, parse, and categorize thousands upon thousands of comments and reviews into major themes. Powered by natural language processing and machine learning.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store