Digital Literacy: A Must For Inclusive Research

Digital literacy concerns aren’t an edge case. They’re the norm — and the future. Here’s how and why to address them in your research.

By Rebecca Van Roy

Darius points to his phone screen perplexed, wondering where all his pictures of jewelry went. He had hundreds of them. Now they were all gone.

We spoke with Darius in his small apartment in Fort Lauderdale, Florida. Due to health ailments, he was retired and mostly housebound. He used Facebook to find friends and post pictures of jewelry he sold to make money while he was ill. He thought his Facebook had been hacked, he told us, because he couldn’t find the pictures he’d posted of his jewelry over the last few years.

We helped him navigate to the right part of his profile. As he scrolled, several jewelry pictures slid onto his phone screen.

He couldn’t believe what he was seeing, as he excitedly scrolled through the photos. He sat back in his chair and smiled, realizing they were all still there.

All of Darius’ pictures were, in fact, still there — but they were located many clicks into his profile under the light-gray button “Uploads”. To Darius, who struggled to master the digital interface of Facebook, they’d been gone.

While he had some idea that these pictures were located on his Profile, he apparently didn’t understand the concept upload, so he had no idea that he should click there to find his pictures. As with many other apps and technologies, using Facebook products requires a certain level of digital literacy.

A damaging oversight

Defined as the ability to understand, navigate, and participate online in a safe way, digital literacy fundamentally shapes the way people in all populations and places use technology. It determines the extent to which people can realize the full economic and communal potential of social media products like Facebook.

In developed markets like the US, digital literacy is often a gap in research. Product teams, and even researchers themselves, often think we’ll uncover problems with familiarity and comprehension only in emerging markets. Yet it lurks beneath the surface of demographically minded survey variables and recruitment criteria, mingling with education and income. When we really look, we find it everywhere.

Here’s how and why researchers should address digital literacy in both qualitative and quantitative research.

Why digital literacy matters for product strategy

In 2019, we conducted a survey to understand the digital literacy levels of the global Facebook user base, using a self-efficacy and comprehension scale. We found that a sizable minority of the global Facebook-using population had low digital skills: 24% of the 38,085 respondents who completed all the self-efficacy questions rated their own understanding of key social media concepts as “some understanding” or less, while 41% of the 30,903 respondents who completed all five comprehension questions got less than half of the questions right.

While this work showed that digital skills are significantly lower in non-US/EU countries, it also showed that digital skills are lower within many other populations, including people who are older, newer to social media, and who use out-of-date devices.

The danger of designing for ourselves

When it comes to digital literacy, the gap between social media users and tech employees is vast. We often overestimate our peoples’ digital abilities. When we build products and designs based on what we can do, we risk leaving behind large segments of the population.

By recruiting people with low digital literacy for qualitative studies, we’ve been able to pinpoint key usability and comprehension issues quickly and efficiently. And by quantitatively measuring how product usage varies by digital literacy level, we’ve also made sure our product roadmaps and strategies account for the types of concerns and problems that people with lower levels of digital literacy tend to experience.

Addressing digital literacy in research can also help companies future-proof their products. Emerging markets, which tend to have lower levels of digital literacy, also have the most growth potential. The same can be said of low-income adults in the United States, whose smartphone adoption is significantly lower than that of high-income adults. Designing products for people in these groups can lower barriers to entry for social media while protecting the safety and privacy of all our users.

Bridging the digital divide

Many research methods, both qualitative and quantitative, can exclude people with lower levels of digital literacy. Panels may contain people with high levels of education or less racial diversity, while our reliance on digital tools for remote research during COVID-19 has carried the risk of excluding people who don’t have reliable access to technology or the internet.

We recommend the following tips to ensure that your research includes people with all levels of digital literacy:

1. Sample thoughtfully

To include people with lower digital literacy from the get-go, consider reaching out to and including groups who:

• Are older
• Are newer to social media or the internet
• Live in emerging markets
• Use Android phones
• Use low-end devices
• Use a small number of devices and social media apps

During a qualitative study in the San Francisco Bay Area, for example, we were able to include several participants with low digital literacy by running extra rounds of recruiting with older adults, and with people who were using older or less capable Android phones.

2. Don’t make technology a barrier

People with lower digital literacy are often harder to reach, because they often have limitations with technology access or use. You can make your research more inclusive by allowing flexibility when it comes to the interview equipment. Allow participants to use their phones instead of webcams, for example, or send data packs if internet affordability is an issue.

If you have the time and budget, you can also do more recruiting through phone calls, local outreach and intercepts, and snowball sampling. To reach participants with lower levels of digital literacy in Memphis, Tennessee, we recruited by putting up flyers at local Dollar Stores and Walmarts. We also conducted intercepts at convenience stores, malls, and bowling alleys.

You might also consider working with vendors who specialize in recruiting hard-to-reach populations, have contacts with local communities, or are equipped to do more hands-on recruiting.

3. Meet people where they are

Technology is often intimidating to people with lower digital literacy, so it’s important to simplify research tools and materials. Try to keep screeners and surveys short, avoid jargon, and write questions and materials for a 5th- or 6th-grade reading level.

In usability testing with low-digital-literacy participants, for example, we made sure the task instructions were clear and direct, avoiding Facebook-specific terms.

• Instructions for joining a group: “Become a member of this group on Facebook: Positive Inspirational Quotes.”
• Instructions for reporting a post: “You see something on Facebook that is inappropriate, and you want to notify Facebook about it.”

Another way to meet people where they are is to make yourself available for support along the way. This can also build rapport and help participants feel comfortable. Consider engaging with participants via more personalized forms of one-to-one communication, such as phone calls, text messages, or WhatsApp threads.

4. Measure digital literacy with the right questions

Many assessment methods like the ones shown in this academic article and book, can help make sure your sample includes diverse levels of digital literacy. Drawing from these methods, you can include questions in recruitment screeners to measure digital literacy. We recommend using two types of questions:

• Questions that measure self-efficacy. These questions ask respondents to rate their own abilities. An example: “On a scale of no understanding to full understanding, how familiar are you with the internet-related term ‘hashtag’?” You can also use questions like those found in this academic paper.
• Questions that measure comprehension. These questions ask respondents about digital concepts, or to complete tasks like searching for information on the internet. A good example is PEW’s cybersecurity knowledge quiz.

When choosing or creating questions, it’s important to recognize that many are prone to bias and error. For example, although women may not differ from men in performance-based skills, they sometimes perceive their skills to be lower than men’s. Whatever questions you use, make sure you pilot them with participants and try to make them as easy to understand as possible.

An ongoing challenge

Keeping digital literacy at the center of research and product development requires sustained, deliberate effort and investment. For researchers, it can mean more work, longer timelines, and additional rounds of outreach. But the more we do it, the more we recognize it for what it is: a fundamental part of ensuring that everyone can use our products easily and safely.

Author: Rebecca Van Roy, UX Researcher at Facebook
Contributors: Nadine Levin, former UX Researcher at Facebook; Carolyn Wei, UX Researcher at Facebook; Kim Brink, People Research Scientist at Facebook
Illustrator: Drew Bardana