Free-listing: the greatest research method you’ve never heard of

Free-listing is a fascinating but under-used research method that allows us to explore the mental models and cultural knowledge of groups. It tells us how people categorise their world.

It’s a close relative of card sorting, in a family of methods referred to as cultural domain analysis. It’s a powerful tool for generating insights — however, 99.9% of user researchers have never heard of it. Time to put an end to that…

How you do it

Free-listing is a quantitative method, but relies on qualitative, free response data.

You give participants a concept, and ask them to make a list of up to 10 words associated with that concept. You then rank the words according to their “saliency” — a measure of ease of recall and repetition between participants.

This can be done remotely or in-person.

You need at least 20 to 30 participants, who belong to a single group. Sometimes it’s worth getting multiple groups to complete the same listing exercise.

Example

Let’s say we’re looking to elicit words associated with AI, to create an engagement strategy for encouraging AI use in Kainos’ Experience Design team.

To create a strategy tailored to discipline, we might get content designers and UX designers to complete the task, and plan based on the differences and similarities between the results.

Groupiness

Defining a group can be tricky, but you needn’t think too much about this. As long as the participants would define themselves according to that group and, can reasonably be considered to be so.

Take our example above: a user researcher who worked as a UXer for a decade and continues to fulfil that role on certain projects could be reasonably included in our UXer sample, but let’s exclude anyone who thinks FigJam is a condiment that goes great with cheese and crackers.

Analysis

You then code the contents of the list. Synonyms are coded as the same word — for example “innovative” and “cutting-edge” might be altered to “innovative”.

This requires discernment. WALL-E and the Terminator could both be coded as “robot” — yet their vibes are world’s apart: cute WALL-E has little in common with the menacing Terminator.

Think about what your research question is: when understanding designers’ concept of AI, threat is an important element. In this case then we would not conflate the two.

Working out the salience

Using R (a coding language for inferential statistics) the researcher then works out the salience score.

This calculates saliency as a function of how often a word occurs between lists, and how high up a list it appears on average: the more it is mentioned, and the earlier it is mentioned, the higher the saliency. Saliency is therefore defined as the level of sharedness and ease of recall.

Output

The result is a table showing the saliency score of participant-generated terms in relation to the concept. It’s a ranked, quantitative score of which words are more salient to a target.

Think of our exercise above. Content designers listed “risky”, “innovative”, and “potential” frequently, and often quite high up the list; this is where a traditional listing exercise would stop. But with free-listing, we can rank these terms to see how psychologically salient they are in relation to AI.

Content designers’ concept of AI

The top five words listed by content designers about AI were:

• “Technology” — .374 saliency — 58.9% frequency
• “Potential” — .333 saliency — 53.6% frequency
• “Risky” — .305 saliency — 41.1% frequency
• “Wall E” — .139 saliency — 23.2% frequency
• “Innovative” — .128 saliency — 19.6% frequency

Here the salience of “potential” is higher than that of “risky”, and “innovative” is lower than both.

UXers listed many of the same words as content designers — however while “risky” has roughly the same saliency score between the two groups, with UXers “innovative” has a higher score than both “risky” and “potential”.

UX designers’ concept of AI

The top five words listed by UX designers about AI were:

• “BingCoPilot” — .422 saliency — 44.4% frequency
• “Innovate” — .368 saliency — 44.4% frequency
• “Risky” — .255 saliency — 55.6% frequency
• “The Terminator” — .222 saliency — 22.2% frequency
• “Potential” — .199 saliency — 33.3% frequency

The different hierarchies in these lists might suggest that, as a result of a different mental model, a comms strategy to encourage the use of AI among UXers should emphasise its innovative potential. For content designers, a good approach could be to emphasise its latent potential, and work with them to unlock this.

Why use free-listing

Simplicity and efficiency: No complex training or equipment — just a pen and paper (or Microsoft Forms).

Reducing bias and being culturally receptive: Participants can freely provide responses, reducing researcher bias and allowing diverse groups to express mental models in their own terms.

Best of both worlds data: Despite being based on qualitative data, it provides a numeric value for terms — allowing comparison and ranking while not limiting responses to pre-set options.

Insights into group psychology: Free-listing provides an framework which balances sharedness with ease of recall. It helps us to understand our users as people whose mental models are both individual, and a result of the social group they belong to.

When to use it

The only limitation on free-listing is whether users have an existing mental model of a concept. This means it works best in discovery (applied to the problem space) or private beta/live (applied to longer term use of the product or service).

It’s rarely useful in alpha, as it cannot tell you about something which has not been part of users’ lives for long.

Conclusion

There you have it — the greatest method you’ve never heard of. Simple to execute yet rich in insights. It’s fast and cheap to deploy. It combines what is best about qualitative and quantitative. What are you waiting for?