Understanding The Influence of Culture on TalkLife
In this post, Sachin Pendse (a Research Fellow at Microsoft Research India) describes his research on cultural differences in how people express themselves when they are feeling distressed on TalkLife. This work is done in collaboration with Yada Pruksachatkun and Amit Sharma.
Through facilitating a form of empowerment through expression, online spaces designed for people to speak openly about their mental health (such as TalkLife) have enabled something rarely seen in the past. Individuals from all over the world, across borders and time zones, are able to cross these boundaries to support one another in the experience of hardship and distress. Speaking about your most intense and distressing experiences can be a deeply personal act, and as we know from past work done in anthropology, doing so is highly influenced by your identity — your race, gender, cultural background, class, and other attributes.
We began our study of how people from around the world use TalkLife to see whether we could use different factors from conversations to predict whether and when someone would say that they feel better during the course of an ongoing conversation. To do this, we created a de-identified dataset of TalkLife conversations where people said “I feel better now!” or “You have a point,” which we called a “moment of change.” Based on attributes from the first posts of the conversation (such as the presence of question marks, the number of words related to mental health used, or how positive or negative the words used are), we tested an algorithm to see if we could predict when someone would say if they felt better.
When running our experiment, we found that we could! And our results (measured in AUC, a commonly used metric for rating the performance of machine learning algorithms) were quite good! We were able to achieve AUCs of around .85, an excellent result.
However, given this past research on how much your expression of your mental health is driven by your identity (what is often called “idioms of distress”), we tried to see whether there would be any change in performance if we used an algorithm fit to conversations that were started by non-Indian users to try to predict when someone would feel better in primarily Indian conversations, and vice-versa. We used Indian users and non-Indian users because we are based in India, and were understandably curious.
Surprisingly for us (but maybe not for a medical anthropologist), there was a significant drop in performance. Algorithms fit to one group and tested on the same group had an AUC of approximately .88, whereas algorithms fit to non-Indian users and tested on Indian users (and vice-versa) had an AUC of approximately .71. This drop in performance shows that people from different communities use similar forms of language (such as punctuation or the positivity of the words used) when expressing if they are feeling better, but that they use this language differently based on identity.
To better understand what these differences were, we did a deeper dive in examining how Indian users expressed themselves on TalkLife by directly comparing their expressions with users who weren’t Indian.
We found that:
- Users from India were more likely than the rest of TalkLife to talk about their country of origin in their first post, with a difference of 1.2% and .4% respectively.
- Posts from Indian users were less likely to have clinical language in them (such as “panic attack” or “anxiety disorder”) than the rest of TalkLife, with a difference of 14.2% and 22.2% of questions and 6.6% and 8.6% of answers respectively.
- Posts from Indian users are more likely to talk about connections to other people (such as feeling alone) than the rest of TalkLife, with the most common words used including references to needing a friend among Indian users, but not among other users.
- Indian users are much more likely to seek support from other users from India, with conversations started by Indian users having 16 times more Indian users on them than average.
- Drawing on this result, we find that peer support along these identity-based lines is also more successful. Conversations with “moments of change” started by Indians have 10.5 times more Indians than average, whereas conversations without moments of change started by Indians have 6.5 times as many Indians compared to average.
Our work shows that just as expected in offline contexts, when we go online to express how we are feeling, we bring parts of our identity with us. Our work also shows that these differences very deeply matter, both for machine learning algorithms and how we come to support each other.
For More information visit:
Download TalkLife Free for iOS Here
Download TalkLife Free for Android Here
