“Are We There yet?”, Changes in Sentiment Towards Mental Health
Firstly, I would like to thank Charissa Rentier for her work that helped form the bases of this research. The API provided a lot of metadata, but a web scrapper was used to gather article text (student subscriptions for the win!) However this data is restricted and therefore has been left out of my GitHub repo.
My Toolkit
- YanyTapi to gather articles through the NYT API with the keyword ‘mental health’ across a 10 year period from Jan 2010 to Dec 2019
- NLTK to remove duplicates (based on article ID). To also perform Tokenization, Lemmatization (which scored better than Stemming), and minor data manipulation for easier processing later
- Vader Sentiment for sentiment analysis, scoring the data from [-1,+1] whereas some other tools used machine learning labelling style (but required a lot more pre-labelled training set.
- Plotly Express in Jupyter Notebooks for quick interactive graphs. It was easy to setup with some small interactive filters viewers can easily toggle.
Overview of Articles
Due to the different HTML structure of article types (Interactive, Blog, Dealbook, Opinion, Article) I have chosen to focus on Article and Opinion type articles. Looking at articles containing mental health vs all other articles:
Over the 10 years, there has been an increase (almost double) in articles mentioning mental health, while a large decline in overall articles from the NYT. However, some articles may be returned for containing the keyword ‘mental health’ but may not be about mental health. There would be multiple ways to weed out these articles, I opted to count the occurrences of mental health and establish a threshold.
Establishing a Threshold
This heat map shows the number of keywords per month from 2010–2019. The ‘Total’ is the count of all articles for that month. This clearly shows 2 key findings.
- From 2016 forward, the discussion on mental health has increased, and
- 2) December of every year has little to no mention of mental health.
Looking at the average count of ‘mental health’ per article per month, 1.7 appears to be a good threshold ( since we can’t have decimal counts, we will round to 2 or higher).
Preprocessing — Topic Modelling
The first step was to perform tokenization, splitting each article into tokens (words), then removing stop words (a, he, the, in) that would provide no value to creating a topic. Additionally, it uses Part of Speech Tagging (POS or POST) to identify it with their appropriate tags (noun, verb, adjective, adverbs, etc) this helps with Entity Removal to clear people, companies, and locations from the articles text.
In terms of topic modelling this helps reduce computation time as there are fewer words to cross validate. After Tokenization and Entity Removal, the program will lemmatize the words. This is a method in which text is transformed to its core meaning. Both Stemming and Lemmatization changes the word to its root meaning but stemming will do so by removing parts of a word, cutting it down. For example “studies” becomes “study” with lemmatize but “studi” with 22 stemming. Lemmatization also scored more accurately in sentiment analysis.
Vectorization then creates a Count Vector to find similarities between words and reduce the dictionary, thus further reducing the text space we need to analyse to find our topics. For example, “Man” and “woman” would be close together as would “king” and “queen”, for our texts we could use this to limit “studies” and “research” to one word, giving us a more accurate set of words for each topic.
Topic Modelling
This involved passes over the texts to find topics. It starts on 10 topics and works up to 20 expected topics. This range was chosen as anything less than 10 topics would yield little insight expected from the corpus, and more than 20 would dilute the meanings of topics too drastically (for example politics could change to 3 new topics; “state-level politics”, “National level politics”, and “international politics”. With 14 topics, limiting the associated words returned per topic to 5 we get [(‘gun’,0.07), (‘blog’,0.04), (‘shooting’,0.03), (‘school’,0.02), (‘law’,0.02)], this can then be identified as “Guns”. The resulting list of 14 topics were:
- Research
- Guns
- Healthcare
- Politics
- Family
- Veterans
- Entertainment/Arts
- Gender/Sexual Healthcare
- Government
- Judicial
- Prison
- Property/Business
- Education
- Disability
Sentiment Analysis
The next step is to gather sentiment of the articles. This used a combination of NLTK to manipulate the text and Vader for sentiment analysis as it had better scoring metrics. Looking at scoring the plain text, for example using an article we have the text (opening paragraph only) of:
Young children in military families are about 10 percent more likely to see a doctor for a mental difficulty when a parent is deployed than when the parent is home, researchers are reporting Monday in the most comprehensive study to date of such families’ use of health insurance during wartime
- Without adjusting the text above, it gives a score of
Text score: {‘neg’: 0.086, ‘neu’: 0.865, ‘pos’: 0.049, ‘compound’: -0.952} - Adjusting the text with entity removal, stop word removal gives a score of
Adjusted Score: {‘neg’: 0.114, ‘neu’: 0.784, ‘pos’: 0.102, ‘compound’: 0.0847} - Stemming the adjusted text, gives a scoring of
Stemmed Score: {‘neg’: 0.104, ‘neu’: 0.826, ‘pos’: 0.07, ‘compound’: -0.7624} - Lemmatizing the adjusted text, method I used:
Lemma: {‘neg’: 0.111, ‘neu’: 0.786, ‘pos’: 0.103, ‘compound’: 0.3246}
Looking at these we can see the changes that occur, after reading a number of articles the scoring after lemmatization felt the most appropriate. To get the best understanding of sentiment, I investigate using the compound score as this gives a better idea of overall score.
Whereas the negative score for example only shows how negative the article is on a scale of [0,1] but compounds scale is [-1,+1]. The scoring works off of each word and its overall impact (i.e. context is not evaluated). Therefore I investigated which sentences appeared to make the evaluator score positively or negatively.
What influenced the scoring most was whether an article was Solution Focused (positive score) or Problem Focused (negative score). So, any shift sentiment towards positive is more accurately described as saying “Mental health is discussed in a solution mindset” rather than “Mental health is viewed more positively”.
Findings
From the above graph, we can see that sentiment towards mental health with Guns and Prison are negative (i.e. Problem Focused, as described above). Whereas Research and Healthcare have slowly become more positive (i.e. Solution Focused).
Considering the topics they are discussed within, it would be expected that any breakthroughs within Research would discuss the problem and a new solution. This is most likely why there doesn’t appear any great shift over the 10 years.
Whereas Guns tend to be problem focused (reports on gun violence, mass shootings, rather than any benefits, such as a sense of security to some, that comes with guns).
If we investigate just the sentiment on Guns topic, we find one of the spikes is in Feb 2018. Looking at these articles within the results shows a spark in gun control conversation after the mass shooting in Parkland, Florida. This came only months after America’s largest mass shooting in Las Vegas, October 2017 and Pulse Nightclub in June 2016.
Before, ‘Gun’ articles focused on deaths or altercations involving guns but began shifting to gun control. These articles refer to the mental health of the shooter and lack of treatment / healthcare. The spikes in negative sentiment within ‘Guns’ correlate to mass shootings.
“Mass shootings by people with serious mental illness represent less than 1% of all yearly gun-related homicides. In contrast, deaths by suicide using firearms account for the majority of yearly gun-related deaths.” — Mass Shootings and Mental Illness, James L. Knoll IV
Change in Sentiment
If we look at the overall mean compound score the trend appears to decline towards 2014 but makes a quicker recovery in the preceding years, even crossing slightly into the positive. This shows that sentiment towards mental health in online media appears to be improving overall, and at a faster rate than it declined between 2010 to 2014.
Both sentiment and count of articles are on the rise. The overall compound score has just crossed the 0 mark, at 0.15, but is enough to be out of the neutral range [-0.05,+0.05] and can therefore be considered positive.
Overall, this project has found a stigma towards mental health within certain topics. For example,the misconception that the relationship between people with mental illness and gun violence is represented significantly by mass shootings is perpetuated by online media when in fact less than 1% of yearly gun-related homicides is related to mass shootings by people with mental illness. While the shift in sentiment is becoming positive some topics continue to have their stigma towards mental health.
Future Work
This was taken on as my final year project and had limitations in terms of processing power (personal laptop), as well as time constraints. I’d recommend anyone interested in researching this further to look at doing:
- A comparison between newspaper sources
- Distinguishing more clearly between articles pulled through API for containing a keyword, compared to articles which fully focus on mental health
- Try another method of Sentiment Analysis, there are tools which use labelling instead of scoring.
