Financial News Sentiment Analysis using FinBERT

I plan to use the pretrained financial news sentiment analyzer known as FinBERT to predict whether a given headline is negative, neutral or positive.

But before we do that let us learn a little bit about the BERT, its structure and training process.

You can also view the application of this concept on the news extracted from stock news website here : Real-Time Stock News Sentiment Analyzer

Who is BERT?

BERT stands for

Bidirectional Encoder Representations from Transformers.

BERT is built on an attention model known as Transformers. Transformers prove to be better than LSTMs in NLP as they are faster, and they learn language and context better.

Essentially Transformers contain 2 parts: Encoders and Decoders

Encoders learn what is the language is, its grammar and the context. The Encoder generates embeddings (Numbers for words-Similar meaning words have closer numbers)

Decoders use these embeddings to generate the next words one by one.

BERT is model trained using encoders from transformers as its building block.

Training of BERT is divided into 2 tasks:

Task 1: Learn the language.

Here the BERT model goes through 2 unsupervised learning steps:

Mass Language Modelling and Next Sentence Prediction.

In MLM the model is given random sentence with a few words masked or replaced by a [Mask] token. BERT then tries to predict the embeddings for these words. In this way it learns the language and grammar.

In NSP step BERT is given 2 sentences, Sentence A and Sentence B and told to predict whether sentence A precedes sentence B in order. This way BERT learns the context of the language.

Task2: Fine tune the model to learn a specific task.

This task varies according to the application BERT is going to be used for.

Applications of BERT :

1. Neural Machine Translation
2. Sentiment Analysis
3. Question Answering
4. Text Summarization

For more information on transformers, BERT and its training process watch this easy to understand YouTube video by CodeEmporium:

BERT Neural Network — EXPLAINED!

FinBERT

FinBERT Pretraining Architecture from https://www.ijcai.org/proceedings/2020/0622.pdf

FinBERT is built on the standard 2 stage (Pretraining and Fine-Tuning) BERT architecture. As mentioned above first step in training BERT involves Masked Language Modelling and Next Sentence Prediction. FinBERT differs slightly over here. Instead of 2 steps in pre training it uses 6 unsupervised pre-training steps divided into 2 categories namely:

Basic Level:

1. Span Release Prediction pre-training task
2. Capitalization Prediction pre-training task
3. Token Passage Prediction pre-training task

High Level:

1. Sentence De-shuffling pre-training task
2. Sentence Distance pre-training task
3. Dialogue Relation pre-training task

Training Data used to Train FinBERT:

1. Original BERT training Data English Wikipedia and BookCorpus (Zhu et al., 2015)
2. Financial News from Financial Web
3. Finance Articles from Yahoo Finance
4. Question-Answer pairs about financial issues from Reddit

FinBERT Supervised Fine Tuning for Sentiment Analysis:

As mentioned above every BERT model has to be fine-tuned to fit a particular task in its training stage. For Financial Sentiment Analysis the following steps were taken.

1. Detect Target Aspects in text.

2. Predict sentiment score for each aspect; continuous numeric values from +1 for positive to -1 for negative.

Above is a very simplified explanation of FinBERT. I recommended you to read the research paper on FinBERT to get a more comprehensive understanding of the development of this model. You can find the research paper on this link:

FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining.

Using FinBERT for Sentiment Analysis

Requirements:

1. Pytorch
2. HuggingFace Transfomers library
3. Pandas
4. Numpy
5. Sklearn

The two basic libraries we need to load the csv and preprocess the data.

To load the csv file we need to use the pandas read_csv() method. Since the file is ‘ISO-8859–1’ encoded we need to mention that as the encoding parameter otherwise it will throw an UnicodeDecodeError.

As you can see above there are 2 columns in the data; headline column containing the news in text format and the sentiment column containing the sentiment for that headline.

We need to separate the data into 2 separate lists for ease of use in the future.

The the above code will separate the headline and sentiment column into 2 lists X and y respectively. As the model is pretrained we do not need to split that data into test and train sections.

Moving to the good part.

We will now load the FinBERT model using the HuggingFace transformers library.

Since FinBERT outputs sentiments in numeric form lets create a dictionary to map these outputs in the format we need.

Finally we can get our output by running the list of finance news (X) through the tokenizer and then through the finBERT sentiment analysis model.

The BertTokenizer object will perform all the preprocessing tasks required as per the good NLP practices before passing the the output to finbert.

Finally we can evaluate the accuracy of FinBERT

The accuracy I obtained was around 80% which is quite good considering the esoteric nature of our topic.

You can get the entire code and dataset on my git profile here:

GitHub - Raviraj2000/Financial-News-Sentiment-Analysis-using-FinBERT

Thanks for reading!😀