OpenAI vs Watson NLP: Sentiment Analysis

Exploring the Pros and Cons of OpenAI and Watson NLP for Sentiment Analysis Tasks

In the last blog on OpenAI vs Watson NLP: Which is Better for your enterprise, we have seen a high-level overview of the comparison between OpenAI and Watson NLP for various NLP tasks. Now, we will dive deeper into comparing the two for Sentiment analysis.

Sentiment Analysis

Sentiment analysis is available in both Watson NLP and OpenAI out of the box using pre-trained models. The performance of the Sentiment classification model is comparable in both Watson NLP and OpenAI.

Here is an output from the OpenAI sentiment classification tasks when all the sentences in the document were properly numbered.

OpenAI sentiment output 1

Watson NLP sentiment output

However, the major difference lies in the usage pattern. If we pass the raw document with the following prompt:

Classify the sentiment "Text..."

The output is Sentiment: Neutral . So, in order to extract the sentiment for each sentence, the next logical step is to change the prompt to

Classify the sentiment in all the sentences "Text..."

The output is

Sentiment 1: Neutral 
Sentiment 2: Negative 
Sentiment 3: Positive

You can also increase the maximum token length to 3700 (the token length also includes 366 tokens in the prompt) to get the sentiment for each sentence in the document. Therefore, with this prompt:

Extract Sentiment from each sentence in the following text with confidence score:\n{text}

You will get sentiment for 24 sentences in a document containing 30 sentences. This exercise was explained to show how difficult it can be to customize the prompt and parameters to get the desired output (table shown below).

Sentiment output variations

Additionally, you cannot process large documents and extract sentiment for individual sentences directly. The Watson NLP, on the other hand, has been developed for enterprises so it provides the capability to handle large documents easily and you can extract sentence-level sentiment directly using the run method on the raw text:

sentiment_model = watson_nlp.load(watson_nlp.download('sentiment-aggregated_cnn-workflow_en_stock'))
sentiment_result = sentiment_model.run(text)

Targeted Sentiment Analysis

In our testing, we have found that the Watson NLP library works better for targeted sentiment analysis as compared to OpenAI.

Let’s consider this example text: “Terrible service and management, but management was kind enough to help with our request.”

OpenAI extracts the sentiment as

'Sentiment towards service: Negative ', 'Sentiment towards management: Mixed'

using the following prompt

Extract Targeted Sentiment from each sentence in the following text:\n{text}

You can also extract the confidence score for the targeted sentiment using another prompt

Extract Targeted Sentiment from each sentence in the following text with the confidence score:\n{text}

'Sentence 1: Terrible service and management: Sentiment - Negative, Confidence Score - High ',
'Sentence 2: Management was kind enough to help with our request: Sentiment - Positive, Confidence Score - High'

The targeted sentiment output for the same sentence using the Watson NLP library is shown below:

Targeted Sentiment Output

You can observe that the Watson NLP works much better for targeted sentiment analysis as it can identify the three targets and their sentiment every single time they are mentioned in the document using a single line of code:

tsa_model = watson_nlp.download_and_load('targets-sentiment_sequence-bert_multi_stock')
tsa_predictions = tsa_model.run(syntax_prediction)

Therefore, Watson NLP is much better for targeted sentiment analysis and also better for sentiment analysis on large documents using the pre-trained model.

Now for customizing the models, as I have explained in my previous blog OpenAI doesn’t provide documentation or examples at this time to fine-tune or custom-train the base models for any other downstream tasks besides binary classification. Therefore, Watson NLP is the clear winner in terms of customizability with lots of options both in fine-tuning a pre-trained model as well as custom training a Deep Learning or Machine Learning based model

Conclusion

We have seen how the NLP tasks for understanding the voice of the customer like Sentiment analysis can be carried out both using OpenAI and the Watson NLP. However, in terms of customizability, rate limit, and token limit, Watson NLP is the clear winner for all of these tasks.

If you want to read about our comparative analysis for the task Emotion and Tone analysis, you can read this blog.

Learn more about IBM Watson libraries with this list of links to help you find the information you need.