Ankit Choudhary
Aug 16 · 8 min read

Introduction

I’m a big fan of hackathons. I’ve learned so much about data science from participating in these hackathons in the past few years. I’ll admit it — I have gained a lot of knowledge through this medium and this, in turn, has accelerated my professional career.

This comes with a caveat — winning a data science hackathon is really hard. Just think about the number of obstacles in your way:

  • A brand new problem statement we haven’t worked on before
  • A plethora of top data scientists competing to rise up the leaderboard
  • Time crunch! We have to understand the problem statement, put together a framework, clean the data, explore it, and build the model in a matter of a few hours
  • And then repeat the process!

A single decimal point could be the difference between the top 10 and the top 50. Isn’t this why we love hackathons in the first place? The thrill of seeing our hard work pay off with a rise in the leaderboard rankings is unparalleled.

So, we’re thrilled to bring to you the top 3 winning approaches from the Innoplexus Sentiment Analysis hackathon! You are going to be awestruck by how these three top data scientists thought through their solutions and came up with their own unique framework.

There is a LOT to learn from these approaches. Trust me, take the time to go through the steps and understand where they came from. And then think if you would have done anything differently. And then — go ahead and take part in these hackathons yourself on our DataHack platform!

So let’s begin, shall we?

About the Innoplexus Sentiment Analysis Hackathon

It’s always an exciting prospect, hosting hackathons with our partner Innoplexus. Each time they come up with problem statements that are based on Natural Language Processing (NLP), an immensely popular field right now. We have seen huge developments in NLP thanks to transfer learning models such as BERT, XLNet, GPT-2, etc.

Moreover, sentiment analysis is one of the most common NLP projects data scientists tend to work on. This Innolpexus hackathon was a 5-day contest with more than 3200 data scientists across the globe competing for job opportunities and exciting prizes offered by Innoplexus.

It was a hard-fought contest with a total of 8000+ submissions and a variety of approaches employed by the best in the business to occupy the top spots.

For those of you who could not make it to the top, or otherwise could not find time to work on the problem, we have collated the winners’ approaches and solutions to help you appreciate and learn from these. So here goes.

Problem Statement for the Innoplexus Sentiment Analysis Hackathon

There are a lot of components that go into building the narrative of a brand. It isn’t just built and controlled by the company that owns the brand. Think about any big brand you are familiar with and you’ll instantly understand what I’m talking about.

For this reason, companies are constantly looking out across various platforms, such as blogs, forums, social media, etc. for checking the sentiment around their various products and also competitor products to learn how their brand resonates in the market. This analysis helps them in various aspects of their post-launch market research.

This is relevant for a lot of industries, including pharma and their drugs.

But this comes with several challenges. Primarily, the language used in this type of content is not strictly grammatically correct. We often come across people using sarcasm. Others cover several topics with different sentiments in one post. Other users post comments to indicate their sentiment around the topic.

Broadly speaking, sentiment can be clubbed into 3 major buckets — Positive, Negative and Neutral Sentiments.

Given the text and drug name, the task was to predict the sentiment for texts contained in the test dataset. Given below is an example of text from the dataset:

Example:

Stelara is still fairly new to Crohn’s treatment. This is why you might not get a lot of replies. I’ve done some research, but most of the “time to work” answers are from Psoriasis boards. For Psoriasis, it seems to be about 4–12 weeks to reach a strong therapeutic level. The good news is, Stelara seems to be getting rave reviews from Crohn’s patients. It seems to be the best med to come along since Remicade. I hope you have good success with it. My daughter was diagnosed Feb. 19/07, (13 yrs. old at the time of diagnosis), with Crohn’s of the Terminal Illium. Has used Prednisone and Pentasa. Started Imuran (02/09), had an abdominal abscess (12/08). 2cm of Stricture. Started ​Remicade in Feb. 2014, along with 100mgs. of Imuran.

The above text is ​positive​ for Stelara and negative for Remicade. Now that we have a solid understanding of what the problem at hand was, let’s dive into the winning approaches!

Winners of the Innoplexus Sentiment Analysis Hackathon

As I mentioned earlier, winning a hackathon is extremely difficult. I loved going through these top solutions and approaches provided by our winners. First, let’s look at who won and congratulate them:

Here are the final rankings of all the participants on the Leaderboard.

The top 3 winners have shared their detailed approach from the competition. I am sure you are eager to know their secrets so let’s begin.

Rank 3: Mohsin Hasan Khan (ML Engineer @HealthifyMe)

Here’s what Mohsin shared with us:

Approach

“My final solution is an ensemble of BERT and XLNet runs.”

  • My first impression of the data suggested there were a lot of wrong labels as per my perception of negative and positive sentiment. So, I felt it would be really difficult to handcraft the features. I decided it would be best to stick to the state-of-art NLP models to learn on noisy data.
  • I started with a simple TF-IDF plus logistic regression model which gave me a cross-validation (CV) score of 0.5. After looking at text data, I realized there were many rows that had a lot of lines unrelated to the drug.
  • Hence, I decided to use only sentences that had a drug name occurring in them. TF-IDF plus logistic regression with drug name-only sentences gave me a CV score of 0.54
  • At this point, I decided to use BERT. Without any fine-tuning, I only got a CV score of 0.45. But, once I let BERT fine-tune on training data, it gave a validation score of 0.60 and a leaderboard score of 0.59. Then, I added sentences that occurred before and after the drug sentence — this increased the CV score slightly. Then I used BERT-large and fine-tuned it which gave me a CV score of 0.65 and a leaderboard score of 0.61. Similarly, I fine-tuned the XLNet base, which gave a CV score of 0.64 and leaderboard 0.58. My final solution is an ensemble of BERT and XLNet runs
  • Note: 5-Fold stratified K-Fold as class distribution was imbalanced

Rank 2: Harini Vengala (Statistical Analyst @WalmartLabs India)

Here’s what Harini shared with us:

Approach

“My final model was an ensemble of 3 BERT and 1 AEN.”

  • My first baseline approach was using a count vectorizer. This got me a public leaderboard score of 0.42. I removed digits, emojis, URLs, punctuations, and converted the text to lowercase.
  • I tried BERT and tuned it for the given data set. Next, I removed stop words from the text as BERT requires more memory if I run it on the entire passage. I took sequence length as 150, but I realized most of the important information is ignored in this approach. I didn’t cross the 0.50 score on the public leaderboard.
  • So, what else could I try? I took only the sentences in which the given drug was present and used BERT again to classify sentiment. This gave me a score of 0.60 on the public leaderboard. I also implemented the Attentional Encoder Network (AEN) for Targeted Sentiment Classification which resulted in 0.56 score
  • My final model was an ensemble of 3 BERT and 1 AEN. The loss function I used was CrossEntropyLoss with class weights = 1/number of observations in each corresponding class.
  • My key takeaway — try different things and check what works according to the data. Spend some time listing what are all the things you can try during the hackathon.

Rank 1: Melwin Babu (Data Scientist @nference)

Here’s what Melwin shared with us:

Approach

“I noticed pretty early that increasing the max sequence length increased the score sufficiently. This observation more or less dictated my approach. I used a basic XLNet model with hardly any feature engineering.”

  • I lowercased all the sentences and masked the relevant drug in the sentences. Then, I proceeded to take the first 1380 tokens after the sentence piece tokenization.
  • I chose to fill the GPU RAM with as much max sequence length as possible and refrained from using extra features. I tried to add variations to the data but made implementation mistakes and ran out of time.
  • In the final model, I have used 6 seeds to average the XLNet base cased model’s predictions. Time didn’t permit me for any other hyperparameter tuning.
  • Really surprised that in a competition where deep learning could be the best solution, I could be competitive with just an 8 GB GPU RAM machine
  • Identifying the differences in train and test distribution may be crucial. Most of the things are the same as other hackathons.

End Notes

It was great fun interacting with these winners and getting to know their approach during the competition. This is a tightly contest hackathon and as you have already seen, the winning approaches were supremely awesome.

I encourage you to head over to the DataHack platform TODAY and participate in the ongoing and upcoming hackathons. It will be an invaluable learning experience!

Originally published at https://www.analyticsvidhya.com on August 16, 2019.

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Ankit Choudhary

Written by

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade