1 line to BERT Word Embeddings with NLU in Python

Including Part of Speech, Named Entity Recognition, Emotion Classification in the same line! With Bonus t-SNE plots!

With the freshly released NLU library which gives you 350+ NLP models and 100+ Word Embeddings, you have infinite possibilities to explore your data and gain insights.

In this tutorial, we will cover how to get the powerful BERT embeddings with 1 line of NLU code and then how to visualize them with t-SNE.

T-SNE [1] is a tool to visualize high-dimensional data. It converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data. t-SNE has a cost function that is not convex, i.e. with different initializations we can get different results.

1. Import NLU, load Bert, and embed a sample string in 1 line

2. Load a larger dataset

The following snippet will download a Reddit sarcasm dataset and load it to a pandas Dataframe

3. Predict on the dataset and also add Part of Speech, Emotion and Sentiment Classifiers

Since adding additional classifiers and getting their predictions is so easy in NLU, we will extend our NLU pipeline with a POS, Emotion and Sentiment classifier wich all achieve results close to the state of the art.

Those extra predictions will also come in handy when plotting our results.
We are limiting ourselves to a subsection of the dataset, because our RAM is sadly limited and we are not running on a cluster. With Spark NLP you can take exactly the same models and run them in a scalable fashion inside of a Spark cluster

4. Emotion Plots

We can quickly plot the distribution of predicted emotions using pandas functions on the dataframe

5. Prepare data for T-SNE

We prepare the data for the T-SNE algorithm by collecting them in a matrix for TSNE

6. Fit T-SNE

Finally, we fit the T-SNE algorithm and get our 2-Dimensional representation of our Bert Word Embeddings

7. Plot Bert Word Embeddings, colored by Part of Speech Tag

The following plots show scatter plots for the 2-D representation of the Word Embeddings. Each point represents a word in a sentence and the color represents the POS class that word belongs to.

8. Plot Bert Word Embeddings, colored by Emotion

The following plots show scatter plots for the 2-D representation of the Word Embeddings. Each point represents a word from a sentence that was classified with a particular emotion, which reflects in the colors

9. Plot Bert Word Embeddings, colored by Sarcasm

The following plots show scatter plots for the 2-D representation of the Word Embeddings. Each point represents a word from a sentence that was classified as sarcastic or not, which reflects in the colors

Data Science, Big Data, Data Engineering, DevOps expert