Natural Language Processing and Word Embeddings (Part 1)

Pranjal Soni
Oct 14 · 7 min read
Image for post
Image for post
Photo by Pixabay from Pexels

This article is based on week 2 of course Sequence Models on Coursera. In this article, I try to summarise and explain the concept of word representation and word embedding.

Word Representation :

V = [a, aaron, …, zulu, <UNK>]

Let’s take the word ‘ Man’ is at position 5391 in the vocabulary, then it can be represented by a one-hot encoded vector (O5391 ). The position of 1 in the sparse vector O₅₃₉₁ is the index of word Man in the vocabulary.

O5391 = [0,0,0,0,...,1,...,0,0,0]

Like that, we can other words in the vocabulary which can are represented by one-hot encoded vectors Women (O9853), King(O4914), Queen(O7157), Apple(O 456), Orange(O6257).

But this method is not an effective method to feed our algorithms to learn sequence models because the algorithm is not able to capture the relationship between different examples.

Suppose we train our model for the sentence:

I want a glass of orange juice.

And want to predict the next word for the sentence:

I want a glass of apple _____.

Even if both examples are almost the same and our algorithm well trained, but it failed to predict the next word in test example. The reason behind this is that in the case of one-hot encoded vector representation the inner product between two one-hot encoded vectors is 0. Even if take the Euclidean distance between for any two vectors it is also 0.

We know that the next word would juice in the example we take, but the algorithm is not able to find any relationship between the words of the above two sentences, and it fails to predict the word in the sentence.

To solve this problem we take the help of word embeddings, it is the featurized representation of words. For each word in the vocabulary, we can learn a set of features and values.

Image for post
Image for post
Word Representation Andrew Ng

Instead of taking a sparse one-hot encoded vector, we take a dense featured vector for each word. We can take different properties of each word and give a weightage of how much the property belongs to the word. For example, Gender property highly belongs to man, woman, king, and queen but not related to Apple and Orange, so high weightage is provided for those four, and less weightage is provided for apple and orange. Thus we are able to establish a relationship between the words which have the same properties. Now our algorithm is also capable to find the relation between apple and orange, and predict the next word according to it.

In the image shown above, there are 300 properties are taken for each word and converted into a vector, now the word man represented by (e₅₃₉₁)

e₅₃₉₁ = [-1, 0.01, 0.03, 0.04, ...]

Here e represents the embedded vector.

We can take help t-SNE (t-distributed stochastic neighbor embedding) machine-learning algorithm to visualize these words into a 2-D plot.

Image for post
Image for post
Visualize word embeddings by Andrew Ng

In this graph, we can see that the words which have the same properties are neighbors and the word which does not have any common property are far away.

Using Word Embedding for Name entity recognition :

Sally Jhonsan is an orange farmer.

We know that Sally Jhonson is the name in the example above, and we train our model for such examples. Now if we give a test example to our algorithm like

Robert Lin is an apple farmer.

By using word embedding the algorithm easily recognize that apple farmer and orange farmer are relatable then it can predict that Rober Lin is the name in the test example.

It is an easy take for our algorithm, now we take another example :

Robert Lin is a durain cultivator.

Durian is a fruit popular in Singapore and a few other countries. And this term is not available in the vocabulary available as training data because our training data set is small. There is a high chance that the algorithm fails to recognize the name in the sentence.

In such case word embedding are useful, it can examine large unlabeled datasets easily. Large datasets can be available online for free. So the word embedding can be learned from that text corpus or dataset and it transferred to our model. In this manner, word embedding can be used through transfer learning for our model. Transfer learning is useful when our text corpus is small.

In the example given above, using embedding for large text corpus and transfer the knowledge gained to our text corpus the algorithm can identify durian and orange share the same properties and framer is related to cultivator, so it can predict the name in the sentence is Robert Lin.

Properties Of Word Embeddings

Image for post
Image for post
Analogies By Andrew Ng

The images show word embedding for a set of words. Now we want to know that if Man is related to Woman in the same manner King belongs to whom?

It can be represented as

Man  ----> Woman
King ----> ?

Here we are using only a 4-dimensional vector instead of using 50 dimensional or 100 dimensional or more. Embedded vectors for words Man, Woman, King, Queen can we wrote as :

eman = [ -1 0.04 0.03 0.09]
ewoman = [ 1 0.02 0.02 0.01]
eking = [ -0.95 0.93 0.70 0.02]
equeen = [ -0.97 0.95 0.69 0.01]

If we take the difference between vectors eman and ewoman then it approximately is :

eman - ewoman ≈[-2 0 0 0]

And if we take the difference between vectors eking and equeen then it is also approximately the same as above.

eking - equeen ≈[-2 0 0 0]

So our algorithm able to figure out if Man is related to Woman then King is also related to Queen in the same manner.

Actually, the algorithm does not simply take the difference between king and queen, the algorithm finds the maximum for the equation given below

eman - ewoman   eking - ew
find word w : argmax Sim(ew, eking - eman +ewoman)

Here w represents the words in the text corpus. So we perform the mathematical operation for eking, eman, and ewoman and check the similarity of the resultant vector with all other embedded vectors. The embedded vector which shows the maximum similarity is the output.

To calculate the similarity we use the Cosine similarity function because when the angle between two vectors is 0 then their Cosin value is 1 which is maximum. Hense we can get the most similar value and if two vectors are at an angle of 90 then it gives 0, which means that two vectors are not related at all. And if the angle between the vector is 180 then it gives -1 which means that vectors are related but in opposite direction. The formula of cosine similarity between two vectors A and B is :

Image for post
Image for post

We can consider the normal euclidian distance formula to calculate the similarity but cosine similarity gives a more convenient result so it is used most often. The main difference between euclidian distance and cosine similarity is that cosine similarity normalizes the vectors as shown in the formula and euclidian distance is generally used for the measure of dissimilarity.

This is how the word embedding is useful for analogy reasoning.

Summary :

Thank you for reading this article. It is just an explanation of lectures in the course Sequence models. If you find any mistake or want to add some information feel to comment. Contact me on LinkedIn for further discussion.

The Startup

Medium's largest active publication, followed by +731K people. Follow to join our community.

Pranjal Soni

Written by

A data science learner and loves to read and write about machine learning and deep learning. Reach me out on LinkedIn

The Startup

Medium's largest active publication, followed by +731K people. Follow to join our community.

Pranjal Soni

Written by

A data science learner and loves to read and write about machine learning and deep learning. Reach me out on LinkedIn

The Startup

Medium's largest active publication, followed by +731K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store