NATURAL LANGUAGE PROCESSING
(PART 2 — Word2Vec)
Word2Vec
One of the biggest problems in natural language processing is to symbolize the meanings of words in a way that machines can understand. If computers can attribute meanings to the words people use, they will come closer to understanding our language. Words must be represented numerically for the computer to understand. When we represent words numerically, we can apply mathematical operations to words.
If we create one-hot vectors, the relationship between these words is lost. So, we need other techniques for words representation. The most modern techniques for creating word vectors: Word2Vec and Glove. In this post i explain Word2Vec. I will tell you about Glove in the next post.
Word2Vec creates a model that tries to guess the words around it using the middle word. Word2Vec is a method to construct such an embedding. It can be obtained using two methods (both involving Neural Networks): Skip Gram and Common Bag Of Words (CBOW)
Skip-Gram
Firstly, we choose our input word. (the input word of this sentence is “motivation”). We select window size 2. Our model tries to guess words o 2 window sizes from the right and left of the input word. To achieve this goal Skip-gram will work as a predictor of the context words within a window of fixed size m for a given center word.
Hence, for a vocabulary of size T the Skip-gram model will want to maximize the following predictive accuracy or likelihood.
For ease of computation, and because we prefer minimizing rather than maximizing functions in machine learning, we will now consider and minimize the average negative log probability:
Let's optimize the word vectors to reduce the loss. Each word has two vectors. One is the vector that word in the middle, and the other is the vector with that word on the edge. Then we will use the softmax function to compute the probability of a context word o (outside word) given a center word c.
We calculate the dot product between two vectors. If the value we got is high, the two words are close. If the dot product result is small, its words are far apart.
Let’s show all parameters as vector Q.
Optimization
Let's take the partial derivative of our negative log-likelihood formula. Our aim is to change the parameters in the most appropriate way and bring the result closer to zero.
CBOW
We predict the middle word according to the words on the edge.
We can modelthis CBOW architecture now as a deep learning classification model such that we take in the context words as our input, X and try to predict the target word, Y. Let’s talk about the details of CBOW in another blog post…
Word2Vec with Python Code
#import numpy as np
#from gensim.models import Word2Vec
#from sklearn.manifold import TSNE
#import matplotlib.pyplot as pltFirstly, i import the necessary libraries. Numpy library to do operations on vectors and matrices. Gensim is a popular open-source NLP library, so i import Word2Vec to this library. Then i import matplotlib library and TSNE from sklearn library to visualize the relationship between words.
#f = open('/content/drive/MyDrive/medium_posts/text.txt','r',encoding='utf8')
#txt = f.read()Let's open text files with the “open” command. Using 'utf8' will ensure correct reading of Turkish characters. I assign the text to the txt variable.
#txt_list = txt.split('\n')Separating from all the new lines I have.
#corpus = []
#for sentence in txt_list:
#corpus.append(sentence.split())Loop takes a sentence with each new loop and splits it into words.
#print(corpus[:3])
#[["It's", 'no', 'coincidence', 'that', 'Russia', 'has', 'christened', 'its', 'Covid', 'vaccine', 'Sputnik', 'V.', 'The', 'first', 'time', 'the', 'world', 'learned', 'the', 'meaning', 'of', 'the', 'Russian', 'word', 'Sputnik', 'was', 'in', '1957', 'when', 'the', 'Soviet', 'Union', 'launched', 'the', 'first', 'man-made', 'satellite', 'into', 'orbit.At', 'the', 'height', 'of', 'the', 'Cold', 'War', 'this', 'startling', 'evidence', 'of', "Moscow's", 'scientific', 'and', 'technical', 'capabilities', 'came', 'as', 'a', 'huge', 'shock', 'to', 'Western', 'powers,', 'which', 'had', 'assumed', 'they', 'enjoyed', 'a', 'comfortable', 'technological', 'lead', 'over', 'the', 'Soviets.'], [], ['Critics', 'of', 'the', 'Putin', 'administration', 'were', 'sceptical', 'when', 'the', 'vaccine', 'was', 'given', 'regulatory', 'approval', 'in', 'Moscow', 'as', 'early', 'as', 'last', 'August.']]Look at the words in the first three sentences.
#model = Word2Vec(corpus, size = 50, window=4, min_count=5,sg=1)The length of our word vectors is 50. With our window span, we will get 4 words from the right and left of the word in the middle. We will take words that occur at least 5 times in the corpus. We are stating that we will use the Skip-gram algorithm with “sg”.
#model.wv[‘vaccine’]array([-0.00601876, 0.00556641, -0.0009739 , 0.00350551, -0.00833465, 0.00864567, 0.005754 , -0.01082865, -0.01034463, -0.00477982, -0.00049233, -0.00924585, 0.00180515, -0.00676984, -0.00527921, -0.00975458, -0.00525116, -0.0011445 , 0.00144782, -0.00268782, -0.00011491, -0.00980046, 0.00990428, -0.00424471, -0.00588545, -0.00075011, -0.004544 , 0.00074751, -0.00642032, -0.00937617, 0.00119038, 0.00496139, -0.00366728, 0.0024138 , 0.00456811, -0.00508722, 0.00816515, 0.00014169, -0.00383437, 0.00237516, -0.00699971, -0.00181684, 0.00478987, 0.00201446, 0.00325257, 0.00989156, -0.0076406 , -0.00603311, -0.00122619, -0.0045702 ], dtype=float32)
Vector representation of the word “mountain”.
#model.wv.most_similar(‘Russia’)
[('turns', 0.43600738048553467),
('make', 0.4169993996620178),
('he', 0.3553329110145569),
('Gamaleya', 0.3478131890296936),
('rather', 0.3375583589076996),
('threat,', 0.33636632561683655),
('Soviets.', 0.32479652762413025),
('Cold', 0.31980639696121216),
('independent', 0.314931184053421),
('now', 0.2847120463848114)]These vectors are the closest to the word “sports”.
#model.save(‘Word2vec.model’) #save the model
# model = Word2Vec.load('Word2vec.model') #reload the saved model#def plot_closest_words(model,word):
word_vectors = np.empty((0,50))
word_labels = [word]
close_vectors = model.wv.most_similar(word)
word_vectors = np.append(word_vectors, np.array([model.wv[word]]), axis=0)
for w,_ in close_vectors:
word_labels.append(w)
word_vectors = np.append(word_vectors,np.array([model.wv[w]]), axis=0)
tsne = TSNE(random_state=0)
Y = tsne.fit_transform(word_vectors)x_coordinate = Y[:,0]
y_coordinate = Y[:,1]plt.scatter(x_coordinate,y_coordinate)for label,x,y in zip(word_labels, x_coordinate, y_coordinate):
plt.annotate(label, xy=(x,y), xytext=(5,-2), textcoords='offset points')plt.show()plot_closest_words(model, 'Covid')
SKIP-GRAM VS CBOW
The Skip-gram model is designed to predict the context. For a center word, all the 6 context words are predicted and loss is calculated based on the 6 predictions. It works well with a small amount of the training data, represents well even rare words or phrases.
CBOW is learning to predict the word by the context. For the 6 words, only one prediction is done and loss is calculated based on a prediction of the center word. Several times faster to train than the skip-gram, slightly better accuracy for the frequent words.
SOURCES
