Paper Review 3: Convolutional Neural Networks for Sentence Classification

In this post, the paper “Convolutional Neural Networks for Sentence Classification” is summarized.

Link to paper: http://www.aclweb.org/anthology/D14-1181

Kim, Yoon., 2014, “Convolutional Neural Networks for Sentence Classification”, EMNLP.

In this paper, series of experiments with convolutional neural networks (CNN), trained on top of pre-trained word vectors for sentence level classification tasks are presented. Its shown that a simple CNN with tuned parameters can achieve excellent results. Authors also propose a simple modification to the architecture to allow for the use of both task-specific and static vectors.

Authors’ CNN variant with 2 channels of word vectors can be seen in Figure 1. In this variant, one of the channel is kept static during the training and the other is fine-tuned via backprogation. In the multichannel variant, filter results of both channels are added to form the feature map.

For the experimental setup, 7 datasets, that include movie reviews/customer reviews/question topics,of given specifications in Table 1 are used. As the pre-trained word vectors, authors used the public word2vec vectors that are trained on 100 billion words from Google News. The vectors have dimensionalty of 300 and were trained using the continious bag of words architecture.

Authors have experimented with multiple model variations which are:

CNN-rand: all words are randomly initialized and the updated at training,

CNN-static: pretrained vectors from word2vec kept static and other parameters are learned,

CNN-non-static: same as CNN-static but pretrained vectors are also fine-tuned,

CNN-multichannel: model with 2 channels as mentioned earlier.

Results on Table 2 suggest that, pretrained vectors are better, further fine-tuning works better then staying static, multichannel model doesn’t prevent overfitting, and dropout is proved to be a great regularizer for sentence classification.