Multi-channel CNN for text

Here is an implementation of the paper by Kim et al. for sentence classification using multi-channel CNNs.

I won’t go into details of the text input/preprocessing steps etc. since they are standard pipelines already available on several blogs, as well as Keras’ examples. [Check this post]

The multi-channel architecture constructs 2 channels of word embeddings using Glove/Word2Vec for the input sentences.

On one channel, pre-trained vectors with static weights are used, while in the other, pre-trained vectors with learnable weights are used. It would be interesting to see if having Glove & Word2Vec vector embeddings as the two channels improves the model.

These two channels are fed to a CNN model.

Here’s a simple model that does this:

embed1 = Sequential()
embed1.add(Embedding(len(word_index) + 1,
                          EMBED_DIM,
                          weights=[embedding_matrix],
                          input_length=MAX_SEQUENCE_LENGTH,
                          trainable=False))
embed2 = Sequential()
embed2.add(Embedding(len(self.word_index) + 1,
                          EMBED_DIM,
                          weights=[embedding_matrix],
                          input_length=MAX_SEQUENCE_LENGTH,
                          trainable=True))
model = Sequential()
model.add(Merge([embed1 ,embed2], mode='concat', concat_axis=-1))
model.add(Reshape((2, MAX_SEQUENCE_LENGTH, EMBED_DIM)))
model.add(Convolution2D(64, 5, EMBED_DIM, activation="relu", border_mode='valid'))
model.add(MaxPooling2D((MAX_SEQUENCE_LENGTH-5+1,1)))
model.add(Flatten())
model.add(Dense(256, activation="relu"))
model.add(Dropout(0.3))
model.add(Dense(len(nb_labels), activation="softmax"))
model.compile(loss='categorical_crossentropy',
                optimizer='rmsprop',
                metrics=['acc'])