Transfer learning attempt. Comparing Classification Models. Part2

ScreenShot from last NVidia web conference on Microsoft CNTK

In a previous post we have compared general Machine Learning models to solve Question classification task from Question-Answering (QA) problem. But Classifying Question to extract appropriate answer from structured database will not completely solve QA. Real human-to-human conversation is far complex than answering to predefined questions. Neural Networks approaches are likely to get solution of this problem into next level, but not necessarily fully cover it. The thing is that in order to be able to speak like a human being, machines need to understand the world around.

As an active research topic, Sequence based Neural Networks algorithms like Long-Short term Networks (LSTM), End-to-end Memory (MemNN) networks seem to do great job in Goal-Oriented Client Consulting. For example in user preference based restaurant menu recommendation and table reservation.

In this post, I would like to share another results of our experiments, already with Neural Network algorithm included. Furthermore, we were able to apply “Transfer learning” technique and achieve surprisingly high results. We call it Transfer learning because we are fascinated by the creativeness that it brings. Some scientists may blame me that it’ more kind of Joined learning than Transfer and will be right as well. Anyway, let’s walk into data details.

For separate learning, our test set consisted of 102 question and answer pairs and 246 semantically related questions. The training set was about 2 times bigger. For Transfer learning, we combined dataset of our one customer with the dataset of another customer and got 932 instances (questions + semantically related questions) in common as a test test, while training set was also 2 times bigger than test set.

Results (Method — accuracy):

  • Separate. LSTM with lemmas — 72%. For features, we simply took lemmas of each word.
  • Transfer. LSTM with lemmas — 78% (accuracy measured for separated part, not all questions)
  • Separate. Random Forest — 89%. Predefined dictionary of synonyms, dictionary of synonymic expressions and keywords were taken for feature engineering purposes
  • Transfer. Random Forest — 91% (accuracy measured for separated part, not all questions)
  • One-shot learning — still processing (will try to publish asap)

As you can see, Transfer learning approach allowed to rise accuracy of LSTM model from 72% to 78%, which means that by increasing training set, it will better understand semantics of words and sentences.