NLU benchmark for intent detection and named-entity recognition in call center conversations

TL;DR

What is NLU ?

Illustration of a customer intent and several entities that are extracted from conversation

Project presentation

  • understand the customer’s intent during the call (i.e.: text classification)
  • and catch the important elements that would make it possible to answer the customer’s request (i.e.: named-entity recognition), for example contract numbers, product type, product color, etc…

Benchmark

Intent detection

  • FastText: library for efficient learning of word representations and sentence classification created by Facebook’s AI Research lab.
  • Ludwig: toolbox that allows to train and test deep learning models without the need to write code, using the command line or the programmatic API. The user just has to provide a CSV file (or a pandas DataFrame with the programmatic API) containing his/her data, a list of columns to use as inputs, and a list of columns to use as outputs, Ludwig will do the rest.
  • Logistic regression with spaCy preprocessing: classic logistic regression using scikit-learn library with custom preprocessing using spaCy library (tokenization, lemmatization, removing stopwords).
  • BERT with spaCy pipeline: spaCy model pipelines that wrap Hugging Face’s transformers package to access state-of-the-art transformer architectures such as BERT easily.
  • LUIS: Microsoft cloud-based API service that applies custom machine-learning intelligence to a user’s conversational, natural language text to predict intent and entities.
  • Flair: framework for state-of-the-art NLP for several tasks such as named entity recognition (NER), part-of-speech tagging (PoS), sense disambiguation and classification.
Performance results of different models for intent detection
  • Overall, in terms of performance, all the solutions achieve good or even very good results (F1-score > 70%).
  • One of the inconveniences of Ludwig and LUIS is that they are very “black-box” models which make them more difficult to understand and fine-tune.
  • LUIS is the only solution tested that is not open source thus it is much more expensive. In addition, the use of its Python API can be complex since it has been initially designed to be used via a click-button interface. However, it can be a solution to prefer if you are in the context of a project that aims to go into production and whose infrastructure is built on Azure for example, the integration of the model will then be easier.

Entities extraction

  • spaCy: open-source library for advanced Natural Language Processing in Python that provides different features including Named Entity Recognition.
  • LUIS: see above
  • Ludwig: see above
  • Flair: see above
Performance results of different models for named entity recognition
  • Two models perform really well on custom named-entity recognition, spaCy and LUIS. Ludwig and Flair would require some fine-tuning to obtain better results, especially in terms of recall.
  • One advantage of LUIS is that the user can leverage some advanced features for entity recognition such as descriptors which provides hints that certain words and phrases are part of an entity domain vocabulary (e.g.: color vocabulary = black, white, red, blue, navy, green, …).

Conclusion

--

--

Artefact is a tech company dedicated to solving data challenges by combining state-of-the-art Machine Learning and advanced software engineering. We leverage our business knowledge to deliver tailor-made solutions and bring value to our clients. www.artefact.com @ Artefact

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store