Empowering Chatbots with Intent Discovery

Hi, I’m Maarten de Raedt, and I’m a Natural Language Processing (NLP) engineer.

Hello, I’m Fréderic Godin, and I’m Head of Artificial Intelligence (AI) at Sinch.

Working with NLP, we see that Large Language Models (LLMs) have gained significant popularity due to their exceptional performance in various Natural Language Processing (NLP) tasks, ranging from summarization to machine translation (ChatGPT is here to prove it).

At Sinch Chatlayer, recognizing the impressive capabilities of LLMs, we are actively conducting fundamental research on how to employ them in real-world applications including response generation, dialogue summarization, and intent discovery. Our goal is to understand and investigate how and where LLMs can improve Sinch Chatlayer’s product.

The Annual Association of Computational Linguistics (ACL) is one of the leading conferences in the field of Natural Language Processing (NLP), along with EMNLP. In the last ACL we introduced our approach called IDAS (Intent Discovery with Abstractive Summarization), which combines both small language models and LLMs to improve intent discovery.

Intent Discovery: A Key to Efficient Chatbot Design

Conversational agents, commonly known as chatbots, rely on annotated training data to detect and respond to specific questions. Identifying question types (or intents, as we called) during the chatbot’s design phase is crucial. However, manually annotating intents from unlabeled conversations is time-consuming. Intent discovery addresses this challenge by automatically identifying question types from conversations without any annotations, streamlining the process of creating efficient chatbots.

Figure 1. Intent discovery takes as input a set of user queries and partitions them into groups such that utterances conveying the same intent are part of the same cluster, whereas utterances of different clusters have distinct intents.

Above, in Figure 1 we illustrate the usual intent discovery task, where user queries from conversations between users and support agents are partitioned into three groups. The aim is to group utterances with the same intent together while ensuring that utterances in different groups also represent distinct intents. Intent discovery is thus an important task that — if done effectively — substantially reduces the manual effort required customers to design and deploy chatbots.

Existing Clustering-Based Methods

Previous intent discovery approaches typically involve two steps:

1. A sentence encoder is used to map textual utterances to numerical vectors.
2. Clustering algorithms are applied to infer latent intents based on these vector representations.

However, these methods may include noisy features unrelated to the underlying intent, leading to less accurate clustering results. Below you can see in Figure 2, how intent discovery methods use a sentence encoder and a clustering algorithm.

Figure 2. A schematic overview of the two-step approach to intent discovery: textual user utterances are first mapped to vectors, after which they are clustered into intent groups.

The IDAS Approach: Leveraging LLMs for Improved Intent Discovery

To address the issue of noisy features, the IDAS approach introduces a pre-processing step that normalizes user queries before using a sentence encoder. The goal is to remove irrelevant information from the textual space, making vector representations more similar for utterances conveying similar intents. This normalization is achieved through abstractive summarization, which succinctly summarizes user queries while abstracting non-relevant information. The powerful summarization capabilities of LLMs, such as InstructGPT (text-davinci-003), enable the generation of concise intent-related descriptions while discarding irrelevant aspects like syntax and nouns.

Below in Table 1 we demonstrate how the summaries preserve intent-related information yet discard irrelevant aspects. As can be seen, the generated summaries by InstructGPT make user queries with similar intents also more similar directly in the textual space.

Table 1. Illustration based on GPT-3 demonstrating how abstractly summarizing utterances retains the core elements while removing non-intent related information.

In Figure 3, we provide a schematic overview of the three steps our IDAS approach to discover intents.

Figure 3. A schematic overview of our proposed IDAS method, where user queries are abstractly summarized before being encoded and clustered into intent groups.

Experimental Setup and Results

To evaluate the effectiveness of the IDAS approach, the research team conducted experiments using widely adopted public intent datasets (Banking77, CLINC150, and StackOverflow) and a private dataset provided by a customer focusing on transportation-related user queries, to assess IDAS’s performance in real-world scenarios.

We use 3 popular external cluster metrics: Adjusted Rand Index (ARI), Normalized Mutual Information (NMI), and Cluster Accuracy (ACC), to evaluate the clustering performance of our IDAS approach. These metrics assign a score between 0 and 100%, with a higher score indicating a better alignment between the induced clusters and the ground truth intents. A score of 0 represents random clusters, where user queries are assigned to clusters without considering their intent-relatedness.

To compare the performance of our IDAS approach, we benchmarked it against the state-of-the-art model for intent discovery, i.e., the MTP-CLNN model of Zhang et al. (2022).

Figure 5 shows the ARI scores for all four datasets, highlighting the performance of both our IDAS approach and the state-of-the-art model we compared it to. The results clearly demonstrate the advantages of leveraging a LLM to improve intent discovery.

Figure 4. Comparison of our IDAS approach and the state-of-the-art on 3 public and 1 private datasets (Transport).

As you can see, here at Sinch, we’re in line with our commitment, we are actively implementing an intent discovery feature within our platform and are confident that this feature will further improve the chatbot building experience for our clients.

For those interested in diving deeper into our research, you can find our full paper, created in a collaboration with Ghent University, with Prof. Demeester ad Prof. Develder, titled “IDAS: Intent Discovery with Abstractive Summarization” at the following link: https://arxiv.org/abs/2305.19783.

Interested to learn more about Sinch and perhaps become a part of our team? Check out our Careers page!

References

Yuwei Zhang, Haode Zhang, Li-Ming Zhan, Xiao-Ming Wu, and Albert Lam. 2022. New intent discovery with pre-training and contrastive learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 256–269, Dublin, Ireland. Association for Computational Linguistics

Maarten De Raedt, Fréderic Godin, Thomas Demeester, and Chris Develder. 2023. IDAS: Intent Discovery with Abstractive Summarization. In Proceedings of the 5th Workshop on NLP for Conversational AI (ACL).