Utilizing Small Language Models for Efficient and Transparent Question Answering

Researchers unveil a new chain-of-reasoning architecture that leverages small language models to make question-answering tasks efficient and transparent

The emergence of large language models (LLMs) has led to the development of improved question-answering models. However, LLMs suffer from challenges such as hallucinations and outdated information issues. In response, researchers have developed a new open-domain question-answering model named “EffiChainQA.” This architecture uses a novel chain-of-reasoning pipeline relying on small language models with an emphasis on cost-effectiveness. The innovative algorithm can pave the way towards efficient, reliable, and transparent question-answering models.

With the recent emergence of large-scale language models (LLMs), computers can now recognise and respond to human languages, resulting in significant improvements in the methods used for complex question-answering (QA) algorithms. However, despite their capabilities, LLMs also have some inherent limitations. For instance, they sometimes experience hallucinations, resulting in false inferences during the reasoning process instead of using factual data. Moreover, once established, LLMs are unable to evolve using new information, making them prone to obsolescence quickly.

Studies have shown that retrieval-based language models (LMs) effectively address these issues. Such LMs use a retriever to obtain highly relevant documents from an external corpus and combine them with the model to provide a suitable response. Due to these advantages, the Retrieve-then-Read pipeline has become quite popular among LLMs. Moreover, several studies have shown that even smaller LMs, when aided by retrieval-based methods, exhibited comparable or even superior performances in addressing problems as compared to LLMs. Furthermore, they are also known to be more efficient and adept at simultaneously mitigating the issues pertaining to hallucinations and outdated information.

Inspired by these findings, Dr. Jihyeon Roh and Dr. Minho Kim, as joint first authors, along with their colleague Dr. Kyoungman Bae, all from the Electronics and Telecommunications Research Institute in Korea, has now developed a novel open-domain QA framework, named “EffiChainQA.” The architecture leverages the efficiency of small LMs instead of using LLMs. “Our findings have the potential to be applied in a variety of real-life applications involving complex question answering. It can be utilized not only to improve search engines but also in educational and personal assistant tools,” says Dr. Roh. Their findings were published in the ETRI Journal.

The innovative EffiChainQA framework utilizes a novel chain-of-reasoning pipeline. This pipeline consists of three modules — a reasoning type classifier, a question decomposer, and a RecomposeNet algorithm for each reasoning type. The reasoning type classifier checks the reasoning type of the incoming question and classifies them into either bridges or comparisons. The question decomposer then breaks down the questions into simpler sub-questions. For the question decomposer, the team leveraged the well-known LLM ChatGPT to generate the training data. Furthermore, the final module RecomposeNet utilizes both the sub-questions and the original question to generate the final answer.

The team tested the performance of their framework by using the HotpotQA dataset. They found that EffiChainQA scored 15 points more than the existing Chain-of-Thoughts algorithm and ten points more than the Self-Consistency algorithm. The researchers also conducted ablation studies to identify the way in which each part of the framework contributes to its performance. These revealed that, for comparison-type questions, the sub-answers significantly improved the performance. Moreover, the use of the original question in the generation of the final answer improved the performance.

“Once the system starts getting better at decomposing and answering complex questions, a wider audience can access and understand vast amounts of information more easily. This can ultimately lead to more information and education being available to the public and can aid in facilitating professional decision-making,” remarks Dr. Roh, highlighting the potential of EffiChainQA.

Overall, this novel framework, with its reliance on small LMs, paves the way for realizing more efficient, reliable, and transparent question-answering artificial intelligence systems!

Reference

Title of original paper: Towards a small language model powered chain-of-reasoning for open-domain question answering

Journal: ETRI Journal

DOI: https://doi.org/10.4218/etrij.2023-0355

About the Electronics and Telecommunications Research Institute (ETRI)

Established in 1976, ETRI is a non-profit government-funded research institute in Daedeok Science Town in Daejeon and is one of the leading research institutes in the wireless communications and artificial intelligence domains. It has filed more than 2500 patents. Equipped with state-of-the-art labs, this institute strives for social and economic development through technology research.

About Dr. Jihyeon Roh

Jihyeon Roh received her B.S. degree in Electronic and Electrical Engineering from Sungkyunkwan University in 2013, and her Ph.D. degree in Electrical Engineering from Korea Advanced Institute of Science and Technology in 2022. Since 2022, she has been a researcher at the Language Intelligence Lab in Electronics and Telecommunications Research Institute. Her primary research interests include neural language models, natural language processing, language generation system, and machine learning for language models.

About Dr. Minho Kim

Minho Kim received his B.S. degree in Electronics from Korea University, Seoul in 1997, and M.S. and Ph.D. degrees in Information and Communications Engineering from Gwangju Institute of Science and Technology, Gwangju in 1999 and 2006, respectively. Since 2006, he has worked at the Electronics and Telecommunications Research Institute as a principal researcher. His research interests include natural language processing, deep neural network language models, and reasoning process in artificial intelligence.

About Dr. Kyoungman Bae

Kyoungman Bae received his B.S., M.S., and Ph.D. degree in Computer Engineering from the Department of Computer Engineering, Dong-A University, Busan, in 2004, 2006, and 2016, respectively. Since 2016, he has worked at the Language Intelligence Lab at Electronics and Telecommunications Research Institute, Daejeon. His primary research interests are large language models, natural language processing, explainable AI, and generative AI.