Kalibot — The chat-bot built for testers
Introduction
In our daily routine, we often have problems and questions. But sometimes, finding an answer or a solution is not easy. Especially in large companies where dozens of teams are working on different projects with different contexts, knowledges sharing is more difficult.
We do not know if knowledges are documented, or where they have been documented. We may not know who the right person is to address. When we try the search function of Content Management Systems (CMS) like SharePoint or Confluence, the search result is usually not what we are expected.
To address these issues, we have performed some studies aimed to improve the knowledge sharing in our test department. As a result, Kalibot, the knowledge sharing chat-bot has been developed and deployed internally. Built on top of the Microsoft Bot Framework.
Why a chat-bot
When we have questions, we will ask someone else via instant messaging. I think replacing that ‘someone’ by a bot is the most natural way for the user experience. Secondly, with mature AI (Artificial Intelligence) technology, finding a question in an AI model is usually much more accurate than keyword-based search function provided by any content management system.
A simple but effective solution
As an internal project, we want to address the issue but also need the solution to be simple, without much need on maintenance. Also, we have already a large documentation based on Confluence, which is maintained by several teammates. These data should be easily reused, so every tester and test team can contribute and improve the coverage of knowledges and questions.
Finally, we have built not only one bot, but two:
The Indexing robot a.k.a “Crawler” scans and analyzes documents on Confluence and SharePoint, generated Q&A pairs then train data models based on these data.
The classic Chat-bot interacts with users via Microsoft Teams based on knowledge base created by the crawler.
Let us have a look.
Indexing Bot
Indexing robot automatically update knowledge base from original HTML based content management systems.
Original documents are stored on Confluence, SharePoint, or potentially other CMS. They can be maintained by users from different teams. When indexing bot is launched, it scans every page where we asked him to, analyzes the HTML document based on tags like: Headings, Paragraphs, Images, and Links, then generate structured question and answer data based on it.
Once the question/answer and their relations are generated, the indexing bot will invoke Q&A Service API (Application Programming Interface) to train Q&A model and publish it on production. Chat-bot will immediately be able to reply to users those newly added questions without redeployment.
In a well-structured document, there are often multiple levels of headings and their associated paragraphs. That means the question may be too big to be answered in a single reply. In the current implementation, we will treat sub-level heading as “follow-up prompts”. the chat-bot can prompt user follow up questions until he gets a correct answer.
For example: “How to implement pairwise testing” is too big to answer. Indexing bot will generate an intermediate answer to explain what pairwise testing is, then promote user about how to get the tool, and how to use it.
As a result, when user asks a question, the chat-bot will filter and refine the question, until user gets a definitive answer.
Kalibot — The interacting Chat-bot
The main Chat-bot named ‘Kalibot’ is published on Microsoft Teams (the Instant Messaging software used in our company), which internal users can interact with.
Kalibot relays on 3 AI Models:
- Language Understanding Model (LU Model): LU model ables to predict user utterances into “intents”. We use this model to manage user’s profile and preferences, like preferred domain. In the future, we can improve this model to develop more capabilities of the chat-bot.
- Question Answering Model (Q&A Model): This model stores all the question answering pairs and can be updated by the Indexing Bot.
- Dispatch Model: This upper-level model determines which model should be addressed: If user is asking a question to the bot, it should search in Q&A Model, If the user wants to manage his preferences, then the LU Model will be used.
When receives chat message from a User, Kalibot will first query Dispatch model, then Q&A model or LU model according to the context.
Featured functionalities of the Chat-bot
Additional to basic functionalities of chat-bot which answer questions, we have developed some more featured functionalities as follow:
Multi-turn conversation
A document is structured with various levels of heading. This structure can be used to generate “follow-up prompts”, to guide user from a big and unanswerable question to a concrete and precise answer, step by step.
As the example already explained in indexing bot: When asked “how to implement pair-wise testing”. The chat-bot suggests also follow-up prompts like “how to install the tool” and “steps to generate test cases with the tool”. To know more, user can just click on the suggestions to see more details.
Manage preferred contents
Our testers work in different project teams. They may have similar question, but the expected answer highly depends on their context. The answer for “Which test data can I use for a nominal test” will be quite different from one project to another.
To address these contextual questions, we have labeled each knowledgebase with a “Context”, and each user can choose his preferred context from a suggested list. For example: as a tester working in Property Insurance tribe, I can change my context to “Property Insurance”. The knowledge base of my proper project will be used in priority before other knowledge bases.
User will always get the answer from their own knowledge base, else the global knowledgebase maintained by engineering team and other project teams.
Conclusion
As of today, the chat-bot has a knowledge base with about 2000 questions covering Test methods, Test tools, Glossary, ISTQB syllabus and some contents managed by project teams. To continuously updating the Q&A model, indexing bot is launched periodically by using the continuous integration platform.
We are still in early stage of this project, but found it beneficial:
- Many well-structured documentations can be reused directly, that significantly reduces the effort of reuse and the complexity of indexing bot.
- The chat-bot can give a good result for questions already in Q&A Model. It is centralized, more natural, and more accurate than search function provided by content management system.
- Everyone can contribute on Q&A model without the knowledge of AI and underlying technologies. Because anyone can create and update a document on Confluence and SharePoint.
Of course, there are points we would like to improve so the chat-bot can be more useful:
Indexing bot can analyze web pages and process Q&A pairs. But in real life, there are still many documents need adaptation before they can be processed:
- They are not well organized with distinct levels of Heading.
- They may contain tables and columns, or exceptionally long paragraphs, which is not ideal for conversational context.
- They may contain Word or PowerPoint attachments and the current version of indexing bot cannot scan the content from attachments.
The next step will be improving indexing bot, make it able to fetch more data and mobilizing the collaborators to contribute on the documentation. Because the knowledge of the chat-bot is based on what users have provided. In a long run, A shared, well-structured, and well-maintained documentation is the key for keeping the chat-bot always useful.
References
The chat-bot is based on Microsoft Bot Framework SDK and hosted on Azure. The following are some technical documents we have followed during its development:
(Thanks to the Team for reviewing this article, special thanks to Etienne DUFOUR, Louis LUCIANI, and Jean-Prince DOTOU-SEGLA)
